Figure out how to run Filebeat on Docker

[zmoog]

I want to run Filebeat 8.11.3 with a custom configuration on Docker for testing purpose.

Requisites:

• Access configuration files filebeat.yml using volume.
• Write events to a local folder using the Filebeat file output.

[zmoog]

Using https://www.elastic.co/guide/en/beats/filebeat/8.11/running-on-docker.html as source material.

[zmoog]

Pull the image

docker pull docker.elastic.co/beats/filebeat:8.11.3

[zmoog]

Configure Filebeat

 cat filebeat.yml

# ================================== General ====================================
fields_under_root: true
fields:
  service:
    environment: ${ENVIRONMENT}
    app: ${APP_NAME}
    prefix: ${SERVICE_PREFIX}
    name: ${SERVICE_PREFIX}-${ENVIRONMENT}-${APP_NAME}

# ============================== Filebeat inputs ===============================

filebeat.inputs:

  # Each - is an input. Most options can be set at the input level, so
  # you can use different inputs for various configurations.
  # Below are the input-specific configurations.

  # apigw-access
  - type: aws-s3
    enabled: true
    queue_url: "${SQS_QUEUE_URL}"
    access_key_id: "${AWS_ACCESS_KEY_ID}"
    secret_access_key: "${AWS_SECRET_ACCESS_KEY}"
    api_timeout: 30s
    visibility_timeout: 10s
    sqs.max_receive_count: -1
    sqs.wait_time: 20s
    fields:
      service:
        log_type: "apigw-access"
    fields_under_root: true

output.file:
  path: "/tmp/filebeat/"
  filename: filebeat
  rotate_every_kb: 10000
  number_of_files: 7

# ================================== Logging ===================================

logging:
  level: ${LOG_LEVEL:debug}
  metrics:
    enabled: false
  to_files: true
  files:
    path: /tmp/filebeat/log/filebeat
    interval: 86400
    rotateeverybytes: 31457280
    keepfiles: 7
    permissions: 0644
    rotateonstartup: false

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
      #  - add_cloud_metadata: ~
      #  - add_docker_metadata: ~

[zmoog]

Run volume-mounted configuration

Run Filebeat using filebeat.yml file using ./tmp/filebeat/ as output:

docker run \
  --rm \
  --name=filebeat \
  --user=root \
  --volume="$(pwd)/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro" \
  --volume="$(pwd)/tmp:/tmp" \
  --volume="/var/lib/docker/containers:/var/lib/docker/containers:ro" \
  --volume="/var/run/docker.sock:/var/run/docker.sock:ro" \
  --volume="registry:/usr/share/filebeat/data:rw" \
  docker.elastic.co/beats/filebeat:8.11.3 filebeat -e --strict.perms=false \
  -E AWS_ACCESS_KEY_ID="<REDACTED>" \
  -E AWS_SECRET_ACCESS_KEY="<REDACTED>" \
  -E SQS_QUEUE_URL="<REDACTED>" \
  -E ENVIRONMENT=dev \
  -E APP_NAME=myapp \
  -E SERVICE_PREFIX=prefix 

[zmoog]

Testing

I use a sample file to the SQS_QUEUE_URL:

aws s3 cp test_file.gz s3://S3_BUCKET/6.gz

And here's what I see in the ./tmp/ folder:

$ tree tmp/
tmp/
└── filebeat
    └── filebeat-20240119.ndjson

1 directory, 1 file