search

sematext / sematext-agent-docker

Sematext Docker Agent - host + container metrics, logs & event collector
https://sematext.com/docker
Apache License 2.0
209 stars 30 forks source link

Add filter to remove unwanted fields from logs #45

Closed megastef closed 6 years ago

megastef commented 6 years ago

SDA enriches logs with metadata fields to Docker Swarm and kubernetes logs such as swarm_node, swarm_task_id, swarm_service, Kubernetes namespace, pod name, UUID etc.

It should be possible to remove unwanted fields from log output to save storage: -e REMOVE_LOG_FIELDS: logSource, swarm_node, swarm_task_id, image_name, container_name

An alternative would be dedicated switches to enable/disable log enrichment:

- e ENABLE_SWARM_ENRICHEMENT=true, -e ENABLE_K8S_ENRICHMENT

megastef commented 6 years ago

The feature is ready in sematext/sematext-agent-docker:1.31.53 image. We will release 1.31.53 as latest image today.

Sematext Docker Agent uses logsene-js environment variable LOGSENE_REMOVE_FIELDS.

e.g.

docker run -v /var/run/docker.sock:/var/run/docker.sock \
-e LOGSENE_TOKEN=YOUR_LOGS_TOKEN \
-e LOGSENE_REMOVE_FIELDS="host,ip,logSource,container_host_name,creditCardNumber, password" 
sematext/sematext-agent-docker:dev"

Logs shipped to [Sematext Cloud]() should not contain the fields listed in LOGSENE_REMOVE_FIELDS. Beside the reduced log volume it is a good feature to remove sensitive data fields from logs :)