Reduce Docker API calls for log collection, when container logs got disabled

[megastef]

The agent should not query logs from Docker API for containers having LOGSENE_ENABLED flag set to false (either by LOGSENE_ENABLED_DEFAULT=false or explicit set of LOGSENE_ENABLED=false per container).

Origin: https://github.com/sematext/sematext-agent-docker/issues/21#issuecomment-292934284

[megastef]

We will try to call container.detach() https://github.com/mcollina/docker-loghose/blob/master/lib/loghose.js#L26 in Docker Loghose to stop the log stream once LOGSENE_ENABLED=false got detected (after docker inspect call ...).

[szakasz]

Ok, thanks for looking into this.

[ReToCode]

As I debugged the other issue, I saw that all the containers get listed by docker-allcontainers and then attached by loghose. And sematext-docker-agent via logsense-js just reacts to the attached event.

So quite a lot of containers get attached and later detached. Is there a reason why you wouldn't filter in the docker-allcontainers library? Maybe add new property like matchByEnv or something like it?

[megastef]

Use v1.31.26 to reduce api calls...

[szakasz]

Thanks, I have installed it, seems to be fine, though I didn't notice any significant changes in Docker performance - still even a few percent counts. :+1: One more minor question: We now have set LOGSENE_ENABLED_DEFAULT=false which effectively turns out the log shipping for the sematext-agent self. I did want to re-enable it by specifying LOGSENE_ENABLED=true as an env var on the sematext-agent pods/containers, but somehow the sematext-agent doesn't get it: after startup when it reaches out for the own container thru Docker API it still thinks LOGSENE_ENABLED=false somehow. Might there be some conflict there, or do I maybe something wrong?

[megastef]

Did you use SKIP_BY_IMAGE? We set it by default to /sematext-docker-agent/, but it might get overwritten when customers set SKIP_BY_IMAGE.

We should set LOGSENE_ENABLED=false for sematext-agent-docker container - so there would be no conflict with SKIP_BY_IMAGE. Build for fix in :dev image is running. Please try sematext/sematext-agent-docker:dev once build is done: https://hub.docker.com/r/sematext/sematext-agent-docker/builds/

[szakasz]

Thanks @megastef, but still no luck with the new :dev image. I had a look at your recent code change, and so I am not really sure that I was clear enough.

So what we would like to achieve is that each of our sematext-agents grabs their own log via docker API and sends that to Logsene. LOGSENE_ENABLED_DEFAULT=false and LOGSENE_ENABLED=true is set for the sematext-agent pods, I also tried to set SKIP_BY_IMAGE to some dummy value. Still I get the following when running oc logs sematext-agent-2iyhf: ... Container 1ecdd7ffc07ef121b00a12cce630bf42c71281ce4bddf0479909ca2901ef226b /k8s_sematext-agent.8a332beb_sematext-agent-2iyhf_logging_236c8b28-30c0-11e7-b4cc-0295f8e72b3b_27626ae1 setting LOGSENE_ENABLED=false ... Following env vars are set:

oc env pod/sematext-agent-2iyhf --list
# pods sematext-agent-2iyhf, container sematext-agent
LOGSENE_TOKEN=d0a82a77-b218-4788-ba60-659ba7477ca3
KUBERNETES=1
PATTERNS_URL=https://code.sbb.ch/projects/KD_CLOUD/repos/sematext/browse/patterns.yml?raw
MAX_CLIENT_SOCKETS=2
ENABLE_LOGSENE_STATS=true
LOGSENE_ENABLED_DEFAULT=false
LOGSENE_ENABLED=true
SKIP_BY_IMAGE=nothing

[megastef]

Ahh, sorry the :dev image disabled agent logging to Logsene completely ... going to revert the change.

Have you tried this config with :latest ? I think it should work and would log to LOGSENE_TOKEN. It works for me - getting SDA logs in Logsene using:

LOGSENE_ENABLED=true
SKIP_BY_IMAGE=nothing

We should remove the default for SKIP_BY_IMAGE and set in SDA start script LOGSENE_ENABLED=false, so you could overwrite it with true, as you did. Then you would not need SKIP_BY_IMAGE=nothing

[szakasz]

Still no luck. Have you also specified LOGSENE_ENABLED_DEFAULT=false? I will otherwise have a look into this with @ReToCode next week.

[megastef]

Yes. Here is the env from my docker-compose file (without token):

sematext-agent-docker:
  image: 'sematext/sematext-agent-docker:latest'
  environment:
  - LOGSENE_TOKEN=44a966f0...
  - LOGSENE_ENABLED_DEFAULT=false
  - LOGSENE_ENABLED=true
  - SKIP_BY_IMAGE=nothing
  restart: always
  volumes:
  - '/var/run/docker.sock:/var/run/docker.sock'

Logsene screenshot with SDA logs:

[ReToCode]

I have found the root cause here. The problem is this line here: https://github.com/sematext/sematext-agent-docker/blob/master/lib/dockerInspect.js#L43

.indexOf("LOGSENE_ENABLED")  is true for "LOGSENE_ENABLED_DEFAULT" which is first in the list

I changed it to this, then it worked:

     var rv = list[i].split('=')
      if (rv[0] == name) {

[megastef]

An application container should not have an Environment variable LOGSENE_ENABLED_DEFAULT, only the Sematext Docker Agent itself. Application containers get tagged with LOGSENE_ENABLED=true/false to overwrite the default used by SDA only. I wrote a blog post about LOGSENE_ENABLED_DEFAULT usage: https://sematext.com/blog/2017/05/15/docker-log-management-enrichment/

Anyhow, the problem of "fuzzy" match of env variable names needs to be fixed. Thank you for reporting!

[ReToCode]

It's clear for other containers. But we want the logs of the SDA container too, there we have the DEFAULT=false but also LOGSENE_ENABLED=true which then results to false.