Inconsistentcy of logs from v3.2+

[iam404]

Some strange issue since upgrade to v3.2 or latest version. We are using logspout with kubernetes daemonset and logstash. Recently we realized logstash was not sending complete logs of container and this issue linked with recent version of logspout. Like if "helloworld" container printed 100 lines of "hello world" then logspout would fetch only 23-40 logs at max.

Luckily downgrading logspout version to v3 seems to have resolved the issue (with >1% error).

Logspout k8 configuration:

      containers:
      - image: gliderlabs/logspout:v3
        name: logspout
        args: ["syslog://localhost:5000"]

        volumeMounts:
          - name: dockersocket
            mountPath: /var/run/docker.sock

[mikelorant]

We are also seeing the same issue. Have just rolled back to version 3.

Be aware that there is an underlying problem with the way Logspout works with Docker to get these logs. Docker creates an inotify watcher and most distributions have this capped by default to 128. This means only 128 containers on a node can be watched. With Kubernetes also creating infrastructure containers, this halves the container count in most cases to 64.

Would recommend bumping your watchers and instances up to more realistic levels.

See the following issue: https://github.com/kubernetes/kops/issues/2912