search

wongnai / kube-slack

Kubernetes Slack Monitoring
MIT License
323 stars 59 forks source link

Logs are not forwarded by fluent-bit by default #51

Closed pierluigilenoci closed 5 years ago

pierluigilenoci commented 5 years ago

Hi, I've installed kube-slack on my k8s cluster.

Helm version

Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"} Server: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}

Installed charts NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE
fluent-bit 6 Wed Jan 30 15:06:16 2019 DEPLOYED fluent-bit-1.5.0 1.0.3 logging
kube-slack 6 Wed Jan 16 15:56:42 2019 DEPLOYED kube-slack-0.4.0 v3.6.0 infrastructure

Basically kube-slack produce this logs: {"name":"kube-slack","hostname":"kube-slack-5fc4b6c55c-chc42","pid":1,"level":30,"msg":"Slack message sent","time":"2019-02-04T16:39:55.107Z","v":0}

This json log have "time" field inside. When docker save on /var/log/containers this log produce this: {"log":"{\"name\":\"kube-slack\",\"hostname\":\"kube-slack-7cf99d5dbd-ffpd7\",\"pid\":1,\"level\":30,\"msg\":\"Slack message sent\",\"time\":\"2019-02-05T13:24:26.193Z\",\"v\":0}\n","stream":"stdout","time":"2019-02-05T13:24:26.193357608Z"}

As you can see this json is not valid because there is the key 'time' twice. For this reason Elasticsearch reject this json forwarded by fluent-bit. Among other things this "time" field inside the kube-slack log is not really needed because docker handle the timestamp on his own with an higher precision.

whs commented 5 years ago

Per #32 the log output is intended to be processed by bunyan.

I think the best we can do here is to add bunyan binary into the container (I believe it's in node_modules/.bin already) and if user have special requirements they can invoke bunyan.

pierluigilenoci commented 5 years ago

I understand your intent but considering that kube-slack is a software for kubernetes and according to CNCF the fluentd is the de facto standard. I suggest making kube-slack compatible with that. In our clusters this "feature" causes a waterfall of useless logs. We were forced to add the annotation fluentbit.io/exclude: true to avoid it.

pierluigilenoci commented 5 years ago

Ref: https://github.com/helm/charts/issues/11000

pierluigilenoci commented 5 years ago

I also created this to have an easy way to add the annotations https://github.com/helm/charts/pull/11149

whs commented 5 years ago

Log changed to winston in cc4f0b9ebc8b8009858ea74e66437a9655e03928