Open vladarts opened 3 years ago
Also, most of such cases happens on Self-managed nodes not Managed node groups but I can not imagine why.
You mean some of the logs from a container do not have metadata? Like the earliest part of the logs- the first logs from the container?
I think that is normal and expected and a bit random (if that is what is happening). Fluent Bit has to get the metadata from the API Server- and that can take time I think so initially logs might not have metadata. That's one hypothesis.
Also- please enable debug logging for Fluent Bit. This came up in another issue recently- the kubernetes filter prints some key information about errors to debug- some sorts of failures can only be seen with debug logging. I created an issue upstream for fixing this: https://github.com/fluent/fluent-bit/issues/2934
Hi! I tried with debug log level for fluent-bit and I did not found any error-like messages - I see blocks like:
[2021/01/19 12:50:26] [debug] [http_client] header=GET /api/v1/namespaces/gitlab-runner-jobs-team/pods/runner-e6wuacxt-project-1337-concurrent-2tkl6w HTTP/1.1
Host: kubernetes.default.svc
Content-Length: 0
User-Agent: Fluent-Bit
Connection: close
Authorization: Bearer <TOKEN>
[2021/01/19 12:50:26] [debug] [filter:kubernetes:kubernetes.0] API Server (ns=gitlab-runner-jobs-team, pod=runner-e6wuacxt-project-1337-concurrent-2tkl6w) http_do=0, HTTP Status: 200
but all logs from such pods are not enriched with kubernetes meta. Such pods lifetime can be up to 1hour and they produce logs constantly.
@xxxbobrxxx Sorry I missed your reply- can you open an issue on the upstream repo for this? I am not sure what is going wrong here.
Hi @xxxbobrxxx, could you please upgrade to the latest image and see if you still have this issue? Thanks.
Hi! I have an issue with kubernetes data enrichment - it works only for part of logs. Example: fluent-bit image itself produces the following logs:
when the following part will be not enriched:
and similar issue happens on other pods. I did not recognize any regularity of bug appearance.
Checked on aws-for-fluent-bit versions:
And helm charts:
Thanks!