Describe the bug
Setting sumologic.logs.container.perContainerAnnotationsEnabled to true does not work in v4.2.0, but worked in v2.9.1 when on fluentd/fluentbit.
The cause of this is the source/containers processor relies on k8s.container.name attribute when sumologic.logs.container.perContainerAnnotationsEnabled is true, but this attribute is not present, as it has been stripped off by the previous processor, sumologic_schema.
Logs
See Anything else do we need to know section below.
Command used to install/upgrade Collection
Using Terraform helm_release resource. Not relevant to the bug.
(probably can set it up to setup the sources and still reproduce this)
To Reproduce
Provision helm chart with above config
Provision a pod with two containers, a and b, with the annotations:
"sumologic.com/a.sourceCategory": "foo/a""sumologic.com/b.sourceCategory": "foo/b"
Generate logs within these containers
Expected behavior
Querying Sumo Logic for _sourceCategory=foo/a shows logs from container a and querying for _sourceCategory=foo/b shows logs from container b
Actual behavior
Logs are not present when querying for _sourceCategory=foo/aor _sourceCategory=foo/b, and are in fact in _sourceCategory=foo/bar`
Environment (please complete the following information):
Collection version (e.g. helm ls -n sumologic): 4.2.0
Kubernetes version: v1.24.17-eks-4f4795d
Cloud provider: AWS
Anything else do we need to know
When debug logging is enabled by adding the following to the Helm values:
Observe k8s.container.name is not present. Repeat the above steps, except additionally remove the sumologic_schema processor
Observe k8s.container.name is present
Workaround
We can workaround this by setting the config container_annotations.container_name_key config on the source/containers processor to be the attribute key that sumologic_schema creates. Add the following to the Helm chart values:
the sumologic_schema processor should keep the k8s.container.name attribute around for subsequent processing
the source processor should default to container instead of k8s.container.name
[recommended] the config generated from the Helm chart values for the source processorshould setcontainer_annotations.container_name_key: containeras it is aware that thesumologic_schemaprocessor executes beforesource/containers`.
Describe the bug Setting
sumologic.logs.container.perContainerAnnotationsEnabled
totrue
does not work in v4.2.0, but worked in v2.9.1 when on fluentd/fluentbit.The cause of this is the
source/containers
processor relies onk8s.container.name
attribute whensumologic.logs.container.perContainerAnnotationsEnabled
is true, but this attribute is not present, as it has been stripped off by the previous processor,sumologic_schema
.Logs See Anything else do we need to know section below.
Command used to install/upgrade Collection Using Terraform
helm_release
resource. Not relevant to the bug.Configuration
(probably can set it up to setup the sources and still reproduce this)
To Reproduce
a
andb
, with the annotations:"sumologic.com/a.sourceCategory": "foo/a"
"sumologic.com/b.sourceCategory": "foo/b"
Expected behavior Querying Sumo Logic for
_sourceCategory=foo/a
shows logs from containera
and querying for_sourceCategory=foo/b
shows logs from containerb
Actual behavior Logs are not present when querying for _sourceCategory=foo/a
or _sourceCategory=foo/b
, and are in fact in _sourceCategory=foo/bar`Environment (please complete the following information):
helm ls -n sumologic
): 4.2.0Anything else do we need to know When debug logging is enabled by adding the following to the Helm values:
and we watch the logs of the log forwarder:
we can see that this is printed a bunch:
This log originates from the source processor.
To check the attributes available to the source processor, do the following:
kubectl -n $NAMESPACE edit configmap/sumologic-sumologic-otelcol-logs
source/containers
kubectl -n $NAMESPACE delete po -l app=sumologic-sumologic-otelcol-logs
kubectl -n $NAMESPACE logs -f -l app=sumologic-sumologic-otelcol-logs --max-log-requests 10 | grep "$POD_ID_OF_LOG_SOURCE" -B 20 -A 20
k8s.container.name
is not present. Repeat the above steps, except additionally remove thesumologic_schema
processork8s.container.name
is presentWorkaround We can workaround this by setting the config
container_annotations.container_name_key
config on thesource/containers
processor to be the attribute key thatsumologic_schema
creates. Add the following to the Helm chart values:Potential fixes
sumologic_schema
processor should keep thek8s.container.name
attribute around for subsequent processingsource
processor should default tocontainer
instead ofk8s.container.name
source
processorshould set
container_annotations.container_name_key: containeras it is aware that the
sumologic_schemaprocessor executes before
source/containers`.