kubernetes.container_image and kubernetes.container_id labels not available for init containers in kubernetes_logs source

SHWETHABHAT1 commented 1 year ago

A note for the community

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Currently for init containers we are seeing the container_image and container_image_id labels missing. One such example is from the canal pod. The same is also seen in the init-container of application pod as well. We see the image-name of the container of the canal pod in its config.

{"file":"/var/log/pods/kube-system_canal-<pod_name>_*****************/mount-bpffs/0.log","kubernetes":{"container_name":"mount-bpffs","namespace_labels":{"field.cattle.io/projectId":"#####","kubernetes.io/metadata.name":"kube-system"},"node_labels":{"beta.kubernetes.io/arch":"####","beta.kubernetes.io/instance-type":"####","beta.kubernetes.io/os":"###","failure-domain.beta.kubernetes.io/region":"####","failure-domain.beta.kubernetes.io/zone":"###","kubernetes.io/arch":"###","kubernetes.io/hostname":"node-name","kubernetes.io/os":"###","pod_ip":"###","pod_ips":["###"],"pod_labels":{"controller-revision-hash":"####","k8s-app":"canal","pod-template-generation":"1"},"pod_name":"canal-###","pod_namespace":"kube-system","pod_node_name":"node-name","pod_owner":"DaemonSet/canal","pod_uid":"####"},"message":"<data>.","source_type":"kubernetes_logs","stream":"stdout","timestamp_end":"time data"}

Configuration

Name:         vector
Namespace:    logging
Labels:       app.kubernetes.io/component=Agent
              app.kubernetes.io/instance=vector
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=vector
              app.kubernetes.io/version=0.30.0-distroless-libc
              helm.sh/chart=vector-0.22.1
Annotations:  meta.helm.sh/release-name: vector
              meta.helm.sh/release-namespace: logging

Data
====
vector.yaml:
----
data_dir: /vector-data-dir
sinks:
  loki-stdout:
    encoding:
      codec: json
    endpoint: <loki-endpoint>
    inputs:
    - stdout-logs
    labels:
      file: '{{ file }}'
      k8s_container_id: '{{ kubernetes.container_id }}'
      k8s_container_image: '{{ kubernetes.container_image }}'
      k8s_container_name: '{{ kubernetes.container_name }}'
      k8s_node_name: '{{ kubernetes.pod_node_name }}'
      k8s_node_os: '{{ kubernetes.node_labels."kubernetes.io/os" }}'
      k8s_pod_ip: '{{ kubernetes.pod_ip }}'
      k8s_pod_name: '{{ kubernetes.pod_name }}'
      k8s_pod_namespace: '{{ kubernetes.pod_namespace }}'
      k8s_pod_uid: '{{ kubernetes.pod_uid }}'
      source_type: '{{ source_type }}'
      stream: '{{ stream }}'
    out_of_order_action: accept
    tls:
      verify_certificate: false
      verify_hostname: false
    type: loki
sources:
  stdout-logs:
    type: kubernetes_logs

BinaryData
====

Events:  <none>

Version

0.30.0

Debug Output

ERROR sink{component_kind="sink" component_id=loki-testing component_type=loki component_name=loki-testing}: vector::     internal_events::template: Internal log [Failed to render template for "label_value "{{ kubernetes.container_image }}" with label_key "k8s_container_image"".] has been suppressed 7710 times.

ERROR sink{component_kind="sink" component_id=loki-testing component_type=loki component_name=loki-testing}: vector::     internal_events::template: Failed to render template for "label_value "{{ kubernetes.container_id }}" with label_key "k8s_container_id"". error     =Missing fields on event: ["kubernetes.container_id"] error_type="template_failed" stage="processing" internal_log_rate_limit=true

Example Data

No response

Additional Context

No response

References

https://github.com/vectordotdev/vector/issues/18627

SHWETHABHAT1 commented 1 year ago

@jszwedko Hi, were you able to reproduce this issue ? Is this being tracked somewhere ?

jszwedko commented 1 year ago

@jszwedko Hi, were you able to reproduce this issue ? Is this being tracked somewhere ?

I unfortunately haven't been able to reproduce this issue yet, but thank you for opening it!

kefiras commented 11 months ago

I am seeing the same for some of the init containers

vectordotdev / vector