Describe your environment OpentelemetryOperator: 0.99.0 OtelWebserverMoudle: 1.0.4

We are using the nginx instrumentation moudle of otel-webserver-module in our environment. But we found the module will generate multi propogation header(traceparent and tracestate), when request header's count is larger than 20.

Steps to reproduce

install the latest opentelemetry-operator

apply the nginx instrumentation yaml: 00-install-instrumentation.yaml refer: nginx instrumentation testcase

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: nginx
spec:
exporter:
endpoint: http://localhost:4317
propagators:
- jaeger
- b3
sampler:
type: parentbased_traceidratio
argument: "0.25"
nginx:
attrs:
- name: NginxModuleOtelMaxQueueSize
  value: "4096"

the nginx deployment: 01-install-app.yaml


apiVersion: apps/v1
kind: Deployment
metadata:
name: my-nginx
spec:
selector:
matchLabels:
  app: my-nginx
replicas: 1
template:
metadata:
  labels:
    app: my-nginx
  annotations:
    #sidecar.opentelemetry.io/inject: "true"
    #instrumentation.opentelemetry.io/inject-nginx: "true"
    instrumentation.opentelemetry.io/inject-nginx: default/nginx
spec:
  securityContext:
    runAsUser: 1000
    runAsGroup: 3000
    fsGroup: 3000
  containers:
  - name: myapp
    image: nginxinc/nginx-unprivileged:1.25.3
    imagePullPolicy: Always
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop: ["ALL"]
    # following to test lifecycle removal in cloned init container
    ports:
    - containerPort: 8765
    env:
    - name: LD_LIBRARY_PATH
      value: /opt
    volumeMounts:
      - name: nginx-conf
        mountPath: /etc/nginx/nginx.conf
        subPath: nginx.conf
        readOnly: true
    imagePullPolicy: Always
    resources:
      limits:
        cpu: "1"
        memory: 500Mi
      requests:
        cpu: 250m
        memory: 100Mi
  volumes:
  - name: nginx-conf
    configMap:
      name: nginx-conf
      items:
        - key: nginx.conf
          path: nginx.conf

apiVersion: v1 kind: ConfigMap metadata: name: nginx-conf data: nginx.conf: |

user nginx;

worker_processes  1;
pid /tmp/nginx.pid;
events {
  worker_connections  10240;
}
http {
  include /etc/nginx/conf.d/*.conf;
  client_body_temp_path /tmp/client_temp;
  proxy_temp_path       /tmp/proxy_temp_path;
  fastcgi_temp_path     /tmp/fastcgi_temp;
  uwsgi_temp_path       /tmp/uwsgi_temp;
  scgi_temp_path        /tmp/scgi_temp;
  server {
    listen       8765;
    server_name  localhost;
    location / {
      root   /usr/share/nginx/html;
      index  index.html index.htm;
    }

    location /api/ {
      proxy_pass http://echo.default:80/api/;
    }
  }
}

4. the upstream echo server: 
```yaml
➜  kong cat echo-service.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    app: echo
  name: echo
spec:
  ports:
  - port: 8080
    name: high
    protocol: TCP
    targetPort: 8080
  - port: 80
    name: low
    protocol: TCP
    targetPort: 8080
  selector:
    app: echo
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: echo
  name: echo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: echo
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: echo
    spec:
      containers:
      - image: gcr.io/kubernetes-e2e-test-images/echoserver:2.2
        name: echo
        ports:
        - containerPort: 8080
        env:
          - name: NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
        resources: {}

test:

with over than 20 headers, key1 ~ key16 + 4 header(auto added)

curl 'localhost:8765/api/hello' \
--header 'key1: value1' \
--header 'key2: value2' \
--header 'key3: value3' \
--header 'key4: value4' \
--header 'key5: value5' \
--header 'key6: value6' \
--header 'key7: value7' \
--header 'key8: value8' \
--header 'key9: value9' \
--header 'key10: value10' \
--header 'key11: value11' \
--header 'key12: value12' \
--header 'key13: value13' \
--header 'key14: value14' \
--header 'key15: value15' \
--header 'key16: value16'

What is the expected behavior? The request send to echo server with only one propogation header

What is the actual behavior? The request send to echo server with two propogation header:


Hostname: echo-67456bbd77-lb9vc

Pod Information:
        node name:      minikube
        pod name:       echo-67456bbd77-lb9vc
        pod namespace:  default
        pod IP: 10.244.0.34

Server values:
        server_version=nginx: 1.12.2 - lua: 10010

Request Information:
        client_address=10.244.0.37
        method=GET
        real path=/api/hello
        query=
        request_version=1
        request_scheme=http
        request_uri=http://echo.default:8080/api/hello

Request Headers:
        accept=*/*
        connection=close
        host=echo.default
        key1=value1
        key10=value10
        key11=value11
        key12=value12
        key13=value13
        key14=value14
        key15=value15
        key16=value16
        key2=value2
        key3=value3
        key4=value4
        key5=value5
        key6=value6
        key7=value7
        key8=value8
        key9=value9
        traceparent=00-59bf9c6263e207f0e0396385591bdcd5-833345068aad29be-01
        traceparent=00-59bf9c6263e207f0e0396385591bdcd5-474530dbae9782b7-01
        tracestate=
        user-agent=curl/7.81.0

Request Body:
        -no body in request-

Additional context Please review my case, thanks! I have found the root cause which is related to header iteration of ngx_list_t and would be very like to make a PR

open-telemetry / opentelemetry-cpp-contrib

Nginx instrumentation generate multi propogation header when request header count >= 20 #448

user nginx;