Opentracing for nginx plus does not work

[myf5]

Describe the bug The opentracing setting is updated to nginx.conf in the ingress controller after creating config map with "kubernetes-ingress/examples/opentracing/nginx-config.yaml".

The configuration is:

opentracing on; opentracing_load_tracer /usr/local/lib/libjaegertracing_plugin.so /etc/tracer-config.json;

But the nginx can not reload the conifg, got coredump.

172.16.20.199 - - [29/Aug/2019:15:51:56 +0000] "GET /tea HTTP/1.1" 200 154 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36" "-"
I0829 16:12:35.686104       1 event.go:209] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"cafe-ingress", UID:"369db247-ca71-11e9-8f84-005056b34685", APIVersion:"extensions/v1beta1", ResourceVersion:"34205110", FieldPath:""}): type: 'Warning' reason: 'UpdatedWithError' Configuration for default/cafe-ingress was updated but was not applied: Error when updating config from ConfigMap: nginx reload failed: Command /usr/sbin/nginx -s reload stdout: ""
stderr: "Segmentation fault (core dumped)\n"
finished with error: exit status 139
I0829 16:12:35.686127       1 event.go:209] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"nginx-ingress", Name:"nginx-config", UID:"b853b442-ca6f-11e9-8f84-005056b34685", APIVersion:"v1", ResourceVersion:"34208574", FieldPath:""}): type: 'Warning' reason: 'UpdatedWithError' Configuration from nginx-ingress/nginx-config was updated but was not applied: Error when updating config from ConfigMap: nginx reload failed: Command /usr/sbin/nginx -s reload stdout: ""
stderr: "Segmentation fault (core dumped)\

The nginx will reload successfully if I manually comment the opentracing setting from the nginx.conf file:

   # opentracing on;
   # opentracing_load_tracer /usr/local/lib/libjaegertracing_plugin.so /etc/tracer-config.json;

To Reproduce Steps to reproduce the behavior:

1. Build an image with： kubernetes-ingress/build/DockerfileWithOpentracingForPlus
2. Create common cafe ingress
3. Deploy daemonset nginx controller
4. verify it is ok that accessing cafe application Then:
5. Install all in one jaeger (change jaeger-query service type to nodePort, so I can access it from external PC)
6. Verify jaeger work well.
7. Apply the config map "kubernetes-ingress/examples/opentracing/nginx-config.yaml"
8. Check the Ingress controller pod's log

Expected behavior A clear and concise description of what you expected to happen.

Your environment

• Version of the Ingress Controller - release version or a specific commit --master. There is no DockerfileWithOpentracingForPlus in build directory if checkout 1.5.4
• Version of Kubernetes 1.10
• Kubernetes platform (e.g. Mini-kube or GCP) Local K8S in vm
• Using NGINX or NGINX Plus NGINX PLUS

Additional context Add any other context about the problem here. Any log files you want to share.

[myf5]

# nginx -V    
nginx version: nginx/1.15.10 (nginx-plus-r18-p1)
built by gcc 6.3.0 20170516 (Debian 6.3.0-18+deb9u1) 
built with OpenSSL 1.1.0j  20 Nov 2018 (running with OpenSSL 1.1.0k  28 May 2019)
TLS SNI support enabled
configure arguments: --build=nginx-plus-r18-p1 --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_jwt_module --with-http_auth_request_module --with-http_dav_module --with-http_f4f_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_hls_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_session_log_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g -O2 -fdebug-prefix-map=/data/builder/debuild/nginx-plus-1.15.10/debian/debuild-base/nginx-plus-1.15.10=. -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie'

[Rulox]

Hi @myf5, thanks for posting the issue.

I have just tried and is working on my end (same version of NGINX Plus and IC than you). In order to understand what might be happening:

1. Are you using any other NGINX Plus module apart from the Opentracing one?
2. Are you using any location or server snippets and/or custom annotations?

Could you post your /etc/nginx/nginx.conf and /etc/nginx/conf.d/... files?

Could you also post the first lines of the logs of the IC to check the version? kubectl logs <IC_POD , something like I0830 13:58:16.341688 1 main.go:155] Starting NGINX Ingress controller Version=edge GitCommit=15c945a8

It seems that the configuration is correct for what you posted but something is causing a segfault in your NGINX Plus instance. This would be more an issue with the module, not with the Ingress Controller itself.

[myf5]

@Rulox Thanks for the help.

1. No other module. I build the image by default file with except of "ADD default.pem /etc/nginx/secrets/default": make PREFIX=myf5/nginx-plus-ingress-opentracing DOCKERFILE=DockerfileWithOpentracingForPlus GENERATE_DEFAULT_CERT_AND_KEY=1

2. No special annotations, here is the cafe-ingress.yaml: Note: the below annotations was added after the above test. With or without the annotation, get same failure.

   [root@k8s-master complete-example]# cat cafe-ingress.yaml 
   apiVersion: extensions/v1beta1
   kind: Ingress
   metadata:
   name: cafe-ingress
   annotations:
   kubernetes.io/ingress.class: "canary"
   spec:
   tls:
   - hosts:
   - cafe.lab.f5se.io
   secretName: cafe-secret
   rules:
   - host: cafe.lab.f5se.io
   http:
     paths:
     - path: /tea
       backend:
         serviceName: tea-svc
         servicePort: 80
     - path: /coffee
       backend:
         serviceName: coffee-svc
         servicePort: 80

I am posting my nginx.conf in below comments.

[myf5]

Here are the files when I am not creating the configmap "kubernetes-ingress/examples/opentracing/nginx-config.yaml" So currently the IC is running with the default nginx-config configmap, which the data part is blank.

Go into the container:

# cat /etc/nginx/nginx.conf

user  nginx;
worker_processes  auto;

daemon off;

error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout 65s;
    keepalive_requests 100;

    #gzip  on;

    server_names_hash_max_size 512;

    variables_hash_bucket_size 256;
    variables_hash_max_size 1024;

    map $http_upgrade $connection_upgrade {
        default upgrade;
        ''      close;
    }
    map $http_upgrade $vs_connection_header {
        default upgrade;
        ''      $default_connection_header;
    }

    server {
        # required to support the Websocket protocol in VirtualServer/VirtualServerRoutes
        set $default_connection_header "";

        listen 80 default_server;
        listen 443 ssl default_server;

        ssl_certificate /etc/nginx/secrets/default;
        ssl_certificate_key /etc/nginx/secrets/default;

        server_name _;
        server_tokens "on";
        access_log off;

        location / {
           return 404;
        }
    }
    # NGINX Plus APIs
    server {
        listen 8888;

        root /usr/share/nginx/html;

        access_log off;

        location  = /dashboard.html {
        }

        allow 172.16.0.0/16;

        deny all;
        location /api {
            api write=off;
        }
    }

    # NGINX Plus API over unix socket
    server {
        listen unix:/var/run/nginx-plus-api.sock;
        access_log off;

        # $config_version_mismatch is defined in /etc/nginx/config-version.conf
        location /configVersionCheck {
            if ($config_version_mismatch) {
                return 503;
            }
            return 200;
        }

        location /api {
            api write=on;
        }
    }

    include /etc/nginx/config-version.conf;
    include /etc/nginx/conf.d/*.conf;
}

stream {
    log_format  stream-main  '$remote_addr [$time_local] '
                      '$protocol $status $bytes_sent $bytes_received '
                      '$session_time';

    access_log  /var/log/nginx/stream-access.log  stream-main;

}

# ls /etc/nginx/conf.d/
default-cafe-ingress.conf
# cat /etc/nginx/conf.d/default-cafe-ingress.conf
# configuration for default/cafe-ingress

upstream default-cafe-ingress-cafe.lab.f5se.io-coffee-svc-80 {
        zone default-cafe-ingress-cafe.lab.f5se.io-coffee-svc-80 256k;
        random two least_conn;

        server 10.244.0.111:80 max_fails=1 fail_timeout=10s max_conns=0;

}
upstream default-cafe-ingress-cafe.lab.f5se.io-tea-svc-80 {
        zone default-cafe-ingress-cafe.lab.f5se.io-tea-svc-80 256k;
        random two least_conn;

        server 10.244.0.112:80 max_fails=1 fail_timeout=10s max_conns=0;

}

server {

        listen 80;

        listen 443 ssl;
        ssl_certificate /etc/nginx/secrets/default-cafe-secret;
        ssl_certificate_key /etc/nginx/secrets/default-cafe-secret;

        server_tokens "on";

        server_name cafe.lab.f5se.io;

        status_zone cafe.lab.f5se.io;

        if ($scheme = http) {
                return 301 https://$host:443$request_uri;
        }

        location /tea {

                proxy_http_version 1.1;

                proxy_connect_timeout 60s;
                proxy_read_timeout 60s;
                proxy_send_timeout 60s;
                client_max_body_size 1m;
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Host $host;
                proxy_set_header X-Forwarded-Port $server_port;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_buffering on;

                proxy_pass http://default-cafe-ingress-cafe.lab.f5se.io-tea-svc-80;

        }
        location /coffee {

                proxy_http_version 1.1;

                proxy_connect_timeout 60s;
                proxy_read_timeout 60s;
                proxy_send_timeout 60s;
                client_max_body_size 1m;
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Host $host;
                proxy_set_header X-Forwarded-Port $server_port;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_buffering on;

                proxy_pass http://default-cafe-ingress-cafe.lab.f5se.io-coffee-svc-80;

        }

}

[myf5]

Now, create the configmap with "kubernetes-ingress/examples/opentracing/nginx-config.yaml" The IC will coredump.

The first line log:

[root@k8s-master opentracing]# kubectl logs nginx-ingress-g9tjd -n nginx-ingress 
I0830 07:57:56.417549       1 main.go:155] Starting NGINX Ingress controller Version=edge GitCommit=924726b

The last lines log:

I0831 09:33:15.417048       1 event.go:209] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"cafe-ingress", UID:"369db247-ca71-11e9-8f84-005056b34685", APIVersion:"extensions/v1beta1", ResourceVersion:"34275299", FieldPath:""}): type: 'Warning' reason: 'UpdatedWithError' Configuration for default/cafe-ingress was updated but was not applied: Error when updating config from ConfigMap: nginx reload failed: Command /usr/sbin/nginx -s reload stdout: ""
stderr: "Segmentation fault (core dumped)\n"
finished with error: exit status 139
I0831 09:33:15.417073       1 event.go:209] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"nginx-ingress", Name:"nginx-config", UID:"b853b442-ca6f-11e9-8f84-005056b34685", APIVersion:"v1", ResourceVersion:"34389127", FieldPath:""}): type: 'Warning' reason: 'UpdatedWithError' Configuration from nginx-ingress/nginx-config was updated but was not applied: Error when updating config from ConfigMap: nginx reload failed: Command /usr/sbin/nginx -s reload stdout: ""
stderr: "Segmentation fault (core dumped)\n"
finished with error: exit status 139

Related files now: NOTE: Why there is " opentracing off;" in the default server definition block?

# cat /etc/nginx/nginx.conf

user  nginx;
worker_processes  auto;

daemon off;

error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;
load_module modules/ngx_http_opentracing_module.so;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout 65s;
    keepalive_requests 100;

    #gzip  on;

    server_names_hash_max_size 512;

    variables_hash_bucket_size 256;
    variables_hash_max_size 1024;

    map $http_upgrade $connection_upgrade {
        default upgrade;
        ''      close;
    }
    map $http_upgrade $vs_connection_header {
        default upgrade;
        ''      $default_connection_header;
    }

    opentracing on;

    opentracing_load_tracer /usr/local/lib/libjaegertracing_plugin.so /etc/tracer-config.json;

    server {
        # required to support the Websocket protocol in VirtualServer/VirtualServerRoutes
        set $default_connection_header "";

        listen 80 default_server;
        listen 443 ssl default_server;

        ssl_certificate /etc/nginx/secrets/default;
        ssl_certificate_key /etc/nginx/secrets/default;

        server_name _;
        server_tokens "on";
        access_log off;

        opentracing off;

        location / {
           return 404;
        }
    }
    # NGINX Plus APIs
    server {
        listen 8888;

        root /usr/share/nginx/html;

        access_log off;

        opentracing off;

        location  = /dashboard.html {
        }

        allow 172.16.0.0/16;

        deny all;
        location /api {
            api write=off;
        }
    }

    # NGINX Plus API over unix socket
    server {
        listen unix:/var/run/nginx-plus-api.sock;
        access_log off;

        opentracing off;

        # $config_version_mismatch is defined in /etc/nginx/config-version.conf
        location /configVersionCheck {
            if ($config_version_mismatch) {
                return 503;
            }
            return 200;
        }

        location /api {
            api write=on;
        }
    }

    include /etc/nginx/config-version.conf;
    include /etc/nginx/conf.d/*.conf;
}

stream {
    log_format  stream-main  '$remote_addr [$time_local] '
                      '$protocol $status $bytes_sent $bytes_received '
                      '$session_time';

    access_log  /var/log/nginx/stream-access.log  stream-main;

}

# cat /etc/nginx/conf.d/default-cafe-ingress.conf
# configuration for default/cafe-ingress

upstream default-cafe-ingress-cafe.lab.f5se.io-coffee-svc-80 {
        zone default-cafe-ingress-cafe.lab.f5se.io-coffee-svc-80 256k;
        random two least_conn;

        server 10.244.0.111:80 max_fails=1 fail_timeout=10s max_conns=0;

}
upstream default-cafe-ingress-cafe.lab.f5se.io-tea-svc-80 {
        zone default-cafe-ingress-cafe.lab.f5se.io-tea-svc-80 256k;
        random two least_conn;

        server 10.244.0.112:80 max_fails=1 fail_timeout=10s max_conns=0;

}

server {

        listen 80;

        listen 443 ssl;
        ssl_certificate /etc/nginx/secrets/default-cafe-secret;
        ssl_certificate_key /etc/nginx/secrets/default-cafe-secret;

        server_tokens "on";

        server_name cafe.lab.f5se.io;

        status_zone cafe.lab.f5se.io;

        if ($scheme = http) {
                return 301 https://$host:443$request_uri;
        }

        location /tea {

                proxy_http_version 1.1;

                proxy_connect_timeout 60s;
                proxy_read_timeout 60s;
                proxy_send_timeout 60s;
                client_max_body_size 1m;
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Host $host;
                proxy_set_header X-Forwarded-Port $server_port;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_buffering on;

                proxy_pass http://default-cafe-ingress-cafe.lab.f5se.io-tea-svc-80;

        }
        location /coffee {

                proxy_http_version 1.1;

                proxy_connect_timeout 60s;
                proxy_read_timeout 60s;
                proxy_send_timeout 60s;
                client_max_body_size 1m;
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Host $host;
                proxy_set_header X-Forwarded-Port $server_port;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_buffering on;

                proxy_pass http://default-cafe-ingress-cafe.lab.f5se.io-coffee-svc-80;

        }

}

[Rulox]

Hi @myf5 thanks for all the data.

I have used exactly the same version and config as you and it is working perfectly on my end, I can't reproduce your error (your pasted configs seem ok too). The only difference is the version of k8s (I am using 1.12). 1.10 is a really old version. I would recommend you to update at least to 1.11 as well anyways.

Also could you provide the list of arguments and/org configmaps you are using? I see you are using for example NginxStatusAllowCIDRs among others, that's the only difference we have in configs.

Another check that you could perform is try to re-build the IC image (make clean && make build...), just in case something went wrong the first time (unlikely)

I'll run the issue with the rest of the team to see if others can reproduce. Will update the issue with any news. Sorry for the inconvenience.

Re: NOTE: Why there is " opentracing off;" in the default server definition block?

This is intended. We decided to disable opentracing in all server/locations that weren't configured by the user, including the default server.

[pleshakov]

@myf5 Thanks for the extensive info. As @Rulox mentioned, we haven't been able to reproduce the issue. Could you send us some additional information below?

Could you possibly run the following command from the IC container: $ dpkg -l | grep nginx?

Could you possibly try to reproduce the bug once again but this time enabling the NGINX debug log? To enable the debug log:

• Enable -nginx-debug cli argument of the IC
• Set error-log-level: "debug" in the ConfigMap
• Change the template and rebuild the image (or attach the template via a ConfigMap) load_module modules/ngx_http_opentracing_module.so; -> load_module modules/ngx_http_opentracing_module-debug.so; so that debug logging is enabled in the OT module as well.

Once the debug logging is enabled, could you reproduce an issue and share the debug log related to the core dump?

If during the building of an image you encounter an error The following packages have unmet dependencies: nginx-plus-module-opentracing : Depends: nginx-plus-r19, please update the Dockerfile and specify the version of the OT module: nginx-plus-module-opentracing=18+0.8.0-1~stretch

Thanks

[myf5]

@Rulox Sorry for the late response as PTO. Here is the arguments of the controller:

[root@k8s-master daemon-set]# cat nginx-plus-ingress.yaml 
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nginx-ingress-apidmeo
  namespace: nginx-ingress
spec:
  selector:
    matchLabels:
      app: nginx-ingress-apidemo
  template:
    metadata:
      labels:
        app: nginx-ingress-apidemo
     #annotations:
       #prometheus.io/scrape: "true"
       #prometheus.io/port: "9113"
    spec:
      serviceAccountName: nginx-ingress
      containers:
      - image: myf5/nginx-plus-ingress-opentracing:edge
        imagePullPolicy: IfNotPresent
        name: nginx-plus-ingress-apidemo
        ports:
        - name: http
          containerPort: 80
          hostPort: 80
        - name: https
          containerPort: 443
          hostPort: 443
        - name: apiport
          containerPort: 8889
          hostPort: 8889

       #- name: prometheus
         #containerPort: 9113
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        args:
          - -nginx-plus
          - -nginx-configmaps=$(POD_NAMESPACE)/nginx-config
          - -default-server-tls-secret=$(POD_NAMESPACE)/default-server-secret
          - -nginx-status
          - -nginx-status-allow-cidrs=172.16.0.0/16
          - -nginx-status-port=8889
         # - -ingress-class=canary
         #- -v=3 # Enables extensive logging. Useful for troubleshooting.
         #- -report-ingress-status
         #- -external-service=nginx-ingress
         #- -enable-leader-election
         #- -enable-prometheus-metrics
         #- -enable-custom-resources

For configmap, I am using the same to the exmaples, which is:

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-config
  namespace: nginx-ingress
data:
  opentracing: "True"
  opentracing-tracer: "/usr/local/lib/libjaegertracing_plugin.so"
  opentracing-tracer-config: |
        {
            "service_name": "nginx-ingress",
            "sampler": {
                "type": "const",
                "param": 1
            },
            "reporter": {
                "localAgentHostPort": "jaeger-agent.default.svc.cluster.local:6831"
            }
        }

[myf5]

@pleshakov

# dpkg -l | grep nginx
ii  nginx-plus                    18-2~stretch                   amd64        NGINX Plus, provided by Nginx, Inc.
ii  nginx-plus-module-opentracing 18+0.8.0-1~stretch             amd64        NGINX Plus, provided by Nginx, Inc. (3rd-party OpenTracing dynamic module)

I will try to debug it follow your instructions

[myf5]

@pleshakov I tried load the debug OT module by Configmap. The logs of the pods:

[root@k8s-master opentracing]# kubectl logs nginx-ingress-apidmeo-tp52z  -n nginx-ingress 
I0918 12:29:09.843313       1 main.go:155] Starting NGINX Ingress controller Version=edge GitCommit=924726b
F0918 12:29:14.041817       1 manager.go:195] Could not get newest config version: could not get expected version: 0

Seem like there was some wrong, and no debug logs?

Here is what I did:

1. deleted existed IC pods
2. deleted the configmap

3. edit the configmap: [root@k8s-master opentracing]# cat nginx-config-debug.yaml

   kind: ConfigMap
   apiVersion: v1
   metadata:
   name: nginx-config
   namespace: nginx-ingress
   data:
   error-log-level: "debug"
   opentracing: "True"
   opentracing-tracer: "/usr/local/lib/libjaegertracing_plugin.so"
   main-template: |
   user  nginx;
   worker_processes  {{.WorkerProcesses}};
   {{- if .WorkerRlimitNofile}}
   ......truncated....
   
   {{- if .OpenTracingLoadModule}}
   load_module modules/ngx_http_opentracing_module-debug.so;
   {{- end}}
   
   ......truncated....
   }  
   opentracing-tracer-config: |
       {
           "service_name": "nginx-ingress",
           "sampler": {
               "type": "const",
               "param": 1
           },
           "reporter": {
               "localAgentHostPort": "jaeger-agent.default.svc.cluster.local:6831"
           }
       }

4. create the above configmap

5. edit IC Daemonset yaml: [root@k8s-master opentracing]# cat ../../deployments/daemon-set/nginx-plus-ingress-debug.yaml

   apiVersion: apps/v1
   kind: DaemonSet
   metadata:
   name: nginx-ingress-apidmeo
   namespace: nginx-ingress
   spec:
   selector:
   matchLabels:
     app: nginx-ingress-apidemo
   template:
   metadata:
     labels:
       app: nginx-ingress-apidemo
    #annotations:
      #prometheus.io/scrape: "true"
      #prometheus.io/port: "9113"
   spec:
     serviceAccountName: nginx-ingress
     containers:
     - image: myf5/nginx-plus-ingress-opentracing:edge
       imagePullPolicy: IfNotPresent
       name: nginx-plus-ingress-apidemo
       ports:
       - name: http
         containerPort: 80
         hostPort: 80
       - name: https
         containerPort: 443
         hostPort: 443
       - name: apiport
         containerPort: 8889
         hostPort: 8889
   
      #- name: prometheus
        #containerPort: 9113
       env:
       - name: POD_NAMESPACE
         valueFrom:
           fieldRef:
             fieldPath: metadata.namespace
       - name: POD_NAME
         valueFrom:
           fieldRef:
             fieldPath: metadata.name
       args:
         - -nginx-plus
         - -nginx-configmaps=$(POD_NAMESPACE)/nginx-config
         - -default-server-tls-secret=$(POD_NAMESPACE)/default-server-secret
         - -nginx-status
         - -nginx-status-allow-cidrs=172.16.0.0/16
         - -nginx-status-port=8889
         - -nginx-debug
        # - -ingress-class=canary
        #- -v=3 # Enables extensive logging. Useful for troubleshooting.
        #- -report-ingress-status
        #- -external-service=nginx-ingress
        #- -enable-leader-election
        #- -enable-prometheus-metrics
        #- -enable-custom-resources

[myf5]

Thanks all. Rebuild image solve the issue.

[Rulox]

Nice!

Closing the issue.