Getting version mismatch while loading the module - `version 1018000 instead of 1019003`

[chrissam]

Describe your environment

Using nginx 1.19.3 that comes with --with-compat

$ nginx -V                                                                                                                                                                                                                 [gke_hasura-lux_us-east1_prod-k8s-cluster/default]
nginx version: nginx/1.19.3
built by gcc 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04) 
built with OpenSSL 1.1.1  11 Sep 2018
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g -O2 -fdebug-prefix-map=/data/builder/debuild/nginx-1.19.3/debian/debuild-base/nginx-1.19.3=. -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie'

I used the module otel_ngx_module-ubuntu-18.04-stable.so downloaded from https://github.com/open-telemetry/opentelemetry-cpp-contrib/actions/runs/705078540

OS Version:

Ubuntu 18.04.3 LTS 

nginx.conf

  user  nginx;
  worker_processes  1;  

  error_log  /var/log/nginx/error.log warn;
  pid        /var/run/nginx.pid;

  load_module /etc/nginx/modules/otel_ngx_module.so;

  events {
      worker_connections  1024;
  }

  http {
      include       /etc/nginx/mime.types;
      default_type  application/octet-stream;

      log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                        '$status $body_bytes_sent "$http_referer" '
                        '"$http_user_agent" "$http_x_forwarded_for"';

      access_log  /var/log/nginx/access.log  main;

      sendfile        on;
      #tcp_nopush     on; 

      keepalive_timeout  65; 

      #gzip  on; 

      include /etc/nginx/conf.d/*.conf;
      include /etc/nginx/sites-enabled/*.conf;
  }

Error

$ sudo nginx -t                                                                                                                                                                                                            
nginx: [emerg] module "/etc/nginx/modules/otel_ngx_module.so" version 1018000 instead of 1019003 in /etc/nginx/nginx.conf:8
nginx: configuration file /etc/nginx/nginx.conf test failed

Steps to reproduce Refer description

What is the expected behavior?

$ sudo nginx -t                                                                                                                                                                                                       
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful                                                                                                                                                                                                         

What is the actual behavior?

$ sudo nginx -t                                                                                                                                                                                                            
nginx: [emerg] module "/etc/nginx/modules/otel_ngx_module.so" version 1018000 instead of 1019003 in /etc/nginx/nginx.conf:8
nginx: configuration file /etc/nginx/nginx.conf test failed

Additional context Add any other context about the problem here.

[chrissam]

Update

The otel_ngx_module-ubuntu-18.04-stable.so module works fine with Ubuntu 20.04, running nginx 1.18.0

$ nginx -V
nginx version: nginx/1.18.0 (Ubuntu)
built with OpenSSL 1.1.1f  31 Mar 2020
TLS SNI support enabled
configure arguments: --with-cc-opt='-g -O2 -fdebug-prefix-map=/build/nginx-KTLRnK/nginx-1.18.0=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -fPIC' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-compat --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module --with-threads --with-http_addition_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_sub_module --with-http_xslt_module=dynamic --with-stream=dynamic --with-stream_ssl_module --with-mail=dynamic --with-mail_ssl_module

[chrissam]

Update

The otel_ngx_module-ubuntu-18.04-stable.so module works fine with Ubuntu 20.04, running nginx 1.18.0

$ nginx -V
nginx version: nginx/1.18.0 (Ubuntu)
built with OpenSSL 1.1.1f  31 Mar 2020
TLS SNI support enabled
configure arguments: --with-cc-opt='-g -O2 -fdebug-prefix-map=/build/nginx-KTLRnK/nginx-1.18.0=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -fPIC' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-compat --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module --with-threads --with-http_addition_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_sub_module --with-http_xslt_module=dynamic --with-stream=dynamic --with-stream_ssl_module --with-mail=dynamic --with-mail_ssl_module

Update

Though nginx -t didn't throw any error, I was not able to start nginx as a result of below error,

dlopen() "/usr/share/nginx/modules/otel_ngx_module.so" failed (/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /usr/share/nginx/modules/otel_ngx_module.so)) in /etc/nginx/modules-enabled/50-mod-otel.conf:1

[seemk]

To clarify stable and mainline refer to the nginx versions here: http://nginx.org/en/download.html, i.e. stable means 1.18.x and mainline 1.19.x. Although now looking at the link, nginx has a new version: 1.20 and 1.21 as of lately. I'll go over the build scripts to incorporate these changes.

Did the dlopen error happen on Ubuntu 20.04 with nginx 1.18.0? Then you could try the so built for this Ubuntu version (20.04): https://github.com/open-telemetry/opentelemetry-cpp-contrib/suites/2845583231/artifacts/63504121

[chrissam]

@seemk I was able to successfully make it run in ubuntu 20.04 (nginx:1.18.0) using otel_ngx_module-ubuntu-20.04-stable.so. I'm basically trying to make this work for openresty nginx:1.19.3. I'll let you know how it goes :)

[seemk]

I'll make the CI build an artifact for the latest OpenResty as well, but might not get to it today yet

[seemk]

It looks like the OpenResty from apt does not have --with-compat enabled. Are you using the OpenResty Docker images or installing it via apt?

Looking for a solution for the apt problem currently

[chrissam]

@seemk I was trying from apt and as you said it doesn't have --with-compat. I'm yet to try in OpenResty Docker or by building it from source. I should probably able to test it by tomorrow.

[ElenaForester]

@chrissam were you able to make it work with openresty?

[ElenaForester]

I'll make the CI build an artifact for the latest OpenResty as well, but might not get to it today yet

any chance that it was done?