Error: "load_module" directive in nginx.conf not binary compatible

[takashi-osako-compass]

I am encountering an issue with the "load_module" directive in my nginx.conf file while using the source code from tag v1.0.0 of the DataDog repository. I am running the code inside a Docker image based on python:3.8.12-slim-buster.

Here are the steps I followed:

1. Cloned the DataDog/nginx-datadog repository.
2. Checked out the v1.0.0 tag.
3. Executed the bin/install_build_tooling.sh script to install the necessary build tooling.
4. Created a nginx-version-info file and set NGINX_VERSION=1.14.2 in it.
5. Ran make build to build the code successfully.
6. Updated the nginx.conf file to include the required module.

Attempted to start the application, but encountered the following error: nginx: [emerg] module "libngx_http_datadog_module.so" is not binary compatible in /docker/webapp/nginx.conf:2

I have double-checked the syntax and path to the module file, ensuring they are correct. However, the error persists. I also removed --with-compat option in Makefile to see if it changes, but the error still persists.

Could you please provide guidance on how to troubleshoot and resolve this error based on the procedures I followed?

[dgoffredo]

I also removed --with-compat option in Makefile to see if it changes, but the error still persists.

Good idea, something like that (an nginx ./configure flag) is probably at the heart of the issue.

Which nginx installation are you using within python:3.8.12-slim-buster? Is it the one available via apt-get? I'll pull that image now and look around...

If I apt update and apt install nginx within that python image, and then nginx -V with some added newlines, I get:

nginx version: nginx/1.14.2
built with OpenSSL 1.1.1n  15 Mar 2022 (running with OpenSSL 1.1.1d  10 Sep 2019)
TLS SNI support enabled
configure arguments:
 --with-cc-opt='-g -O2 -fdebug-prefix-map=/build/nginx-FlcIR2/nginx-1.14.2=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -Wdate-time -D_FORTIFY_SOURCE=2'
 --with-ld-opt='-Wl,-z,relro -Wl,-z,now -fPIC'
 --prefix=/usr/share/nginx
 --conf-path=/etc/nginx/nginx.conf
 --http-log-path=/var/log/nginx/access.log
 --error-log-path=/var/log/nginx/error.log
 --lock-path=/var/lock/nginx.lock
 --pid-path=/run/nginx.pid
 --modules-path=/usr/lib/nginx/modules
 --http-client-body-temp-path=/var/lib/nginx/body
 --http-fastcgi-temp-path=/var/lib/nginx/fastcgi
 --http-proxy-temp-path=/var/lib/nginx/proxy
 --http-scgi-temp-path=/var/lib/nginx/scgi
 --http-uwsgi-temp-path=/var/lib/nginx/uwsgi
 --with-debug
 --with-pcre-jit
 --with-http_ssl_module
 --with-http_stub_status_module
 --with-http_realip_module
 --with-http_auth_request_module
 --with-http_v2_module
 --with-http_dav_module
 --with-http_slice_module
 --with-threads
 --with-http_addition_module
 --with-http_geoip_module=dynamic
 --with-http_gunzip_module
 --with-http_gzip_static_module
 --with-http_image_filter_module=dynamic
 --with-http_sub_module
 --with-http_xslt_module=dynamic
 --with-stream=dynamic
 --with-stream_ssl_module
 --with-stream_ssl_preread_module
 --with-mail=dynamic
 --with-mail_ssl_module
 --add-dynamic-module=/build/nginx-FlcIR2/nginx-1.14.2/debian/modules/http-auth-pam
 --add-dynamic-module=/build/nginx-FlcIR2/nginx-1.14.2/debian/modules/http-dav-ext
 --add-dynamic-module=/build/nginx-FlcIR2/nginx-1.14.2/debian/modules/http-echo
 --add-dynamic-module=/build/nginx-FlcIR2/nginx-1.14.2/debian/modules/http-upstream-fair
 --add-dynamic-module=/build/nginx-FlcIR2/nginx-1.14.2/debian/modules/http-subs-filter

The error message within nginx that causes a "binary mismatch" is here: https://github.com/nginx/nginx/blob/6915d2fb2e88e0c339fe37b37ce14f5fe446c1c6/src/core/ngx_module.c#L177-L182

NGX_MODULE_SIGNATURE is a build-time string literal that depends on a bunch of (but not all of) ./configure parameters: https://github.com/nginx/nginx/blob/6915d2fb2e88e0c339fe37b37ce14f5fe446c1c6/src/core/ngx_module.h#L200

These: https://github.com/nginx/nginx/blob/6915d2fb2e88e0c339fe37b37ce14f5fe446c1c6/src/core/ngx_module.h#L21-L198

The error message does not say where the mismatch is, and it does not print the values of NGX_MODULE_SIGNATURE, so we can't figure it out ourselves.

Debian buster is a common distro that I imagine must be used by some of the nginx docker images.

Might it be that the Debian packagers used ./configure arguments that differ from those used by the nginx docker image maintainers? Could be.

I once attempted to use objdump and friends to divine the value of NGX_MODULE_SIGNATURE from a built binary, but it might be impossible.

Here's what some cleaned-up-and-sorted configure arguments look like in the build available via apt in the python image:

 --with-debug
 --with-http_addition_module
 --with-http_auth_request_module
 --with-http_dav_module
 --with-http_geoip_module=dynamic
 --with-http_gunzip_module
 --with-http_gzip_static_module
 --with-http_image_filter_module=dynamic
 --with-http_realip_module
 --with-http_slice_module
 --with-http_ssl_module
 --with-http_stub_status_module
 --with-http_sub_module
 --with-http_v2_module
 --with-http_xslt_module=dynamic
 --with-ld-opt='-Wl,-z,relro -Wl,-z,now -fPIC'
 --with-mail=dynamic
 --with-mail_ssl_module
 --with-pcre-jit
 --with-stream=dynamic
 --with-stream_ssl_module
 --with-stream_ssl_preread_module
 --with-threads

Here's the corresponding flags from the nginx:1.14.2 image (which uses Debian stretch, not buster):

 --with-compat
 --with-file-aio
 --with-http_addition_module
 --with-http_auth_request_module
 --with-http_dav_module
 --with-http_flv_module
 --with-http_gunzip_module
 --with-http_gzip_static_module
 --with-http_mp4_module
 --with-http_random_index_module
 --with-http_realip_module
 --with-http_secure_link_module
 --with-http_slice_module
 --with-http_ssl_module
 --with-http_stub_status_module
 --with-http_sub_module
 --with-http_v2_module
 --with-ld-opt='-specs=/usr/share/dpkg/no-pie-link.specs -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie'
 --with-mail
 --with-mail_ssl_module
 --with-stream
 --with-stream_realip_module
 --with-stream_ssl_module
 --with-stream_ssl_preread_module
 --with-threads

Here's diff $first_snippet $second_snippet:

1c1,2
<  --with-debug
---
>  --with-compat
>  --with-file-aio
5c6
<  --with-http_geoip_module=dynamic
---
>  --with-http_flv_module
8c9,10
<  --with-http_image_filter_module=dynamic
---
>  --with-http_mp4_module
>  --with-http_random_index_module
9a12
>  --with-http_secure_link_module
15,17c18,19
<  --with-http_xslt_module=dynamic
<  --with-ld-opt='-Wl,-z,relro -Wl,-z,now -fPIC'
<  --with-mail=dynamic
---
>  --with-ld-opt='-specs=/usr/share/dpkg/no-pie-link.specs -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie'
>  --with-mail
19,20c21,22
<  --with-pcre-jit
<  --with-stream=dynamic
---
>  --with-stream
>  --with-stream_realip_module

If I pull the source tree for nginx 1.14.2 and look at the auto/options script, which is called by configure, I see the defaults (which is what would be used in our build):

NGX_DEBUG=NO
NGX_CC_OPT=
NGX_LD_OPT=
CPU=NO

NGX_RPATH=NO

NGX_TEST_BUILD_DEVPOLL=NO
NGX_TEST_BUILD_EVENTPORT=NO
NGX_TEST_BUILD_EPOLL=NO
NGX_TEST_BUILD_SOLARIS_SENDFILEV=NO

NGX_PLATFORM=
NGX_WINE=

EVENT_FOUND=NO

EVENT_SELECT=NO
EVENT_POLL=NO

USE_THREADS=NO

NGX_FILE_AIO=NO

HTTP=YES

NGX_HTTP_LOG_PATH=
NGX_HTTP_CLIENT_TEMP_PATH=
NGX_HTTP_PROXY_TEMP_PATH=
NGX_HTTP_FASTCGI_TEMP_PATH=
NGX_HTTP_UWSGI_TEMP_PATH=
NGX_HTTP_SCGI_TEMP_PATH=

HTTP_CACHE=YES
HTTP_CHARSET=YES
HTTP_GZIP=YES
HTTP_SSL=NO
HTTP_V2=NO
HTTP_SSI=YES
HTTP_POSTPONE=NO
HTTP_REALIP=NO
HTTP_XSLT=NO
HTTP_IMAGE_FILTER=NO
HTTP_SUB=NO
HTTP_ADDITION=NO
HTTP_DAV=NO
HTTP_ACCESS=YES
HTTP_AUTH_BASIC=YES
HTTP_AUTH_REQUEST=NO
HTTP_MIRROR=YES
HTTP_USERID=YES
HTTP_SLICE=NO
HTTP_AUTOINDEX=YES
HTTP_RANDOM_INDEX=NO
HTTP_STATUS=NO
HTTP_GEO=YES
HTTP_GEOIP=NO
HTTP_MAP=YES
HTTP_SPLIT_CLIENTS=YES
HTTP_REFERER=YES
HTTP_REWRITE=YES
HTTP_PROXY=YES
HTTP_FASTCGI=YES
HTTP_UWSGI=YES
HTTP_SCGI=YES
HTTP_GRPC=YES
HTTP_PERL=NO
HTTP_MEMCACHED=YES
HTTP_LIMIT_CONN=YES
HTTP_LIMIT_REQ=YES
HTTP_EMPTY_GIF=YES
HTTP_BROWSER=YES
HTTP_SECURE_LINK=NO
HTTP_DEGRADATION=NO
HTTP_FLV=NO
HTTP_MP4=NO
HTTP_GUNZIP=NO
HTTP_GZIP_STATIC=NO
HTTP_UPSTREAM_HASH=YES
HTTP_UPSTREAM_IP_HASH=YES
HTTP_UPSTREAM_LEAST_CONN=YES
HTTP_UPSTREAM_KEEPALIVE=YES
HTTP_UPSTREAM_ZONE=YES

# STUB
HTTP_STUB_STATUS=NO

MAIL=NO
MAIL_SSL=NO
MAIL_POP3=YES
MAIL_IMAP=YES
MAIL_SMTP=YES

STREAM=NO
STREAM_SSL=NO
STREAM_REALIP=NO
STREAM_LIMIT_CONN=YES
STREAM_ACCESS=YES
STREAM_GEO=YES
STREAM_GEOIP=NO
STREAM_MAP=YES
STREAM_SPLIT_CLIENTS=YES
STREAM_RETURN=YES
STREAM_UPSTREAM_HASH=YES
STREAM_UPSTREAM_LEAST_CONN=YES
STREAM_UPSTREAM_ZONE=YES
STREAM_SSL_PREREAD=NO

DYNAMIC_MODULES=

NGX_ADDONS=
NGX_ADDON_DEPS=
DYNAMIC_ADDONS=

NGX_COMPAT=NO

USE_PCRE=NO
PCRE=NONE
PCRE_OPT=
PCRE_CONF_OPT=
PCRE_JIT=NO

USE_OPENSSL=NO
OPENSSL=NONE

USE_ZLIB=NO
ZLIB=NONE
ZLIB_OPT=
ZLIB_ASM=NO

USE_PERL=NO
NGX_PERL=perl

USE_LIBXSLT=NO
USE_LIBGD=NO
USE_GEOIP=NO

NGX_GOOGLE_PERFTOOLS=NO
NGX_CPP_TEST=NO

NGX_LIBATOMIC=NO

... the question is: Which ./configure flags do we need to specify to be compatible with the nginx packaged with the python image?

One guess is to use whatever is reported by nginx -V within the python image. Keep in mind that only flags corresponding to these macros matter.

I would avoid --with-debug. There might be latent bugs caused by nginx's attempts to log connection information. That's something I intend to look into (noticed the possible issue in a different project).

So, consider building the module with the following ./configure flags. This is what nginx -V reported in the python image, but with some hopefully irrelevant stuff omitted:

 --with-http_addition_module
 --with-http_auth_request_module
 --with-http_dav_module
 --with-http_gunzip_module
 --with-http_gzip_static_module
 --with-http_realip_module
 --with-http_slice_module
 --with-http_ssl_module
 --with-http_stub_status_module
 --with-http_sub_module
 --with-http_v2_module
 --with-mail_ssl_module
 --with-pcre-jit
 --with-stream_ssl_module
 --with-stream_ssl_preread_module
 --with-threads

We might need to whittle down that list further. I have yet to try it.

[takashi-osako-compass]

I have gained valuable insight into how nginx is compiled, thanks to your comments. It became clear to me that in order to create a module for an existing nginx installation, it is essential to ensure that the parameters align with the ones already installed. After examining the nginx parameters using the nginx -V command and removing any unnecessary ones, as well as installing additional packages, I successfully compiled the code and resolved the "not binary compatible" error. As a result, I have submitted a Pull Request with my changes.

[dgoffredo]

Per my comment in the PR, I'll leave this issue open until we have a general solution to the nginx ./configure problem. It will look a lot like the PR.

[monelgordillo]

What's the solution for this?