EugenMayer
commented
12 months ago Confirmed that using nginx:1.25.1-bookworm as the baseline image works, no segfaults happen.
Using nginx:mainline-bookworm causes worker crashes on most xhr calls.
Crawling through the we i found that 'non matching nginx modules' could cause this. We indeed use ndk/lua/brotli in our stack which we add like described in the docs using this dockerfile
ARG FROM_IMAGE=nginx:mainline-bookworm
# this is what we use to build the modules
# this has been copied from https://github.com/nginxinc/docker-nginx/blob/master/modules/Dockerfile#L5
# also see https://github.com/nginxinc/docker-nginx/blob/master/modules/README.md
# and is using ENABLED_MODULES as ENV (changed from ARG)
FROM $FROM_IMAGE as builder
# This is how we select the modules to select. Be sure to symlink them later (way below in our HACK)
# we want ndk/lua for lua support, brotli for brotli compression
ENV ENABLED_MODULES="ndk lua brotli"
#### PURE COPY FROM GIT ############
RUN set -ex \
&& if [ "$ENABLED_MODULES" = "" ]; then \
echo "No additional modules enabled, exiting"; \
exit 1; \
fi
COPY ./ /modules/
RUN set -ex \
&& apt update \
&& apt install -y --no-install-suggests --no-install-recommends \
patch make wget mercurial devscripts debhelper dpkg-dev \
quilt lsb-release build-essential libxml2-utils xsltproc \
equivs git g++ libparse-recdescent-perl \
&& XSLSCRIPT_SHA512="f7194c5198daeab9b3b0c3aebf006922c7df1d345d454bd8474489ff2eb6b4bf8e2ffe442489a45d1aab80da6ecebe0097759a1e12cc26b5f0613d05b7c09ffa *stdin" \
&& wget -O /tmp/xslscript.pl https://hg.nginx.org/xslscript/raw-file/01dc9ba12e1b/xslscript.pl \
&& if [ "$(cat /tmp/xslscript.pl | openssl sha512 -r)" = "$XSLSCRIPT_SHA512" ]; then \
echo "XSLScript checksum verification succeeded!"; \
chmod +x /tmp/xslscript.pl; \
mv /tmp/xslscript.pl /usr/local/bin/; \
else \
echo "XSLScript checksum verification failed!"; \
exit 1; \
fi \
&& hg clone -r ${NGINX_VERSION}-${PKG_RELEASE%%~*} https://hg.nginx.org/pkg-oss/ \
&& cd pkg-oss \
&& mkdir /tmp/packages \
&& for module in $ENABLED_MODULES; do \
echo "Building $module for nginx-$NGINX_VERSION"; \
if [ -d /modules/$module ]; then \
echo "Building $module from user-supplied sources"; \
# check if module sources file is there and not empty
if [ ! -s /modules/$module/source ]; then \
echo "No source file for $module in modules/$module/source, exiting"; \
exit 1; \
fi; \
# some modules require build dependencies
if [ -f /modules/$module/build-deps ]; then \
echo "Installing $module build dependencies"; \
apt update && apt install -y --no-install-suggests --no-install-recommends $(cat /modules/$module/build-deps | xargs); \
fi; \
# if a module has a build dependency that is not in a distro, provide a
# shell script to fetch/build/install those
# note that shared libraries produced as a result of this script will
# not be copied from the builder image to the main one so build static
if [ -x /modules/$module/prebuild ]; then \
echo "Running prebuild script for $module"; \
/modules/$module/prebuild; \
fi; \
/pkg-oss/build_module.sh -v $NGINX_VERSION -f -y -o /tmp/packages -n $module $(cat /modules/$module/source); \
BUILT_MODULES="$BUILT_MODULES $(echo $module | tr '[A-Z]' '[a-z]' | tr -d '[/_\-\.\t ]')"; \
elif make -C /pkg-oss/debian list | grep -P "^$module\s+\d" > /dev/null; then \
echo "Building $module from pkg-oss sources"; \
cd /pkg-oss/debian; \
make rules-module-$module BASE_VERSION=$NGINX_VERSION NGINX_VERSION=$NGINX_VERSION; \
mk-build-deps --install --tool="apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends --yes" debuild-module-$module/nginx-$NGINX_VERSION/debian/control; \
make module-$module BASE_VERSION=$NGINX_VERSION NGINX_VERSION=$NGINX_VERSION; \
find ../../ -maxdepth 1 -mindepth 1 -type f -name "*.deb" -exec mv -v {} /tmp/packages/ \;; \
BUILT_MODULES="$BUILT_MODULES $module"; \
else \
echo "Don't know how to build $module module, exiting"; \
exit 1; \
fi; \
done \
&& echo "BUILT_MODULES=\"$BUILT_MODULES\"" > /tmp/packages/modules.env
FROM $FROM_IMAGE
COPY --from=builder /tmp/packages /tmp/packages
RUN set -ex \
&& apt update \
&& . /tmp/packages/modules.env \
&& for module in $BUILT_MODULES; do \
apt install --no-install-suggests --no-install-recommends -y /tmp/packages/nginx-module-${module}_${NGINX_VERSION}*.deb; \
done \
&& rm -rf /tmp/packages \
&& rm -rf /var/lib/apt/lists/
#### PURE COPY FROM GIT ############
################# HACK ###########
# align the docker-hub image with our debian based expectations
RUN rm -f /etc/nginx/conf.d/default.conf \
&& mkdir -p /etc/nginx/sites-enabled /etc/nginx/modules-enabled \
# Load all the modules you want / compiled
# the order for LUA is important, NDK first!
&& echo "load_module modules/ndk_http_module.so;" > /etc/nginx/modules-enabled/00_ndk_http_module.conf \
&& echo "load_module modules/ngx_http_lua_module.so;" > /etc/nginx/modules-enabled/01_ngx_http_lua_module.conf \
&& echo "load_module modules/ngx_http_image_filter_module.so;" > /etc/nginx/modules-enabled/ngx_http_image_filter_module.conf \
&& echo "load_module modules/ngx_http_xslt_filter_module.so;" > /etc/nginx/modules-enabled/ngx_http_xslt_filter_module.conf \
&& echo "load_module modules/ngx_http_js_module.so;" > /etc/nginx/modules-enabled/ngx_http_js_module.conf \
&& echo "load_module modules/ngx_http_brotli_filter_module.so;" > /etc/nginx/modules-enabled/ngx_http_brotli_filter_module.conf \
&& echo "load_module modules/ngx_http_brotli_static_module.so;" > /etc/nginx/modules-enabled/ngx_http_brotli_static_module.confSo we use the script provided under https://github.com/nginxinc/docker-nginx/blob/master/modules/Dockerfile#L5 - and we used it before for about 6 months without an issue or changes. It broke with the recent 1.25.2 upgrade on mainline. Maybe some sources are no longer matching?
I would love to provide a proper core-dump but yet fail to get the header for the symbols
thresheek
Environment
Bug
When building our image including ndk/lua/brotli with 1.25.2 as baseline, most of our XHR calls return an empty response with an crash / coredump.
Details
To be more of a help, i tried to get the core dump. I have it, but i miss the symbols (starting with
nginx-debug). I could not find any hints/docs how it is intended to get the symbols forgdb backtraceso i could share any more useful information here.Can someone hint me to the docs or explain how to get the symbols (without recompiling nginx myself)?