Memory leaks when using module

[pbirkants]

Hello!

Using this module for a high-traffic web application, I've observed that the server gradually runs out of memory due to ever-increasing httpd worker process memory usage.

I've reproduced the issue with a very basic configuration, please see attached Dockerfile, it builds the module, enables JWT token auth, starts httpd and runs ab to generate requests.

Sample output of build:

 ---> Running in 99174bd2f0d0
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 172.17.0.5. Set the 'ServerName' directive globally to suppress this message
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  3.5  0.0   2392   752 ?        Ss   16:51   0:00 /bin/sh -c httpd && sleep 3 && ps aux && ab -q -n 1000000 -c 50 http://localhost/ >/dev/null && ps aux
root           7  0.0  0.0  11832  4016 ?        Ss   16:51   0:00 httpd
daemon         9  0.0  0.0 2002932 12252 ?       Sl   16:51   0:00 httpd
daemon        10  0.0  0.0 2002932 12252 ?       Sl   16:51   0:00 httpd
daemon        11  0.0  0.0 2002932 12252 ?       Sl   16:51   0:00 httpd
root          93  0.0  0.0   7644  2800 ?        R    16:51   0:00 ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.4  0.0   2392   752 ?        Ss   16:51   0:00 /bin/sh -c httpd && sleep 3 && ps aux && ab -q -n 1000000 -c 50 http://localhost/ >/dev/null && ps aux
root           7  0.0  0.0  11832  4016 ?        Ss   16:51   0:00 httpd
daemon         9 60.0  1.0 3067060 542372 ?      Sl   16:51   0:18 httpd
daemon        10  113  2.1 4144172 1082132 ?     Sl   16:51   0:34 httpd
daemon        11  108  2.1 4069540 1044720 ?     Sl   16:51   0:32 httpd
daemon        95  165  2.7 4735308 1378620 ?     Sl   16:51   0:42 httpd
root         123  0.0  0.0   7644  2724 ?        R    16:51   0:00 ps aux

As you can see, after 1M requests, RSS column adds up to almost 4GiB. If Require valid-user is commented out, this does not happen.

[AnthonyDeroche]

Hello, Thanks for the investigation. I will have a look on this asap.

Maybe you can downgrade the module version or the libjwt version to check if it still occurs.

[GRRedWings]

Was there ever any resolution to this? In looking at using this module I'd be fearful of a known memory leak

[AnthonyDeroche]

I am not able to reproduce the leak with last version of everything :

• Apache 2.4.46
• libjwt 1.12.1
• mod_authnz_jwt 1.2.0

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 1.3 0.0 2384 696 ? Ss 09:50 0:00 /bin/sh -c httpd && sleep 3 && ps aux && ab -q -n 1000000 -c 50 http://localhost/ >/dev/null && ps aux root 8 0.0 0.0 11832 4152 ? Ss 09:50 0:00 httpd daemon 10 0.0 0.0 2002932 4200 ? Sl 09:50 0:00 httpd daemon 11 0.0 0.0 2002932 4176 ? Sl 09:50 0:00 httpd daemon 12 0.0 0.0 2002932 4192 ? Sl 09:50 0:00 httpd root 94 0.0 0.0 7636 2740 ? R 09:50 0:00 ps aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.0 2384 696 ? Ss 09:50 0:00 /bin/sh -c httpd && sleep 3 && ps aux && ab -q -n 1000000 -c 50 http://localhost/ >/dev/null && ps aux root 8 0.0 0.0 11832 4152 ? Ss 09:50 0:00 httpd daemon 10 9.6 0.0 2004084 5580 ? Sl 09:50 0:09 httpd daemon 11 20.9 0.0 2004028 5268 ? Sl 09:50 0:21 httpd daemon 12 54.5 0.0 2004028 5404 ? Sl 09:50 0:56 httpd daemon 96 139 0.0 2004028 5196 ? Sl 09:50 2:17 httpd root 124 0.0 0.0 7636 2760 ? R 09:52 0:00 ps aux

Dockerfile I used

FROM debian:buster-slim as build

WORKDIR /build

RUN apt-get update && \ apt-get install -y ca-certificates make automake git g++ libtool pkg-config autoconf libssl-dev check libjansson-dev libz-dev procps apache2 apache2-dev

ARG LIBJWT_VERSION=1.12.1 ARG MOD_AUTHNZ_JWT_VERSION=1.2.0

RUN git clone https://github.com/benmcollins/libjwt.git && \ cd libjwt && \ git checkout tags/v$LIBJWT_VERSION && \ autoreconf -i && \ ./configure && \ make && \ make install

RUN git clone https://github.com/AnthonyDeroche/mod_authnz_jwt.git && \ cd mod_authnz_jwt && \ git checkout tags/v$MOD_AUTHNZ_JWT_VERSION && \ autoreconf -ivf && \ PKG_CONFIG_PATH=/usr/local ./configure && \ make && \ make install

FROM httpd:2.4

COPY --from=build /usr/local/lib/libjwt.so /usr/lib/x86_64-linux-gnu/libjwt.so.1 COPY --from=build /usr/lib/apache2/modules/mod_authnz_jwt.so /usr/local/apache2/modules/mod_authnz_jwt.so

RUN echo "LoadModule auth_jwt_module modules/mod_authnz_jwt.so" >> /usr/local/apache2/conf/httpd.conf

RUN echo "AuthJWTSignatureAlgorithm HS512\n \ AuthJWTSignatureSharedSecret test\n \ AuthJWTIss test\n \ \n \ AuthType jwt\n \ AuthName \"test\"\n \ Require valid-user\n \ \n \ " >>/usr/local/apache2/conf/httpd.conf

RUN apt-get update && apt-get install procps -y

RUN httpd && sleep 3 && ps aux && ab -q -n 1000000 -c 50 http://localhost/ >/dev/null && ps aux

[jbloggz]

I found a memory leak the the create_token() function (see pull request #55). I doubt this is the same leak, but thought it worth mentioning