Core dumps with Lua nginx module and OCSP enabled on ingress-nginx

strongjz commented 1 month ago

Ingress-nginx updated the lua-nginx-module to version b5d1688ae722538ba4dc8a7ec08820a08abfb93d and started seeing core dumps only when OCSP is enabled.

nginx version: 1.25.5 lua jit: v2.1-20240314 lua Resty Core: v0.1.28 Alpine: 3.20.0

All versions of modules running: https://github.com/kubernetes/ingress-nginx/blob/release-1.11/images/nginx-1.25/rootfs/build.sh

We reverted to it to known good version v0.10.26 and the core dump issues with OCSP went away.

To reproduce, run ingress-nginx version 1.11.0 with enable-ocsp: true https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#enable-ocsp and test it with a cert manager and an HTTPS enabled ingress.

Core Dumps

sudo gdb ./nginx core/core.99

...
#0  0x00005626f775dd3e in ngx_http_lua_ffi_ssl_validate_ocsp_response (resp=<optimized out>, resp_len=<optimized out>, chain_data=<optimized out>, chain_len=<optimized out>, errbuf=0x7fa407769500 "schemeupstream_name\005", errbuf_size=0x7fa4077ad818, valid=0x600)
    at /tmp/build/lua-nginx-module/src/ngx_http_lua_ssl_ocsp.c:483

...
(gdb) backtrace
#18 0x00005626f775db20 in ?? () at /tmp/build/lua-nginx-module/src/ngx_http_lua_ssl_ocsp.c:235

lldb -c core.766
(lldb) target create --core "core.766"
Core file '/Users/strongjz/go/src/github/kubernetes/ingress-nginx/core.766' (x86_64) was loaded.
(lldb) bt all
* thread #1, name = 'nginx', stop reason = signal SIGSEGV: address not mapped to object
  * frame #0: 0x00005626f775dd3e nginx`ngx_http_lua_ffi_ssl_validate_ocsp_response(resp=<unavailable>, resp_len=<unavailable>, chain_data=<unavailable>, chain_len=<unavailable>, errbuf="0S0Q0O0M0K0\t\U00000006\U00000005+\U0000000e\U00000003\U00000002\U0000001a\U00000005", errbuf_size=0x00007fa4077ad818, valid=0x0000000000000600) at ngx_http_lua_ssl_ocsp.c:388:16
    frame #1: 0x00007fa40eb68f92

Please see https://github.com/kubernetes/ingress-nginx/issues/11588 for more information.

Core dumps are available at

zhuizhuhaomeng commented 1 month ago

@strongjz I don't have the ENV to test ocsp.

Would you please test the following patch?

diff --git a/src/ngx_http_lua_ssl_ocsp.c b/src/ngx_http_lua_ssl_ocsp.c
index c9f24cd2..73e1b9c7 100644
--- a/src/ngx_http_lua_ssl_ocsp.c
+++ b/src/ngx_http_lua_ssl_ocsp.c
@@ -280,7 +280,7 @@ ngx_http_lua_ffi_ssl_validate_ocsp_response(const u_char *resp,
     OCSP_RESPONSE         *ocsp = NULL;
     OCSP_BASICRESP        *basic = NULL;
     STACK_OF(X509)        *chain = NULL;
-    ASN1_GENERALIZEDTIME  *thisupdate, *nextupdate;
+    ASN1_GENERALIZEDTIME  *thisupdate = NULL, *nextupdate = NULL;

     ocsp = d2i_OCSP_RESPONSE(NULL, &resp, resp_len);
     if (ocsp == NULL) {

Would you please start the nginx with valgrind if the above patch does not work?

zhuizhuhaomeng commented 1 week ago

@strongjz any progress on this issue?

openresty / lua-nginx-module

Core dumps with Lua nginx module and OCSP enabled on ingress-nginx #2339