help request: how to send mTLS requests in apisix from Lua plugin

[pimg]

Description

hi, Im trying to make an mTLS call from a custom plugin. Using vanilla Openresty I was able to do this using the resty.http library also used within APISIX by various other plugins (e.g. the http-logger. However when using the resty.http lib in Apisix the client certificate is not set on the outgoing connection. Which I can see in the server logs as well as the resulting error inside APISIX which is: SSL_read() failed (SSL: error:1409445C:SSL routines:ssl3_read_bytes:tlsv13 alert certificate required:SSL alert number 116) I tried running in different phases and changing the priority of the custom plugin making the request, but to no avail. In vanilla Openresty this works fine in both the access as well as the content phase. It seems there is something conflicting in the APISIX functionality preventing the proper execution of an mTLS request?

The error differs if I use TLSv1.2 or TLSv1.3, although I believe the root cause is the same, just the error code is different between TLSv1.2 and TLSv1.3 TLSv1.3 error code: 221#221: *393121 SSL_read() failed (SSL: error:1409445C:SSL routines:ssl3_read_bytes:tlsv13 alert certificate required:SSL alert number 116) TLSv1.2 error code: SSL routines:SSL3_READ_BYTES:sslv3 alert bad certificate:s3_pkt.c:1259:SSL alert number 42

I also inspected the traffic with Wireshark, and indeed the client certificates are not being sent (edited)

I created a self contained example. I also included a "vanilla" openresty setup issuing the same mtls requests. The repo contains a Make file making it easy to run the example. https://github.com/pimg/mtls-plugin-test If you need some further assistence just let me know.

I also reached out on the APISIX Slack channel, see the conversation here: https://the-asf.slack.com/archives/CUC5MN17A/p1704362165540319

Environment

• APISIX version (run apisix version): 3.2, 3.6, 3.7, 3.8
• Operating system (run uname -a): run APISIX on Docker on Ubuntu (e.g for version 3.8 apache/apisix:3.8.0-debian)
• OpenResty / Nginx version (run openresty -V or nginx -V): N/A run docker image
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info): N/A run standalone mode
• APISIX Dashboard version, if relevant: N/A run standalone mode
• Plugin runner version, for issues related to plugin runners: N/A run LUA plugin
• LuaRocks version, for installation issues (run luarocks --version): N/A run on Docker image

[shreemaan-abhishek]

could you check the differences in the version of the http library? APISIX uses this: https://github.com/api7/lua-resty-http

[pimg]

Thank you @shreemaan-abhishek for your reply. The Lua plugin is mounted in the APISIX Docker container and executed from there. Looking at the Rockspect from APISIX 3.8 release it looks like the resty-http version is: api7-lua-resty-http = 0.2.2-0

https://github.com/apache/apisix/blob/release/3.8/apisix-master-0.rockspec#L38

[shreemaan-abhishek]

yes, the api7-lua-resty-http is a fork of https://github.com/ledgetech/lua-resty-http and it has some missing features.

[pimg]

ok, but how can it be used to issue a mTLS request with APISIX? Does this differ from the upstream resty-http library, or does it contain a bug?

[pimg]

ok I found the issue. Indeed the API of the forked resty-http library differs from the upstream.

so instead of:

local ok, err = httpc:connect {
    ssl_client_cert = cert,
    ssl_client_priv_key = key,
}

The key material of the client should be provided as:

local ok, err = httpc:connect {
    ssl_cert = cert_in_pem_format,
    ssl_key = key_in_pem_format,
}