bug: Response-rewrite plug-in BUG

flylan commented 1 year ago

Current Behavior

When the upstream service is nginx+php, the plug-in body filters will return garbled code

Expected Behavior

The body can display normally without garbled code

Error Logs

no error logs

Steps to Reproduce

The response-rewrite plug-in in the document is set to httpbin upstream, and forwarding is normal. However, if I set the upstream to nginx+php and configure the body filter rule, the body will return garbled code

with https://dog.ceo/api/breeds/image/random The link is an example. The dog.ceo is a third-party system. Each request will randomly return a picture of a dog. The normal response is as follows:

I added the following rules in Apisix： curl -s "http://127.0.0.1:9180/apisix/admin/routes/1000" \ -H "X-API-KEY: admin-key" -X PUT -d ' { "host": "dog.ceo", "uri": "/*", "plugins": { "response-rewrite": { "filters": [{ "regex": "message", "scope": "global", "replace": "message_rewrites" }] } }, "upstream": { "type": "roundrobin", "scheme": "https", "nodes": { "dog.ceo:443": 1 } } }'

The local binding domain name dog.ceo is 127.0.0.1, and the request is made through postman https://dog.ceo/api/breeds/image/random

When I turn off Accept-Encoding, it can display normally

Environment

APISIX version (run apisix version):
3.1.0
Operating system (run uname -a):
Linux ace855b5e2cf 5.15.77-amd64-desktop #1 SMP Wed Nov 9 15:59:34 CST 2022 x86_64 GNU/Linux
OpenResty / Nginx version (run openresty -V or nginx -V):
nginx version: openresty/1.21.4.1 built by gcc 10.2.1 20210110 (Debian 10.2.1-6) built with OpenSSL 1.1.1s 1 Nov 2022 TLS SNI support enabled configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt='-O2 -DAPISIX_BASE_VER=1.21.4.1.5 -DNGX_GRPC_CLI_ENGINE_PATH=/usr/local/openresty/libgrpc_engine.so -DNGX_HTTP_GRPC_CLI_ENGINE_PATH=/usr/local/openresty/libgrpc_engine.so -DNGX_LUA_ABORT_AT_PANIC -I/usr/local/openresty/zlib/include -I/usr/local/openresty/pcre/include -I/usr/local/openresty/openssl111/include' --add-module=../ngx_devel_kit-0.3.1 --add-module=../echo-nginx-module-0.62 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2 --add-module=../set-misc-nginx-module-0.33 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.09 --add-module=../srcache-nginx-module-0.32 --add-module=../ngx_lua-0.10.21 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.33 --add-module=../array-var-nginx-module-0.05 --add-module=../memc-nginx-module-0.19 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.9 --add-module=../ngx_stream_lua-0.0.11 --with-ld-opt='-Wl,-rpath,/usr/local/openresty/luajit/lib -Wl,-rpath,/usr/local/openresty/wasmtime-c-api/lib -L/usr/local/openresty/zlib/lib -L/usr/local/openresty/pcre/lib -L/usr/local/openresty/openssl111/lib -Wl,-rpath,/usr/local/openresty/zlib/lib:/usr/local/openresty/pcre/lib:/usr/local/openresty/openssl111/lib' --add-module=/tmp/tmp.4LxUDwBgw8/openresty-1.21.4.1/../mod_dubbo-1.0.2 --add-module=/tmp/tmp.4LxUDwBgw8/openresty-1.21.4.1/../ngx_multi_upstream_module-1.1.1 --add-module=/tmp/tmp.4LxUDwBgw8/openresty-1.21.4.1/../apisix-nginx-module-1.12.0 --add-module=/tmp/tmp.4LxUDwBgw8/openresty-1.21.4.1/../apisix-nginx-module-1.12.0/src/stream --add-module=/tmp/tmp.4LxUDwBgw8/openresty-1.21.4.1/../apisix-nginx-module-1.12.0/src/meta --add-module=/tmp/tmp.4LxUDwBgw8/openresty-1.21.4.1/../wasm-nginx-module-0.6.4 --add-module=/tmp/tmp.4LxUDwBgw8/openresty-1.21.4.1/../lua-var-nginx-module-v0.5.3 --add-module=/tmp/tmp.4LxUDwBgw8/openresty-1.21.4.1/../grpc-client-nginx-module-v0.4.0 --with-poll_module --with-pcre-jit --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-http_v2_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --with-http_stub_status_module --with-http_realip_module --with-http_addition_module --with-http_auth_request_module --with-http_secure_link_module --with-http_random_index_module --with-http_gzip_static_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-threads --with-compat --with-stream --with-http_ssl_module
etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info):
{"hostname":"ace855b5e2cf","version":"3.1.0","boot_time":1676887331,"etcd_version":"3.5.0","id":"87d1ced7-236d-4e06-8c51-46c4715f4178"}
APISIX Dashboard version, if relevant:
Plugin runner version, for issues related to plugin runners:
LuaRocks version, for installation issues (run luarocks --version):

flylan commented 1 year ago

I debugged the Response-rewrite plug-in and found the problem in the core.response.hold_body_chunk method, when I configure the upstream as httpbin, the body data obtained by this method is normal. However, when I configure the upstream as dog.ceo, the data obtained by this method is garbled, and I cannot fix this problem by myself

flylan commented 1 year ago

I tried to set the upstream as the local nginx-php service, but the problem remained. This plug-in can rewrite the response header normally without garbled code, but it cannot rewrite the response body normally

soulbird commented 1 year ago

According to your description, the response received by APISIX from the upstream is compressed by gzip, which makes the response-rewrite plugin unable to perform text recognition and replacement. You can turn off the Accept-Encoding request header as you describe, or let the upstream send the uncompressed content

flylan commented 1 year ago

According to your description, the response received by APISIX from the upstream is compressed by gzip, which makes the response-rewrite plugin unable to perform text recognition and replacement. You can turn off the Accept-Encoding request header as you describe, or let the upstream send the uncompressed content

OK。There is no garbled code after removing Accept-Encoding in the M. rewrite() process

apache / apisix