THREESCALE-9315 fix limit header extension for async mode

What

Backend listener in async mode (falcon HTTP server) failed to return HTTP response with headers with integer value. This happened when 3scale-Options: limit_headers=1 header was `present in the authrep request.

Request and response

GET /transactions/authrep.xml?service_id=2&service_token=*****&usage%5Bhits%5D=1&user_key=***** HTTP/1.1\r
Connection: Keep-Alive\r
User-Agent: APIcast/3.14.0 (Linux; x64; env:staging)\r
Host: 10.131.4.59\r
3scale-Options: rejection_reason_header=1&limit_headers=1&no_body=1\r
\r
< 2023/10/16 16:22:29.000083450  length=161 from=0 to=160
HTTP/1.1 500 Internal Server Error\r
content-type: text/plain\r
vary: accept-encoding\r
content-length: 54\r
\r
NoMethodError: undefined method `split' for -1:Integer2023/10/16 16:23:29 socat[9] N socket 1 (fd 6) is at EOF

Logs:

1m    error: Protocol::Rack::Adapter::Rack2 [oid=0x1c98] [ec=0x1cac] [pid=140] [2023-10-16 19:01:49 +0000]
               |   NoMethodError: undefined method `split' for -1:Integer
               |   → vendor/bundle/ruby/3.0.0/gems/protocol-rack-0.2.4/lib/protocol/rack/adapter/rack2.rb:112 in `block in wrap_headers'
               |     vendor/bundle/ruby/3.0.0/gems/rack-2.2.6.4/lib/rack/utils.rb:461 in `block in each'
               |     vendor/bundle/ruby/3.0.0/gems/rack-2.2.6.4/lib/rack/utils.rb:460 in `each'
               |     vendor/bundle/ruby/3.0.0/gems/rack-2.2.6.4/lib/rack/utils.rb:460 in `each'
               |     vendor/bundle/ruby/3.0.0/gems/protocol-rack-0.2.4/lib/protocol/rack/adapter/rack2.rb:106 in `wrap_headers'
               |     vendor/bundle/ruby/3.0.0/gems/protocol-rack-0.2.4/lib/protocol/rack/adapter/rack2.rb:84 in `call'
               |     vendor/bundle/ruby/3.0.0/gems/protocol-http-0.24.1/lib/protocol/http/middleware.rb:33 in `call'
               |     vendor/bundle/ruby/3.0.0/gems/protocol-rack-0.2.4/lib/protocol/rack/rewindable.rb:58 in `call'
               |     vendor/bundle/ruby/3.0.0/gems/protocol-http-0.24.1/lib/protocol/http/middleware.rb:33 in `call'
               |     vendor/bundle/ruby/3.0.0/gems/protocol-http-0.24.1/lib/protocol/http/content_encoding.rb:29 in `call'
               |     vendor/bundle/ruby/3.0.0/gems/protocol-http-0.24.1/lib/protocol/http/middleware.rb:33 in `call'
               |     vendor/bundle/ruby/3.0.0/gems/async-http-0.60.1/lib/async/http/server.rb:67 in `block in accept'
               |     vendor/bundle/ruby/3.0.0/gems/async-http-0.60.1/lib/async/http/protocol/http1/server.rb:62 in `each'
               |     vendor/bundle/ruby/3.0.0/gems/async-http-0.60.1/lib/async/http/server.rb:56 in `accept'
               |     vendor/bundle/ruby/3.0.0/gems/async-io-1.34.3/lib/async/io/server.rb:32 in `block in accept_each'
               |     vendor/bundle/ruby/3.0.0/gems/async-io-1.34.3/lib/async/io/socket.rb:73 in `block in accept'
               |     vendor/bundle/ruby/3.0.0/gems/async-1.31.0/lib/async/task.rb:261 in `block in make_fiber'

Root issue

Falcon was upgraded from 0.35.6 to 0.42.3 when apisonator was upgraded to ruby 3 (see[PR #347), which introduced the issue. Header values are expected to be strings. This issue cannot be reproduced with falcon 0.35.6

Verification steps

First let's reproduce the error

Deploy redis instances

docker run --net=host -ti --rm --name redis-stats registry.redhat.io/rhscl/redis-5-rhel7:5 redis-server --port 26379
docker run --net=host -ti --rm --name redis-queue registry.redhat.io/rhscl/redis-5-rhel7:5 redis-server --port 36379

Deploy backend listener in async mode

docker run -it --rm --net=host --name listener --env CONFIG_REDIS_PROXY=127.0.0.1:26379 --env CONFIG_QUEUES_MASTER_NAME=127.0.0.1:36379 --env RACK_ENV=production --env CONFIG_INTERNAL_API_USER=internal_api_user --env CONFIG_INTERNAL_API_PASSWORD=internal_api_password --env CONFIG_REDIS_ASYNC=true --env LISTENER_WORKERS=1 quay.io/3scale/apisonator:latest 3scale_backend -s falcon start -e production -p 3000 -x /dev/stdout

Populate some data

docker run -it --rm --name buddhi --net=host quay.io/3scale/perftest-toolkit:buddhi-v1.1.0 -I http://127.0.0.1:3000 -B http://127.0.0.1:3000 -U internal_api_user -P internal_api_password -E http://echo-api.3scale.net -T simple -A http://benchmark.3sca.net

Get backend paths


curl http://127.0.0.1:8089/paths/backend?lines=1

/transactions/authrep.xml?provider_key=c07adf00-a7f4-49c4-a98e-6326fd98feba&service_id=c20e7600-4604-45aa-9f81-048f6bc0ce42&user_key=ffdce02cd1c84e4e&usage%5B15407441-9e73-47f9-9dc7-74a098e6c98e%5D=1


* Run backend request with the given path. Add the header that triggers the issue

curl -v -H "3scale-Options: limit_headers=1" "http://127.0.0.1:3000/transactions/authrep.xml?provider_key=c07adf00-a7f4-49c4-a98e-6326fd98feba&service_id=c20e7600-4604-45aa-9f81-048f6bc0ce42&user_key=ffdce02cd1c84e4e&usage%5B15407441-9e73-47f9-9dc7-74a098e6c98e%5D=1"

Trying 127.0.0.1:3000...
Connected to 127.0.0.1 (127.0.0.1) port 3000 (#0)

GET /transactions/authrep.xml?provider_key=c07adf00-a7f4-49c4-a98e-6326fd98feba&service_id=c20e7600-4604-45aa-9f81-048f6bc0ce42&user_key=ffdce02cd1c84e4e&usage%5B15407441-9e73-47f9-9dc7-74a098e6c98e%5D=1 HTTP/1.1 Host: 127.0.0.1:3000 User-Agent: curl/7.81.0 Accept: / 3scale-Options: limit_headers=1
Mark bundle as not supporting multiuse < HTTP/1.1 500 Internal Server Error < content-type: text/plain < vary: accept-encoding < content-length: 61 <
Connection #0 to host 127.0.0.1 left intact NoMethodError: undefined method `split' for 999999999:Integer

The listener shows logs

3m    error: Protocol::Rack::Adapter::Rack2 [oid=0x17fc] [ec=0x1798] [pid=8] [2023-10-17 16:11:29 +0000]
               |   NoMethodError: undefined method `split' for 999999999:Integer
               |   → /usr/share/gems/gems/protocol-rack-0.2.4/lib/protocol/rack/adapter/rack2.rb:112 in `block in wrap_headers'
               |     /usr/share/gems/gems/rack-2.2.6.4/lib/rack/utils.rb:461 in `block in each'
               |     /usr/share/gems/gems/rack-2.2.6.4/lib/rack/utils.rb:460 in `each'
               |     /usr/share/gems/gems/rack-2.2.6.4/lib/rack/utils.rb:460 in `each'
               |     /usr/share/gems/gems/protocol-rack-0.2.4/lib/protocol/rack/adapter/rack2.rb:106 in `wrap_headers'
               |     /usr/share/gems/gems/protocol-rack-0.2.4/lib/protocol/rack/adapter/rack2.rb:84 in `call'
               |     /usr/share/gems/gems/protocol-http-0.24.1/lib/protocol/http/middleware.rb:33 in `call'
               |     /usr/share/gems/gems/protocol-rack-0.2.4/lib/protocol/rack/rewindable.rb:58 in `call'
               |     /usr/share/gems/gems/protocol-http-0.24.1/lib/protocol/http/middleware.rb:33 in `call'
               |     /usr/share/gems/gems/protocol-http-0.24.1/lib/protocol/http/content_encoding.rb:29 in `call'
               |     /usr/share/gems/gems/protocol-http-0.24.1/lib/protocol/http/middleware.rb:33 in `call'
               |     /usr/share/gems/gems/async-http-0.60.1/lib/async/http/server.rb:67 in `block in accept'
               |     /usr/share/gems/gems/async-http-0.60.1/lib/async/http/protocol/http1/server.rb:62 in `each'
               |     /usr/share/gems/gems/async-http-0.60.1/lib/async/http/server.rb:56 in `accept'
               |     /usr/share/gems/gems/async-io-1.34.3/lib/async/io/server.rb:32 in `block in accept_each'
               |     /usr/share/gems/gems/async-io-1.34.3/lib/async/io/socket.rb:73 in `block in accept'
               |     /usr/share/gems/gems/async-1.31.0/lib/async/task.rb:261 in `block in make_fiber'

When running apisonator image before the upgrade of falcon, that is quay.io/3scale/apisonator:3scale-2.13.1-GA, the issue does not happen

Stop the listener and re-run with the 2.13.1 image

docker run -it --rm --net=host --name listener --env CONFIG_REDIS_PROXY=127.0.0.1:26379 --env CONFIG_QUEUES_MASTER_NAME=127.0.0.1:36379 --env RACK_ENV=production --env CONFIG_INTERNAL_API_USER=internal_api_user --env CONFIG_INTERNAL_API_PASSWORD=internal_api_password --env CONFIG_REDIS_ASYNC=true --env LISTENER_WORKERS=1 quay.io/3scale/apisonator:3scale-2.13.1-GA 3scale_backend -s falcon start -e production -p 3000 -x /dev/stdout

Re-run backend request with the given path. Exactly the same request.

curl -v -H "3scale-Options: limit_headers=1" "http://127.0.0.1:3000/transactions/authrep.xml?provider_key=c07adf00-a7f4-49c4-a98e-6326fd98feba&service_id=c20e7600-4604-45aa-9f81-048f6bc0ce42&user_key=ffdce02cd1c84e4e&usage%5B15407441-9e73-47f9-9dc7-74a098e6c98e%5D=1"

The response is 200 OK.

Create image from the fix (checkout to the git branch first)

docker build -t apisonator-test -f openshift/distro/ubi/8/release/Dockerfile .

run the new image with the fix

docker run -it --rm --net=host --name listener --env CONFIG_REDIS_PROXY=127.0.0.1:26379 --env CONFIG_QUEUES_MASTER_NAME=127.0.0.1:36379 --env RACK_ENV=production --env CONFIG_INTERNAL_API_USER=internal_api_user --env CONFIG_INTERNAL_API_PASSWORD=internal_api_password --env CONFIG_REDIS_ASYNC=true --env LISTENER_WORKERS=1 apisonator-test 3scale_backend -s falcon start -e production -p 3000 -x /dev/stdout

Re-run backend request with the given path. Exactly the same request.

curl -v -H "3scale-Options: limit_headers=1" "http://127.0.0.1:3000/transactions/authrep.xml?provider_key=c07adf00-a7f4-49c4-a98e-6326fd98feba&service_id=c20e7600-4604-45aa-9f81-048f6bc0ce42&user_key=ffdce02cd1c84e4e&usage%5B15407441-9e73-47f9-9dc7-74a098e6c98e%5D=1"

The response is 200 OK.

3scale / apisonator