The problem with io-event update to 1.3.0

[skirushkin]

Hi, first I would like to thank you for the excellent work with all socketry infrastructure!

I'm running rails app with falcon in kubernetes. Base docker image is ruby:3.2.2-alpine. After io-event updated to 1.3.0 (I tried 1.3.1 and 1.3.2 too) I'm experiencing problems with falcon pod start in k8s.

In pod log I can see that falcon start is success, no issues

{"time":"2023-08-25T07:12:41+00:00","severity":"info","oid":6180,"pid":10,"subject":"Falcon::Command::Host","message":"Falcon Host v0.42.3 taking flight!\n- Configuration: falcon.rb\n- To terminate: Ctrl-C or kill 10\n- To reload: kill -HUP 
{"time":"2023-08-25T07:12:41+00:00","severity":"info","oid":6200,"pid":10,"subject":"Falcon::Service::Application","message":"Binding to #<Async::HTTP::Endpoint http://0.0.0.0:3000/ {}>..."}                                                   
{"time":"2023-08-25T07:12:41+00:00","severity":"info","oid":6200,"pid":11,"subject":"Falcon::Service::Application","message":"Starting application server for /app..."}

But when I try to curl readiness probe in pod, it starts and waits for response forever. If I stop falcon then I get curl error :

backend-falcon-675d99c95d-68dnm:/app$ curl http://0.0.0.0:3000/ready
curl: (56) Recv failure: Connection reset by peer

And the same behaviour I can see in k8s, it doesn't start pod due to no response in readiness probe. It's simple api_only=true rails app My falcon.rb

# frozen_string_literal: true

load :rack

hostname = File.basename(__dir__)
port = ENV.fetch("PORT", 3000)
http_endpoint = "http://0.0.0.0:#{port}"
workers_count = ENV.fetch("FALCON_COUNT", 1).to_i

rack(hostname) do
  cache false
  count workers_count

  endpoint Async::HTTP::Endpoint.parse(http_endpoint)
end

And when I hold io-event version to 1.2.3 in Gemfile than falcon works perfect.

So steps for reproduce:

1. New rails app
2. Create simple route with 200 response
3. Basic falcon.rb
4. Put everything to ruby:3.2.2-alpine based container
5. Start with bundle exec falcon host
6. And try to request this simple route with curl or wget

I'm not so good in issue investigation. But I will provide any additional information according to your advise!

[ioquatix]

Thanks, I'll investigate.

[ioquatix]

I have not updated the falcon test suite or tested it on the latest version of io-event so I've done that here: https://github.com/socketry/falcon/commit/9170a2a59c360945837f5c993e5684bce63b0672 which is running all the tests on ruby 3.2.2 and the latest version of io-event. It appears to be passing, so I'll continue to investigate.

[ioquatix]

I could not reproduce your issue on my Linux dev machine.

Do you mind telling me the version of the Linux kernel you are using?

Also, I tried to make sure we have a consistent reproduction, can you confirm https://github.com/socketry/falcon-my_api is capable of reproducing the issue?

Thanks!

[skirushkin]

Hi, thank you for reply!

Linux kernel version is

backend-falcon-686c874c68-6xql8:/app$ uname -a
Linux backend-falcon-686c874c68-7r9hz 5.4.0-148-generic #165-Ubuntu SMP Tue Apr 18 08:53:12 UTC 2023 x86_64 Linux

backend-falcon-686c874c68-6xql8:/app$ cat /proc/version
Linux version 5.4.0-148-generic (buildd@lcy02-amd64-112) (gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1)) #165-Ubuntu SMP Tue Apr 18 08:53:12 UTC 2023

backend-falcon-686c874c68-6xql8:/app$ cat /etc/os-release
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.18.3
PRETTY_NAME="Alpine Linux v3.18"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://gitlab.alpinelinux.org/alpine/aports/-/issues"

It's ruby:3.2.2-alpine docker image.

I've tested your repository and I can confirm that it's consistent reproduction. Falcon starts but doesn't respond to basic curl. Also I can see a warning message on start

backend-falcon-688d5957bc-6xql8:/app/falcon-my_api$ bundle exec falcon host
/app/vendor/bundle/ruby/3.2.0/gems/io-event-1.3.2/lib/io/event/support.rb:24: warning: IO::Buffer is experimental and both the Ruby and C interface may change in the future!
  0.0s     info: Falcon::Command::Host [oid=0x1824] [ec=0x1838] [pid=26] [2023-08-27 13:57:47 +0000]
               | Falcon Host v0.42.3 taking flight!
               | - Configuration: falcon.rb
               | - To terminate: Ctrl-C or kill 26
               | - To reload: kill -HUP 26
 0.04s     info: Falcon::Service::Application [oid=0x184c] [ec=0x1838] [pid=26] [2023-08-27 13:57:47 +0000]
               | Binding to #<Async::HTTP::Endpoint http://0.0.0.0:3000/ {}>...
 0.05s     info: Falcon::Service::Application [oid=0x184c] [ec=0x1874] [pid=27] [2023-08-27 13:57:47 +0000]
               | Starting application server for /app/falcon-my_api...

In parallel shell curl with forever await and it fails once I stop falcon.

backend-falcon-688d5957bc-6xql8:/app$ curl http://0.0.0.0:3000/
curl: (56) Recv failure: Connection reset by peer

[skirushkin]

Hi again! I've found the problem in my project Gemfile. The problem was that gem pry-rails was defined in :default Rails.group and required on application launch. I've placed it to :test group and now falcon starts correct and respond to routes. When I tried your repository seems there was an issue with recursive Gemfile load (I've placed your repository inside mine project) that's why simple application did't boot correct (with pry-rails on start).

Thank you so much for you time and sorry for disturb! I believe that it's not a socketry infrastructure problem. So you can close this issue 🙏

[ioquatix]

@skirushkin thanks for updating the issue. If you are using io-event 1.3 in production, do you mind reporting back if you notice any other issues? Thanks.

[skirushkin]

FYI, we running 150k+ lines of code RoR project with falcon. With high rps. It's part of microservice infrastructure (so any request to project produce several http or grpc request inside our infrastructure). It uses Redis (redis-rb) and Postgres (sequel). Also we implemented your idea about FiberedConnectionPool from async-sequel repository. Code is here https://github.com/umbrellio/umbrellio-sequel-plugins/blob/master/lib/sequel/extensions/fibered_connection_pool.rb

We got blue-green deployment system with k8s. So for us it will be a pleasure to test any new falcon feature for you. And of course I'll let you know if something goes wrong, thank you for such opportunity!