Gracefully restart running application

kinnalru commented 2 years ago

Hi. I investigate possibility to gracefully restart currently running rails application from inside running instance. Main aim is to restart application after serving certain number of requests (ex. After 4000 requests) or after hitting memory limit.

In Passenger or Puma this is possible from middleware through passenger-config dettach-process $pid or something like this.

I see some methods is async::containers like reload or other but I can't understand how to use them from inside middleware. Is it sufficient to "kill self by INT signal" ?

ioquatix commented 2 years ago

Yes, but that will terminate all current requests handled by that process... try it out and let me know how you get on?

kinnalru commented 2 years ago

Yes, but that will terminate all current requests handled by that process... try it out and let me know how you get on?

I make some experimets... Test server through falcon serve and falcon host ... killing self from inside application (from middleware) kills current request immidiate ... ... falcon supervisor restart stops application I don't understand why ...

But killing through signal HUP whole server works as expected but some requests are broken. Example (2 forks):

starting rails server (without DB just api) by RAILS_ENV=production bundle exec falcon host
```
# falcon.rb
```

load :rack, :supervisor

rack 'prober' do scheme 'http' protocol { Async::HTTP::Protocol::HTTP1 }

count 2

endpoint do Async::HTTP::Endpoint.for(scheme, '0.0.0.0', port: 3000, protocol: protocol) end

end

supervisor



2. requesting with Yandex Tank 10 rps. 
3. killing with single `kill -HUP $PID`
4. ... aplication reloads...
5. see broken requests :(

![Screenshot_20220905_191608](https://user-images.githubusercontent.com/1270997/188488002-a866b20b-e04a-47ff-9402-d2547ade9303.png)

ioquatix commented 2 years ago

Okay let me take a look.

trevorturk commented 1 year ago

I believe this is a related issue: https://github.com/socketry/falcon/issues/71, and also maybe https://github.com/socketry/falcon/issues/22

ioquatix commented 1 year ago

Sorry, I did not follow up on this, I will take a look today.

ioquatix commented 1 year ago

I won't get a chance to look at this but I've put it on my planning board.

trevorturk commented 1 year ago

No worries, it's a nice-to-have, but I appreciate your attention in any case!

kinnalru commented 1 year ago

Sometimes when I go back I see this in Service::Supervisor:

    # Restart the process group that the supervisor belongs to.
    def do_restart(message)
        # Tell the parent of this process group to spin up a new process group/container.
        # Wait for that to start accepting new connections.
        # Stop accepting connections.
        # Wait for existing connnections to drain.
        # Terminate this process group.
        signal = message[:signal] || :INT

        Process.kill(signal, Process.ppid)
    end

...but there is no such feature in Service::Application :(

socketry / falcon

Gracefully restart running application #188