search

theforeman / puppet-pulpcore

Puppet module for setting up Pulp 3 as part of Katello installation
GNU General Public License v3.0
2 stars 28 forks source link

Configure automatic gunicorn worker restart in the pulpcore-api service configuration #285

Closed dralley closed 1 year ago

dralley commented 1 year ago

This will help to address https://bugzilla.redhat.com/show_bug.cgi?id=2122872

Gunicorn provides --max-requests and --max-requests-jitter settings which will restart the web worker process after handling some number of requests. This will help to eliminate any memory leaks which may be present.

I advise to set 

--max-requests 50 \
--max-requests-jitter 30 \

, which will restart workers after handling some random number of requests between 50 and 80.

https://github.com/theforeman/puppet-pulpcore/blob/master/templates/pulpcore-api.service.erb

evgeni commented 1 year ago

this would apply to API only, right? the content app, while also being gunicorn, shouldn't leak?

dralley commented 1 year ago

Yes, that's correct.

I'm testing --max-requests-jitter currently, and actually it does not appear to necessarily be working? --max-requests 80 visibly does work well, but I cannot tell that --max-requests-jitter is doing anything.

evgeni commented 1 year ago

do you call it with --max-requests-jitter J --max-request X or just --max-requests-jitter J?

https://docs.gunicorn.org/en/latest/settings.html#max-requests-jitter is not exactly clear in the wording, but I think that the restarts happen after X+rand(0, J) requests if you provide both, and if you don't provide --max-requests the jitter is just ignored (vs executed at rand(0,J))

dralley commented 1 year ago

Ah, you are correct, you have to provide both.

evgeni commented 1 year ago

How did you get those numbers? They feel very low, but I can't really tell why.

ekohl commented 1 year ago

They do feel very low, because if you serve just 10 requests/second you'll have a restart every 5 seconds. In really doubt you'll have memory leaks in such a short amount of time. You may be trading in better memory consumption for higher CPU usage. I'm not sure which one I'd rather have.

When you have these settings I often see values of 1000 or 10000 requests to start with and then tune them further.

dralley commented 1 year ago

They are pretty low, but unfortunately at the moment it seems we're leaking memory quite quickly during repository index operations.

The API server is easily the least sensitive to this sort of thing, so I think the low limits are OK, though of course we do want to be able to raise them later.

If you look at the BZ it's pretty clear we should prefer the lower memory usage at the moment.

evgeni commented 1 year ago

moving to https://projects.theforeman.org/issues/36438