The CPFT server is running out of memory over time. We suspect that something in the server code is leaking memory and the celery workers are hanging on to it.
We already set max-tasks-per-child to 1000 but probably because the server is lightly used, the workers are not being replaced very often. As an experiment I've also set max-memory-per-child to 1,000,000 Kib so the total memory consumed by celery should never exceed 4Gb (1/4 of the memory available).
It would be good to identify where the memory is being leaked and an easier additional measure would be to set a sensible default for max-memory-per-child in the sample config file.
The CPFT server is running out of memory over time. We suspect that something in the server code is leaking memory and the celery workers are hanging on to it.
We already set
max-tasks-per-child
to 1000 but probably because the server is lightly used, the workers are not being replaced very often. As an experiment I've also setmax-memory-per-child
to 1,000,000 Kib so the total memory consumed by celery should never exceed 4Gb (1/4 of the memory available).It would be good to identify where the memory is being leaked and an easier additional measure would be to set a sensible default for
max-memory-per-child
in the sample config file.