ucam-department-of-psychiatry / camcops

Cambridge Cognitive and Psychiatric Test Kit (CamCOPS)
Other
12 stars 8 forks source link

Celery workers grow over time #280

Closed martinburchell closed 1 year ago

martinburchell commented 1 year ago

The CPFT server is running out of memory over time. We suspect that something in the server code is leaking memory and the celery workers are hanging on to it.

We already set max-tasks-per-child to 1000 but probably because the server is lightly used, the workers are not being replaced very often. As an experiment I've also set max-memory-per-child to 1,000,000 Kib so the total memory consumed by celery should never exceed 4Gb (1/4 of the memory available).

It would be good to identify where the memory is being leaked and an easier additional measure would be to set a sensible default for max-memory-per-child in the sample config file.