Move batch loads to slow worker

internetstandards / Internet.nl

Internet standards compliance test suite

https://internet.nl

164 stars 36 forks source link

Move batch loads to slow worker #1420

Open aequitas opened 1 month ago

aequitas commented 1 month ago

To solve memory issues on batch installations:

Move batch_slow queue to slow worker
Make concurrency for slow worker configurable and reduce to ~1~ 2

baknu commented 1 month ago

Two remarks:

Note that according to an older mail from GT (subject: "worker_slow configuration" / dated 3 December 2021), the concurrency for the slow worker that handled the batch_slow queue with the async_register and async_generate_results was 2.
We should also make sure that each of the workers gets sufficient max memory. The total available memory on each of the Docker based batch servers (dev-docker.batch and docker.batch) is currently 16 GB.

aequitas commented 1 month ago

PR https://github.com/internetstandards/Internet.nl/pull/1426 moves batch_slow to slow worker and reduces concurrency on that worker.

Memory for both the normal and slow worker can be configured already in local.env by overriding the WORKER_MEMORY_LIMIT and WORKER_SLOW_MEMORY_LIMIT respectively.