SwissDataScienceCenter / renku-python

A Python library for the Renku collaborative data science platform.
https://renku-python.readthedocs.io/
Apache License 2.0
37 stars 29 forks source link

Core service scaling: separate RQ workers #3101

Open olevski opened 2 years ago

olevski commented 2 years ago

From #3094

Dealing with the RQ workers:

  • Should we separate out the normal workers with their own ephemeral cache for background jobs and a common queue?
  • Some queues need to be local to each service (those clean up the local cache after ~30 minutes and also clean up the uploaded file cache)
  • if we decide to do the above in a follow-up, we need at least separate redis queues per core-service instance so they don't interfere with each other
Panaetius commented 2 years ago

check #3100 first