search

ethyca / fidesops

Privacy as Code for DSAR Orchestration: Privacy Request automation to fulfill GDPR, CCPA, and LGPD data subject requests.
https://ethyca.github.io/fidesops
Apache License 2.0
48 stars 16 forks source link

Make Celery `--concurrency` a config option #957

Open seanpreston opened 2 years ago

seanpreston commented 2 years ago

As part of https://github.com/ethyca/fidesops/pull/944 we made some changes to the execution layer pertaining to the number of open DB connections each worker could use at any one time. As part of this we fixed the --concurrency option at 2. Let's make this configurable so it can be overridden depending on a user's infra.

This could be configurable in the future, but this matches our state before the move to Celery. The default is the number of cores on your machine, but limiting the number of PrivacyRequests that can be run simultaneously per worker has a lot of positive effects. This includes reducing the number of simultaneous connections on the application database, as well as the customers' owned databases, and reduces simultaneous requests against their connected API's.

_Originally posted by @pattisdr in https://github.com/ethyca/fidesops/pull/944#discussion_r930980203_

pattisdr commented 2 years ago

Related, let's make things like the connection pool size and max overflow for the fidesops application db configurable

https://github.com/ethyca/fidesops/pull/1107#discussion_r949180257