[Ingest] Change Worker pool size and Download thread count (EFS performance related)

CDLUC3 / mrt-doc

Documentation and Information regarding the Merritt repository

8 stars 4 forks source link

Closed mreyescdl closed 2 years ago

mreyescdl commented 2 years ago

EFS performance issue with UCSF Glantz and large UCB submission. Analysis for root cause is still ongoing, but it seems to be related to:

Short term fix is to reduce the processing capacity for each Ingest worker:

Worker thread count is SSM var
Download thread count is hard coded (part of this tix is to make this an SSM var)

mreyescdl commented 2 years ago

STAGE Created new SSM parm /uc3/mrt/stg/ingest/config/numDownloadThreads 3

Altered parm /uc3/mrt/stg/ingest/config/numThreads 3

Deployed to stage tag: 0.0.26.dev01 (threadPool branch)

elopatin-uc3 commented 2 years ago

The current plan is to make this change in production when Mark is back (after 6/16).

mreyescdl commented 2 years ago

PRODUCTION Created new SSM parm /uc3/mrt/prd/ingest/config/numDownloadThreads 3

Altered parm /uc3/mrt/prd/ingest/config/numThreads 3

Will deploy using tag 0.0.26

mreyescdl commented 2 years ago