klay-music / klay-beam

Our Apache Beam Transforms and Pipelines
1 stars 0 forks source link

Troubleshoot memory consumption #34

Closed CharlesHolbrow closed 1 year ago

CharlesHolbrow commented 1 year ago

I ran a hyperparameter search to help me understand and optimize worker node parallelism and CPU+memory consumption. This commit history is mostly just a documentation of the specific parameters used in the process. It includes a few minor changes as well.

More details about the search are in Notion.