search

leap-stc / data-management

Collection of code to manually populate the persistent cloud bucket with data
https://catalog.leap.columbia.edu/
Apache License 2.0
0 stars 5 forks source link

Hack in a way to set `max_num_workers` from label #37

Closed cisaacstern closed 1 year ago

cisaacstern commented 1 year ago

I don't necessarily think this is the best way to set this value, but just wanted to throw together something to hopefully unblock the rate limiting issue discussed in #36.

cisaacstern commented 1 year ago

@jbusecke, with this PR, setting a label of max_num_workers:n will pass the value of n to the pangeo-forge-runner config, as seen here:

https://github.com/leap-stc/data-management/actions/runs/5733238531/job/15537613410?pr=37#step:7:41

Without the label, max_num_workers defaults to 1000.

Going to merge this to hopefully unblock ClimSim. Happy to revise this interface together on follow-on PRs!