search

leap-stc / data-management

Collection of code to manually populate the persistent cloud bucket with data
https://catalog.leap.columbia.edu/
Apache License 2.0
0 stars 5 forks source link

Redeploy ClimSim with only 50 workers, due to caching errors #38

Closed cisaacstern closed 1 year ago

cisaacstern commented 1 year ago

See discussion in #36.

cisaacstern commented 1 year ago

The test Dataflow jobs deployed from this PR were correctly assigned max_num_workers=50 (based on the label here).

Screen Shot 2023-08-01 at 7 09 19 PM

but I now realize that this label will not be discoverable (with the current implementation from #37) on push events... 🤔

cisaacstern commented 1 year ago

Ok! The rewritten workflow logic successfully deployed the test jobs to dataflow with max_num_workers=50. I believe this logic should now work for push events as well, so going to merge this and hope we get the desired result (production runs of both recipes with only 50 workers each).