search

ocean-transport / coiled_collaboration

Repository to track and link issues across other repositories, which are relevant to the Abernathey Lab and Coiled
0 stars 0 forks source link

Coiled help with workflow to process 'cloud-to-cloud' data #16

Open jbusecke opened 3 years ago

jbusecke commented 3 years ago

Hi everyone,

I have started a (currently very bare bones) repo to figure out a way how people could efficiently generate and save generic 'derived' datasets from the CMIP6 archive here: https://github.com/jbusecke/cmip6_derived_cloud_datasets

I am currently using coiled clusters to do the actual processing. These are started from within github actions. I expect that I will run into more issues in the coming days, and thought it might be a good idea to ping issues here?

This one is my most recent issue. If folks here know a way how to tell coiled to wait until CPUs are freed up instead of failing due to CPU limits, I would greatly appreciate pointers.

mrocklin commented 3 years ago

Hi Julius!

I'm excited about Coiled being used in regular, CI triggered workflows on climate data.

Are you looking for the client.wait_for_workers method perhaps?

I saw that Ryan just responded with this same advice :)

On Mon, Sep 13, 2021 at 11:22 AM Julius Busecke @.***> wrote:

Hi everyone,

I have started a (currently very bare bones) repo to figure out a way how people could efficiently generate and save generic 'derived' datasets from the CMIP6 archive here: https://github.com/jbusecke/cmip6_derived_cloud_datasets

I am currently using coiled clusters to do the actual processing. These are started from within github actions. I expect that I will run into more issues in the coming days, and thought it might be a good idea to ping issues here?

This one https://github.com/jbusecke/cmip6_derived_cloud_datasets/issues/4 is my most recent issue. If folks here know a way how to tell coiled to wait until CPUs are freed up instead of failing due to CPU limits, I would greatly appreciate pointers.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ocean-transport/coiled_collaboration/issues/16, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACKZTEJUCZTNVZXVOGAEH3UBYQLVANCNFSM5D6GOXRA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

jrbourbeau commented 3 years ago

Thanks for raising an issue @jbusecke -- I'm also excited about using Coiled here. I left a comment over in https://github.com/jbusecke/cmip6_derived_cloud_datasets/issues/4

jbusecke commented 3 years ago

I am currently trying to implement saving to my own s3 bucket here. I am basically not sure what the best practices are to pass key/secret for s3 buckets. Is there a way this can be piped through the coiled configuration by any chance?