search

pangeo-forge / pangeo-forge-runner

Run pangeo-forge recipes on Apache Beam
https://pangeo-forge-runner.readthedocs.io
Apache License 2.0
9 stars 9 forks source link

Try Flink on HPC #128

Open cisaacstern opened 1 year ago

cisaacstern commented 1 year ago

With #114, we should be able to run Pangeo Forge on Flink on HPC. This may not be quite as simple as an eventual Dask deployment, but #109 is realistically a longer-term project, so Flink is likely our best option for now.

Tagging @negin513 and @TomNicholas as possibly interested in trying this out at NCAR. If either of you do have interest in this, #124 should be helpful once it goes in.

yuvipanda commented 1 year ago

Note that I don't think we should try flink on HPC, described as more traditional slurm-like setup.

I remember @negin513 mentioning they may have a kubernetes cluster - so this should be about trying flink on an on prem kubernetes cluster, rather than any HPC system. The on-prem cluster may be run by a HPC center though :)

cisaacstern commented 1 year ago

Important clarification, thanks @yuvipanda!