Running pipelines on AWS? - Githubissues

google / xarray-beam

Distributed Xarray with Apache Beam

https://xarray-beam.readthedocs.io

Apache License 2.0

124 stars 7 forks source link

Running pipelines on AWS? #42

Open rsignell-usgs opened 2 years ago

rsignell-usgs commented 2 years ago

Hello! Newbie here. :) Any tips for someone who would like to try an xarray-beam rechunking pipeline on AWS?
(USGS is wedded to AWS at the moment)

alxmrs commented 2 years ago

Sorry for the late reply. I'm not too familiar with how we can run this on AWS (I haven't done it myself), but here's how I would do it:

Set up a (py)Spark cluster in AWS EMR, see the AWS docs or this step-by-step Medium post.
Specify the SparkRunner in the --runners flag of your python Beam Pipeline.
- See also: https://beam.apache.org/documentation/runners/spark/#running-on-a-pre-deployed-spark-cluster
- Likely, you'll need to install apache_beam with AWS's extra requirements: pip install apache_beam[aws].

I hope that helps.