search

USRSE / monthly-community-calls

Location for gathering community call topics
MIT License
4 stars 0 forks source link

Workflow tools #11

Open christophernhill opened 2 years ago

christophernhill commented 2 years ago

What workflow tools are people using, learning. What features are useful, what are missing.

Aariq commented 2 years ago

I've been learning and using the targets R package for workflow management, and I really love it. It's not language agnostic, of course, but it's really nice for a lot of other reasons including relatively easy setup to work with HPC.

jmelot commented 1 year ago

We use Apache Airflow and Beam on the google cloud platform and although they are not without their quirks they have worked well for us for automating and scaling ETL, model predictions, etc. for a few years now. I'd definitely be curious to hear about other people's data workflows/pipelines!