Intending this to stay as draft for a little while so we can take some time to figure out if we're actually interested in prefect. Wanted to explore prefect as an orchestration tool. Could certainly branch off of this just to get a PR for conversion of the pipeline from bash to python, which felt like the better way to do this rather than having lots of subprocess calls to run bash scripts
Enjoyed prefect so far. I'll get a presentation ready for sharing. Nice note is that for local development, there's really no change to user workflow - you just run python scripts via CLI as you would if prefect were not being used, and the general organization of code and logic is pretty similar to just basic python. So little overhead to use. Airflow seems like it would have just a hair more, but I really should give it a shot to actually see
Intending this to stay as draft for a little while so we can take some time to figure out if we're actually interested in prefect. Wanted to explore prefect as an orchestration tool. Could certainly branch off of this just to get a PR for conversion of the pipeline from bash to python, which felt like the better way to do this rather than having lots of
subprocesscalls to run bash scriptsEnjoyed prefect so far. I'll get a presentation ready for sharing. Nice note is that for local development, there's really no change to user workflow - you just run python scripts via CLI as you would if prefect were not being used, and the general organization of code and logic is pretty similar to just basic python. So little overhead to use. Airflow seems like it would have just a hair more, but I really should give it a shot to actually see