pgulley
commented
6 months ago First steps in a recent commit- we CAN load a recipe directory directly from S3 via a new entry point. That entry point could use some massaging. Next step would be to create a sous-chef prefect deployment file that's JUST sous-chef. Then, we might want a script to make uploading recipes to s3 more straightforward.
Implementing this versioning system makes me feel that this stack is slightly over-convoluted as it stands- especially having a whole separate deployment for each job. A simpler way to do it would be to have a single deployment for each entry point (conceivably in a single prefect.yaml), and store recipe configuration stuff in S3 or another external storage. This is a significant refactor of the approach I've been using so far, but I think it's a preferable separation of concerns.