I have been very happily using the dagster extension with meltano integration now for at least 6 months, very stable. I am running it in a single docker instance on Azure ACS. Dagster also orchestrates my downstream DBT models, I use a dbt group with the same name as my meltano job and everything flows very nicely.
One issue I have discovered is that I cannot reload my code definitions without restarting the entire docker container. This interrupts any ongoing runs. So a single line change in a dbt model means a full tear down of the container,
I know with dagster it is possible to have your code locations separate to your main container, and just update your code, without interrupting running jobs. But is this workflow even possible using this extension, given that dagster sits 'under' the main meltano install?
I could definitely live with always having to do a full redeploy if my meltano.yml changes, as this mostly stays the same. But being able to do the 'reload definitions' for minor dbt tweaks would be a game changer.
I have been very happily using the dagster extension with meltano integration now for at least 6 months, very stable. I am running it in a single docker instance on Azure ACS. Dagster also orchestrates my downstream DBT models, I use a dbt group with the same name as my meltano job and everything flows very nicely.
One issue I have discovered is that I cannot reload my code definitions without restarting the entire docker container. This interrupts any ongoing runs. So a single line change in a dbt model means a full tear down of the container,
I know with dagster it is possible to have your code locations separate to your main container, and just update your code, without interrupting running jobs. But is this workflow even possible using this extension, given that dagster sits 'under' the main meltano install?
I could definitely live with always having to do a full redeploy if my meltano.yml changes, as this mostly stays the same. But being able to do the 'reload definitions' for minor dbt tweaks would be a game changer.