search

quantile-development / dagster-meltano

A Dagster plugin that allows you to run Meltano in Dagster
MIT License
41 stars 17 forks source link

Selecting #11

Closed jlloyd-widen closed 1 year ago

jlloyd-widen commented 2 years ago

Added meltano select capabilities into the Meltano ELT solid. The architecture path I decided to follow included the abstraction of the cli command core from the actual meltano elt command construction. This was done so that other solid types could be created in the future (e.g., an independent solid for each command in Meltano's cli). The other option would be to include each command as a separate method under the same monolithic class which might be simpler for this implementation. I worried though about future extensibility more than ease of development in this case.

Select commands can be run prior to the meltano elt command by providing a list of select command to run omitting the meltano select tap-whatever portion. This seemed to be the most straight forward way of accomplishing it.

JulesHuisman commented 1 year ago

Moving to a new system of the Dagster software defined assets.