quantile-development / dagster-meltano

A Dagster plugin that allows you to run Meltano in Dagster
MIT License
41 stars 17 forks source link

Multiple ways of configuring the Meltano ELT Solid #5

Closed JulesHuisman closed 2 years ago

JulesHuisman commented 2 years ago

There are now three ways of configuring the meltano elt solid.

  1. By defining them directly during the construction of the object
    
    import json
    from dagster import OutputDefinition, Nothing, pipeline
    from dagster_meltano.solids import MeltanoEltSolid

@pipeline def meltano_pipeline(): MeltanoEltSolid( name="tap_csv_target_jsonl", tap="tap-csv", target="target-jsonl", job_id="csv-to-jsonl", tap_config={}, target_config={ "destination_path": "load" }, env_vars={"TAP_CSV__SELECT": json.dumps(["sample.id"])}, ).solid()


2. By injecting them from upstream solids
```python
import json
from dagster import OutputDefinition, Nothing, pipeline
from dagster_meltano.solids import MeltanoEltSolid
from dagster_meltano.dagster_types import MeltanoEltArgsType, MeltanoEnvVarsType

@solid
def elt_args() -> MeltanoEltArgsType:
    return {
        "tap": "tap-csv",
        "target": "target-jsonl",
        "job_id": "csv-to-jsonl",
    }

@solid
def env_vars() -> MeltanoEnvVarsType:
    return {"TAP_CSV__SELECT": json.dumps(["sample.id"])}

@pipeline
def meltano_pipeline():
    MeltanoEltSolid(
        name="tap_csv_target_jsonl",
        tap_config={},
        target_config={
            "destination_path": "load"
        }
    ).solid(
        elt_args=elt_args(), 
        env_vars=env_vars()
    )
  1. Or by configuring them using the Dagster playground (or Dagster configuration in general)
JulesHuisman commented 2 years ago

@jlloyd-widen What do you think about this approach? I refactored the code so that all three approaches of configuration (upstream, hardcoded and dagster-configuration) are possible using the same class.

I still need to add some docstrings, and add tests for the different configuration methods.