quantile-development / dagster-meltano

A Dagster plugin that allows you to run Meltano in Dagster
MIT License
41 stars 17 forks source link

meltano_run_op: expected value to be of Python type dict. #46

Closed ReneTC closed 6 months ago

ReneTC commented 9 months ago

Again I am really sorry spamming the repo. But I do think this is a bug.

I have to jobs:

jobs:
- name: fetch_raw_data
  tasks:
  - utils:raw_data
- name: raw_data_to_duckdb
  tasks:
  - tap-spreadsheets-anywhere target-duckdb

They work perfectly and can even be executed with this dagster with load_jobs_from_meltano_project() from this repo.

However, if I make these two as meltano_run_op() instead:

@job(resource_defs={"meltano": meltano_resource})
def meltano_etl():
    raw_data = meltano_run_op("utils:raw_data")()
    make_duck_db = meltano_run_op("tap-spreadsheets-anywhere target-duckdb")(raw_data)

@repository
def dev_meltano_etl():
    return [
        meltano_etl
    ]

Second op will fail, with dagster returning: dagster._core.errors.DagsterTypeCheckDidNotPass: Type check failed for step input "env" - expected type "dict?". Description: Value of type <class 'str'> failed type check for Dagster type dict, expected value to be of Python type dict.