quantile-development / dagster-meltano

A Dagster plugin that allows you to run Meltano in Dagster
MIT License
41 stars 17 forks source link

Should you be able to import dagster jobs as ops? #36

Closed ReneTC closed 9 months ago

ReneTC commented 10 months ago

Beginner here, But to me many of the meltano jobs are actually an operation instead.

This would allow you to make a nice meltano: extract op, transform op and load op. Then join them together nicely in the UI as a job in dagster

ReneTC commented 9 months ago

Asking because currently the packages just loads a meltano job as an dagster operation, but inside a dagster job. If I have this in my meltano.yml:

jobs:
- name: fetch_raw_data
  tasks:
  - utils:raw_data

We will get a job called "fetch_raw_data", with the op called "utils_raw_data" image

This is great. However, you are often interested in chaining this with another operation in dagster.

As I understand, only dagster OP can be chained, not jobs.

I have 5 Meltano Jobs I am looking to chain together, such that I can press execute somewhere, and the jobs run in a DAG series.

ReneTC commented 9 months ago

Closing as you can chain with by using meltano_command_op

example from readme:

from dagster import repository, job
from dagster_meltano import meltano_resource, meltano_run_op

@job(resource_defs={"meltano": meltano_resource})
def meltano_run_job():
    tap_done = meltano_run_op("tap-1 target-1")()
    meltano_run_op("tap-2 target-2")(tap_done)

@repository()
def repository():
    return [meltano_run_job]
JulesHuisman commented 9 months ago

You are also able to use multiple tasks. Or doesn't that work for your usecase?

jobs:
- name: fetch_raw_data
  tasks:
  - utils:raw_data
  - another:task
ReneTC commented 9 months ago

it works brilliantly. Well done.