dagster-io / dagster

An orchestration platform for the development, production, and observation of data assets.
https://dagster.io
Apache License 2.0
11.17k stars 1.4k forks source link

[docs] - Make example in DynamicOut API docs plug and play #5562

Open Martin-Carlsson opened 2 years ago

Martin-Carlsson commented 2 years ago

It would be nice if the example was self-contained (including all ops and imports), had no dependencies on the executing machine, and only focused on the relevant API.

Example:

from dagster import DynamicOut, DynamicOutput, job, op

@op(out=DynamicOut(str))
def output_fruits():
    fruits = ["Mango", "Durian", "Kiwi"]
    for fruit in fruits:
        yield DynamicOutput(fruit, mapping_key=fruit)

@op
def log_single_fruit(context, fruit: str) -> str:
    context.log.info(f"log_single_fruit: {fruit}")
    return fruit

@op
def log_all_fruits(context, fruits):
    context.log.info(f"log_all_fruits: {fruits}")

@job
def process_fruits():
    fruit = output_fruits()
    fruit_results = fruit.map(log_single_fruit)  # pylint: disable=E1101

    log_all_fruits(fruit_results.collect())  # pylint: disable=E1120
yuhan commented 2 years ago

Hi @Martin-Carlsson Thanks for the feedback! When you say dependencies on the executing machine, what exactly did you mean?

Martin-Carlsson commented 2 years ago

@yuhan Yeah, that description didn't make much sense 😄

What I mean by dependencies on the executing machine is that the code example in _apidocs/dynamic#dagster.DynamicOut uses a folder with files. This means that I would need to create a folder with files on the ~executing machine~ machine that runs dagster.

I think the documentation would be easier to follow if the code examples would be more plug and play, without setting other things up on the machine that executes the dagster code.