dagster-io / dagster

An orchestration platform for the development, production, and observation of data assets.
https://dagster.io
Apache License 2.0
11.73k stars 1.48k forks source link

Lauchpad scaffolded configuration doesn't match selected Asset #18268

Open HynekBlaha opened 11 months ago

HynekBlaha commented 11 months ago

Dagster version

dagster, version 1.5.9

What's the issue?

Hello, I came across an issue with Launchpad config scaffolding. In the first example, there is extra configuration. In the second example, there is missing configuration.

Extra configuration

When selecting Open launchpad for credentials_asset, run config shows extra section for bucket: FileStoreBucket (used only in bucket_asset ).

from dagster import Definitions, ConfigurableResource, asset, get_dagster_logger

log = get_dagster_logger()

class CredentialsResource(ConfigurableResource):
    username: str
    password: str

class FileStoreBucket(ConfigurableResource):
    bucket_id: str
    credentials: CredentialsResource

@asset
def credentials_asset(context, credentials: CredentialsResource):
    context.log.warning(credentials.model_dump_json())

@asset
def bucket_asset(context, bucket: FileStoreBucket):
    context.log.warning(bucket.model_dump_json())

credentials = CredentialsResource(username="my_username", password="my_password")
bucket = FileStoreBucket(bucket_id="my_bucket_id", credentials=credentials)

defs = Definitions(
    assets=[credentials_asset, bucket_asset],
    resources={
        "credentials": credentials,
        "bucket": bucket,
    },
)
Screenshot 2023-11-27 at 12 31 35

Missing configuration

In this example, the Lauchpad UI shows only configuration for credentials: CredentialsResource, but configuration for bucket1, bucket2: FileStoreBucket are missing

from dagster import Definitions, ConfigurableResource, asset, AssetExecutionContext, get_dagster_logger

log = get_dagster_logger()

class CredentialsResource(ConfigurableResource):
    username: str
    password: str

class FileStoreBucket(ConfigurableResource):
    bucket_id: str
    credentials: CredentialsResource

@asset
def multibucket_asset(context: AssetExecutionContext, bucket1: FileStoreBucket, bucket2: FileStoreBucket):
    context.log.debug(f"{id(bucket1.credentials)=}")
    context.log.debug(f"{id(bucket2.credentials)=}")

credentials = CredentialsResource(username="username", password="password")
bucket1 = FileStoreBucket(bucket_id="bucket_1", credentials=credentials)
bucket2 = FileStoreBucket(bucket_id="bucket_2", credentials=credentials)

defs = Definitions(
    assets=[multibucket_asset],
    resources={
        "credentials": credentials,
        "bucket1": bucket1,
        "bucket2": bucket2,
    },
)
Screenshot 2023-11-27 at 12 48 54

What did you expect to happen?

I would expect to see run materialization configuration only for resources it depends on.

How to reproduce?

Run code mentioned above.

Deployment type

Local

Deployment details

No response

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

smackesey commented 11 months ago

@hellendag @bengotow