search

dagster-io / dagster

An orchestration platform for the development, production, and observation of data assets.
https://dagster.io
Apache License 2.0
11.71k stars 1.48k forks source link

Validate asset definition metadata against metadata access by io managers #14376

Open leozqin opened 1 year ago

leozqin commented 1 year ago

What's the use case?

Dagster allows io managers to access asset definition metadata from the io manager context. However, it does not validate that the metadata provided in the asset definition meets the needs of the metadata that is accessed by the io manager. Therefore, it is possible to define asset definition metadata insufficient for the io manager to successfully run, leading to a runtime error.

Given that the io manager used for an asset may not fully be in control of the asset writer, one way to avoid preventable runtime errors is to validate the asset definition while parsing the Definition, similar to how resource configurations are validated.

Ideas of implementation

Validate the asset definition while parsing the Definition, similar to how resource configurations are validated.

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

jamiedemaria commented 1 year ago

thank you for opening this feature request!

somewhat related to #14375, if I/O managers get separate metadata than the validation would happen for that class