Open jrouly opened 1 year ago
One other potential angle on this -- when you're looking at a subset of the asset graph, and click on one asset that's part of the multi-asset, all assets within that multi-asset could be highlighted (assuming the multi-asset is not subsettable). This would make it much easier to reliably select the desired assets from those views.
What's the use case?
We have a
@multi_asset
(def featurize
) that produces multiple outputs (imputed_features
andfeatures
). Both of these outputs are always (unconditionally) materialized from thefeaturize
multi-asset.We have modeled the asset something like this:
When you load Dagit and go to materialize
featurize
and both of its outputs, you can't! Instead, you can only seeimputed_features
andfeatures
in the asset catalog. Which means, when you want to materializefeaturize
, you have to select bothimputed_features
andfeatures
to materialize. Else, Dagit will present you with an error (since both outputs are required).This is pretty inconvenient for large asset graphs -- you have to know all of the outputs for a multi-asset, and how many may/may not be required, before you can even make a selection and hit materialize.
What we would very much like to be able to do is simply "materialize
featurize
". In other words, see the multi-asset itself in Dagit in the asset graph, with the outputs linked to it. It would probably look very similar to the way asset groups look. And then be able to materialize all of the outputs together.subsets
Of note, we can achieve the desired behavior by (not super intuitively) making the asset outputs optional.
In this case, you can materialize one or the other output, and then both outputs will be materialized. However, there are some drawbacks to this approach.
imputed_features
for materialization, the run display and run history both only includeimputed_features
, even thoughfeatures
also gets materialized.features
also gets materialized shows up as awarning
in the logs, that an "unexpected asset" is being materialized.Ideas of implementation
Treat multi-assets more like asset groups in Dagit. Allow the
@multi_asset
itself to be featurized as a whole.It's important to have
@multi_asset
distinct from an asset group, because the code that is generating these outputs cannot be teased out into separate assets. Otherwise we'd use an asset group. But being able to "materialize all" for a@multi_asset
, like we can for asset groups, would be ideal.Additional information
No response
Message from the maintainers
Impacted by this issue? Give it a 👍! We factor engagement into prioritization.