Graph In/Out Mapping type alignment removal follow-up

[alangenfeld]

CompositeSolid enforced that types had to be aligned along input/output mapping. In the move to job/graph/op https://github.com/dagster-io/dagster/pull/4876 dropped dagster_type, effectively removing the enforcement.

Other parts of the system were potentially built making implicit assumptions about this type alignment which may require fixes or UX remediations.

Observed issues:

• 6499

  • In this issue a user has a confusing experience due to the graph level type for an input mapping being the default Any while the op end of the mapping is string. Any types don't support the ScalarUnion input loading schema, requiring {'value': ...} resulting in a puzzling user experience.
  • possible fixes
  • smarter coercion for unspecified mapping type instead of falling back to Any. Can do this for 1:1 mappings or when types all align
  • use a type besides Any that doesn't support load from config
• GraphIn does not support dagster_type but you can set one via type hints
  • I believe only allowing the information via typehints was unlikely an explicit design choice.
• dagit
  • How the IO types for graphs are displayed in dagit likely needs to take in to account how we want people to mentally model the existence or lack there of type information at the graph level
• @dpeng817 observed issue during implementation of direct value support in @job
  • During execution plan creation, ensure the appropriate dagster_type is used based on where in the mapping chain the value was specified. May be using wrong type with no longer valid assumption on type alignment.

• 7211

• https://github.com/dagster-io/dagster/issues/7452

[sryza]

Why do all destinations for input mapping need to be the same type? I'm imagining a situation where the same outputs gets routed to inputs of two different ops. The output has 20 columns, but each of the ops only needs 5 columns to be present, and those 5 columns are different between the two ops.

[alangenfeld]

Why do all destinations for input mapping need to be the same type? ... output has 20 columns, but each of the ops only needs 5 columns to be present, and those 5 columns are different between the two ops

Thats a good point. Probably makes sense for the mappings to follow similar enforcement to our input/output type alignment rules aka basically nothing.

Will update the issue.

[alangenfeld]

GraphIn does not support dagster_type but you can set one via type hints I believe only allowing the information via typehints was unlikely an explicit design choice.

Some discussion on this from https://github.com/dagster-io/dagster/pull/7741, summarized:

• we should remove it, but need to be careful not to regress type loader based config value setting
• could try to follow the mappings and infer the type from the destination
• tricky if mixed destination types, especially on accident, maybe want to to be able to explicit set (so maybe don't remove?)