dagster-io / dagster

An orchestration platform for the development, production, and observation of data assets.
https://dagster.io
Apache License 2.0
11.55k stars 1.45k forks source link

SourceAssets don't support InputManagers #12993

Open danielgafni opened 1 year ago

danielgafni commented 1 year ago

Dagster version

1.2.2

What's the issue?

Why does SourceAsset take io_manager_def or io_manager_key as input, but doesn't support input_manager_def or input_manager_key? I'm writing InputManagers specifically for my SourceAssets, but passing them creates typing errors. Doesn't seem right to me

dagster._check.ParameterCheckError: Param "io_manager_def" is not a IOManagerDefinition. Got <dagster._core.storage.input_manager.InputManagerDefinition object at 0x7fc9f0028e80> which is type <class 'dagster._core.storage.input_manager.InputManagerDefinition'>.

What did you expect to happen?

SourceAssets should support InputManagers since they can only be loaded, not materialized

How to reproduce?

No response

Deployment type

Other

Deployment details

No response

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

tacastillo commented 1 year ago

Hey! Thanks for reporting this.

@smackesey would you be the right person to give a heads-up about this?

smackesey commented 1 year ago

yes I'll investigate

ion-elgreco commented 8 months ago

@smackesey any plans to address this?

I was looking at using the InputManager since it seems would go greatly hand in hand with the concept of a source asset, since you only need to define Handle Input.

Obviously I can do this with a custom IO manager that does pass in handle_output but I would prefer to use the Input Manager for this..

smackesey commented 7 months ago

@smackesey any plans to address this?

Looking today.

czroth commented 5 months ago

Possibly related issue: #21532

ion-elgreco commented 5 months ago

@smackesey any plans to address this?

Looking today.

No success, I guess?