dagster-io / dagster

An orchestration platform for the development, production, and observation of data assets.
https://dagster.io
Apache License 2.0
10.99k stars 1.37k forks source link

Nested Resources in Jobs result in CheckError #22995

Open AgentK9 opened 1 month ago

AgentK9 commented 1 month ago

Dagster version

1.7.13

What's the issue?

I cannot use nested resources in jobs, even though it is implied to be supported here: https://docs.dagster.io/concepts/resources#resources-that-depend-on-other-resources

What did you expect to happen?

I would not get an error when running the below code.

How to reproduce?

from dagster import ConfigurableResource, ResourceDependency, job, op

class CredentialResource(ConfigurableResource):
    username: str
    password: str

class ServiceResource(ConfigurableResource):
    creds: ResourceDependency[CredentialResource]

@op
def basic_op(service_1: ServiceResource) -> None:
    print(f"{service_1.creds.username} {service_1.creds.password}")

@job
def basic_job() -> None:
    basic_op()

_cred_resource = CredentialResource.configure_at_launch()

basic_job.execute_in_process(
    run_config={
        "resources": {
            "creds": {
                "config": {
                    "username": "user",
                    "password": "pass"
                }
            }
        }
    },
    resources={
        "creds": _cred_resource,
        "service_1": ServiceResource(
            creds=_cred_resource
        )
    }
)

Deployment type

Local

Deployment details

N/A - the above code runs anywhere with dagster installed

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

garethbrickman commented 1 month ago

For clarity, this is the kind of error encountered when trying to load the code above:

dagster._check.CheckError: Object <dagster._core.execution.context.init.InitResourceContext object at 0x106031a90> is not a InitResourceContextWithKeyMapping. Got <dagster._core.execution.context.init.InitResourceContext object at 0x106031a90> with type <class 'dagster._core.execution.context.init.InitResourceContext'>. This ConfiguredResource contains unresolved partially-specified nested resources, and so can only be initialized using a InitResourceContextWithKeyMapping