error with 1.7.14 : 'AssetSpecWithPartitionsDef' object has no attribute 'auto_materialize_policy'

the4thamigo-uk commented 1 month ago

Dagster version

1.7.14

What's the issue?

This appears to be related to the workaround for partitioned external assets suggested in this post https://github.com/dagster-io/dagster/discussions/18211.

It looks like this is a problem caused by:

https://github.com/dagster-io/dagster/blame/master/python_modules/dagster/dagster/_core/definitions/asset_spec.py#L131

We are currently using this workaround, and hit this error on startup.

The above exception was caused by the following exception:
AttributeError: 'AssetSpecWithPartitionsDef' object has no attribute 'auto_materialize_policy'

Stack Trace:
  File "/home/me/company/project/data-pipeline/dagster/pipeline/venv/lib/python3.10/site-packages/dagster/_core/errors.py", line 287, in user_code_error_boundary
    yield
  File "/home/me/company/project/data-pipeline/dagster/pipeline/venv/lib/python3.10/site-packages/dagster/_grpc/server.py", line 245, in __init__
    loadable_targets = get_loadable_targets(
  File "/home/me/company/project/data-pipeline/dagster/pipeline/venv/lib/python3.10/site-packages/dagster/_grpc/utils.py", line 50, in get_loadable_targets
    else loadable_targets_from_python_module(module_name, working_directory)
  File "/home/me/company/project/data-pipeline/dagster/pipeline/venv/lib/python3.10/site-packages/dagster/_core/workspace/autodiscovery.py", line 31, in loadable_targets_from_python_module
    module = load_python_module(
  File "/home/me/company/project/data-pipeline/dagster/pipeline/venv/lib/python3.10/site-packages/dagster/_core/code_pointer.py", line 134, in load_python_module
    return importlib.import_module(module_name)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/me/company/project/data-pipeline/dagster/pipeline/pipeline/__init__.py", line 4, in <module>
    from . import assets, job, definitions
  File "/home/me/company/project/data-pipeline/dagster/pipeline/pipeline/assets/__init__.py", line 8, in <module>
    all_assets, all_ops, all_checks = load_dagster_artifacts(__name__, __file__)
  File "/home/me/company/project/data-pipeline/dagster/pipeline/pipeline/loader/__init__.py", line 43, in load_dagster_artifacts
    module = import_module("." + pkg, package=calling_name)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/home/me/company/project/data-pipeline/dagster/pipeline/pipeline/assets/extract/__init__.py", line 3, in <module>
    all_assets, all_ops, all_checks = load_dagster_artifacts(__name__, __file__)
  File "/home/me/company/project/data-pipeline/dagster/pipeline/pipeline/loader/__init__.py", line 49, in load_dagster_artifacts
    module = import_module("." + module_name, package=calling_name)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/home/me/company/project/data-pipeline/dagster/pipeline/pipeline/assets/extract/freewheel.py", line 23, in <module>
    freewheel_daily_file = create_asset_from_asset_spec_with_partition(
  File "/home/me/company/project/data-pipeline/dagster/pipeline/pipeline/assets/extract/external_asset.py", line 20, in create_asset_from_asset_spec_with_partition
    asset = external_asset_from_spec(
  File "/home/me/company/project/data-pipeline/dagster/pipeline/venv/lib/python3.10/site-packages/dagster/_core/definitions/external_asset.py", line 21, in external_asset_from_spec
    return external_assets_from_specs([spec])[0]
  File "/home/me/company/project/data-pipeline/dagster/pipeline/venv/lib/python3.10/site-packages/dagster/_core/definitions/external_asset.py", line 92, in external_assets_from_specs
    spec.auto_materialize_policy is None,

What did you expect to happen?

dagit should startup cleanly

How to reproduce?

Create a partitioned external asset using the workaround detailed in the linked discussion, and start dagit locally.

Deployment type

Local

Deployment details

No response

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

nickvazz commented 2 weeks ago

If you change the implementation of the partitioned asset spec from:

class AssetSpecWithPartitionsDef(
    namedtuple(
        'AssetSpecWithPartitionsDef',
        (*dagster.AssetSpec._fields, 'partitions_def'),
        defaults=(None,) * (1 + len(dagster.AssetSpec._fields)),
    ),
):
    '''Add partition to AssetSpec (external asset).'''
    __slots__ = ()

to:

class AssetSpecWithPartitionsDef(
    namedtuple(
        'AssetSpecWithPartitionsDef',
        (*dagster.AssetSpec._fields, 'auto_materialize_policy'),
        defaults=(None,) * (1 + len(dagster.AssetSpec._fields)),
    ),
):
    '''Add partition to AssetSpec (external asset).'''
    __slots__ = ()

Then it should work - just ran into this myself

nickvazz commented 2 weeks ago

as of:

python -m pip list | grep -e "dagster"
dagster                   1.8.4
dagster-dask              0.24.4
dagster-graphql           1.8.4
dagster-pipes             1.8.4
dagster-postgres          0.24.4
dagster-slack             0.24.4
dagster-webserver         1.8.4

dagster-io / dagster