[1.2.4] Can only serialize whitelisted namedtuples, received DimensionPartitionMapping

drawnwren commented 1 year ago

Dagster version

1.2.4

What's the issue?

Are MultiPartitionMappings officially implemented? The first bug is that DimensionPartitionMapping is not exported from dagster.

The second is that, when attempting to use MultiPartitionMapping as below:

MultiPartitionMapping(
                {
                    "days": DimensionPartitionMapping(
                        dimension_name="days",
                        partition_mapping=TimeWindowPartitionMapping(
                            start_offset=-window, end_offset=0
                        )
                    ),
                    "instrument": DimensionPartitionMapping(
                        dimension_name="instrument",
                        partition_mapping=IdentityPartitionMapping()
                    ),
                }
            )

I get the following SerializationError:

dagster._serdes.errors.SerializationError: Can only serialize whitelisted namedtuples, received DimensionPartitionMapping(dimension_name='days', partition_mapping=TimeWindowPartitionMapping(start_offset=-3, end_offset=0)).
Descent path: <root:dict>.days
  File "/home/wing/.cache/pypoetry/virtualenvs/syncdag-NSDSaiv4-py3.11/lib/python3.11/site-packages/dagster/_grpc/server.py", line 528, in _get_serialized_external_repository_data
    return serialize_value(
           ^^^^^^^^^^^^^^^^
  File "/home/wing/.cache/pypoetry/virtualenvs/syncdag-NSDSaiv4-py3.11/lib/python3.11/site-packages/dagster/_serdes/serdes.py", line 434, in serialize_value
    packed_value = pack_value(val, whitelist_map=whitelist_map)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wing/.cache/pypoetry/virtualenvs/syncdag-NSDSaiv4-py3.11/lib/python3.11/site-packages/dagster/_serdes/serdes.py", line 479, in pack_value
    return _pack_value(val, whitelist_map=whitelist_map, descent_path=_root(val))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wing/.cache/pypoetry/virtualenvs/syncdag-NSDSaiv4-py3.11/lib/python3.11/site-packages/dagster/_serdes/serdes.py", line 495, in _pack_value
    return serializer.pack(val, whitelist_map, descent_path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wing/.cache/pypoetry/virtualenvs/syncdag-NSDSaiv4-py3.11/lib/python3.11/site-packages/dagster/_serdes/serdes.py", line 393, in pack
    packed[storage_key] = pack_value(inner_value, whitelist_map, f"{descent_path}.{key}")
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wing/.cache/pypoetry/virtualenvs/syncdag-NSDSaiv4-py3.11/lib/python3.11/site-packages/dagster/_serdes/serdes.py", line 479, in pack_value
    return _pack_value(val, whitelist_map=whitelist_map, descent_path=_root(val))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wing/.cache/pypoetry/virtualenvs/syncdag-NSDSaiv4-py3.11/lib/python3.11/site-packages/dagster/_serdes/serdes.py", line 510, in _pack_value
    return [
           ^
  File "/home/wing/.cache/pypoetry/virtualenvs/syncdag-NSDSaiv4-py3.11/lib/python3.11/site-packages/dagster/_serdes/serdes.py", line 511, in <listcomp>
    _pack_value(item, whitelist_map, f"{descent_path}[{idx}]")
  File "/home/wing/.cache/pypoetry/virtualenvs/syncdag-NSDSaiv4-py3.11/lib/python3.11/site-packages/dagster/_serdes/serdes.py", line 495, in _pack_value
    return serializer.pack(val, whitelist_map, descent_path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wing/.cache/pypoetry/virtualenvs/syncdag-NSDSaiv4-py3.11/lib/python3.11/site-packages/dagster/_serdes/serdes.py", line 393, in pack
    packed[storage_key] = pack_value(inner_value, whitelist_map, f"{descent_path}.{key}")
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wing/.cache/pypoetry/virtualenvs/syncdag-NSDSaiv4-py3.11/lib/python3.11/site-packages/dagster/_serdes/serdes.py", line 479, in pack_value
    return _pack_value(val, whitelist_map=whitelist_map, descent_path=_root(val))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wing/.cache/pypoetry/virtualenvs/syncdag-NSDSaiv4-py3.11/lib/python3.11/site-packages/dagster/_serdes/serdes.py", line 510, in _pack_value
    return [
           ^

What did you expect to happen?

No response

How to reproduce?

No response

Deployment type

None

Deployment details

No response

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

sryza commented 1 year ago

@clairelin135

clairelin135 commented 1 year ago

Thanks for reporting @DrawnWren. https://github.com/dagster-io/dagster/pull/13307 will fix in the next release

dagster-io / dagster