Open treilly1 opened 1 year ago
Thanks for pointing this out, we need to either support this or handle it more gracefully.
Seems like right now even specific-length tuples are not supported in the Pythonic config system?
from dagster import asset, materialize, Config
def test_tuple_config():
class MyConfig(Config):
tuple_field: Tuple[int, int] = (1, 2)
@asset
def my_asset(config: MyConfig) -> int:
return config.tuple_field[0] + config.tuple_field[1]
materialize([my_asset])
raises
Traceback (most recent call last):
File "/home/dan/.local/share/JetBrains/Toolbox/apps/PyCharm-P/ch-0/222.4459.20/plugins/python/helpers/pycharm/_jb_unittest_runner.py", line 35, in <module>
sys.exit(main(argv=args, module=None, testRunner=unittestpy.TeamcityTestRunner, buffer=not JB_DISABLE_BUFFERING))
File "/home/dan/.pyenv/versions/3.10.10/lib/python3.10/unittest/main.py", line 100, in __init__
self.parseArgs(argv)
File "/home/dan/.pyenv/versions/3.10.10/lib/python3.10/unittest/main.py", line 147, in parseArgs
self.createTests()
File "/home/dan/.pyenv/versions/3.10.10/lib/python3.10/unittest/main.py", line 158, in createTests
self.test = self.testLoader.loadTestsFromNames(self.testNames,
File "/home/dan/.pyenv/versions/3.10.10/lib/python3.10/unittest/loader.py", line 220, in loadTestsFromNames
suites = [self.loadTestsFromName(name, module) for name in names]
File "/home/dan/.pyenv/versions/3.10.10/lib/python3.10/unittest/loader.py", line 220, in <listcomp>
suites = [self.loadTestsFromName(name, module) for name in names]
File "/home/dan/.pyenv/versions/3.10.10/lib/python3.10/unittest/loader.py", line 205, in loadTestsFromName
test = obj()
File "/home/dan/Projects/sanas/science/mlops/feature-extractors/teacher-whisper/tests/test_definitions.py", line 23, in test_tuple_config
def my_asset(config: MyConfig) -> int:
File "/home/dan/Projects/sanas/science/mlops/feature-extractors/teacher-whisper/.venv/lib/python3.10/site-packages/dagster/_core/definitions/decorators/asset_decorator.py", line 208, in asset
return create_asset()(compute_fn)
File "/home/dan/Projects/sanas/science/mlops/feature-extractors/teacher-whisper/.venv/lib/python3.10/site-packages/dagster/_core/definitions/decorators/asset_decorator.py", line 346, in __call__
op = _Op(
File "/home/dan/Projects/sanas/science/mlops/feature-extractors/teacher-whisper/.venv/lib/python3.10/site-packages/dagster/_core/definitions/decorators/op_decorator.py", line 105, in __call__
self.config_schema = infer_schema_from_config_annotation(
File "/home/dan/Projects/sanas/science/mlops/feature-extractors/teacher-whisper/.venv/lib/python3.10/site-packages/dagster/_config/pythonic_config/__init__.py", line 1673, in infer_schema_from_config_annotation
return infer_schema_from_config_class(model_cls)
File "/home/dan/Projects/sanas/science/mlops/feature-extractors/teacher-whisper/.venv/lib/python3.10/site-packages/dagster/_config/pythonic_config/__init__.py", line 1715, in infer_schema_from_config_class
fields[pydantic_field.alias] = _convert_pydantic_field(
File "/home/dan/Projects/sanas/science/mlops/feature-extractors/teacher-whisper/.venv/lib/python3.10/site-packages/dagster/_config/pythonic_config/__init__.py", line 1487, in _convert_pydantic_field
inner_field = _get_inner_field_if_exists(pydantic_field.shape, pydantic_field)
File "/home/dan/Projects/sanas/science/mlops/feature-extractors/teacher-whisper/.venv/lib/python3.10/site-packages/dagster/_config/pythonic_config/__init__.py", line 1448, in _get_inner_field_if_exists
raise NotImplementedError(f"Pydantic shape type {shape_type} not supported.")
NotImplementedError: Pydantic shape type 5 not supported.
Process finished with exit code 1
Empty suite
cc @benpankow
still seeing issues with tuples not being supported: Trying to define this list of tuples results in:
class EvalConfig(Config):
eval_benchmarks: List[Tuple[str, int]] = [] # list of tuples of (benchmark, eval_k)
dagster._core.errors.DagsterInvalidDefinitionError: You have passed in typing.Tuple[str, int] to the config system. Types from the typing module in python are not allowed in the config system. You must use types defined by dagster or primitives
for context, this is running a locally deployed dagster deployment on my k8s cluster. @smackesey any ideas on what might be causing this issue?
@smackesey & @benpankow will this be resolved?
What's the use case?
According to PEP 484 type hints
However, when using that syntax dagster throws error
Ideas of implementation
No response
Additional information
using dagster 1.0.10
Message from the maintainers
Impacted by this issue? Give it a 👍! We factor engagement into prioritization.