search

microsoft / yardl

Tooling for streaming instrument data
https://microsoft.github.io/yardl/
MIT License
29 stars 5 forks source link

Python RecursionError with aliased generics #81

Closed naegelejd closed 8 months ago

naegelejd commented 8 months ago

Using yardl commit ae9b826 with the following model:

GenericRecord<T>: !record
  fields:
    v: T

AliasedRecord<T>: GenericRecord<T>

AliasedOpenGeneric<T>: AliasedRecord<T>
AliasedClosedGeneric: AliasedRecord<string>

To reproduce, generate Python for this model, then import the generated Python module. The module won't import, failing with the following error:

Traceback (most recent call last):
  File "/workspaces/yardl/joe/models/bug/python/test.py", line 1, in <module>
    import bug
  File "/workspaces/yardl/joe/models/bug/python/bug/__init__.py", line 21, in <module>
    from .types import (
  File "/workspaces/yardl/joe/models/bug/python/bug/types.py", line 56, in <module>
    get_dtype = _mk_get_dtype()
                ^^^^^^^^^^^^^^^
  File "/workspaces/yardl/joe/models/bug/python/bug/types.py", line 52, in _mk_get_dtype
    dtype_map[AliasedClosedGeneric] = get_dtype(types.GenericAlias(AliasedRecord, (str,)))
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/yardl/joe/models/bug/python/bug/_dtypes.py", line 107, in <lambda>
    return lambda t: get_dtype_impl(dtype_map, t)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/yardl/joe/models/bug/python/bug/_dtypes.py", line 90, in get_dtype_impl
    return res(get_args(t))
           ^^^^^^^^^^^^^^^^
  File "/workspaces/yardl/joe/models/bug/python/bug/types.py", line 51, in <lambda>
    dtype_map[AliasedOpenGeneric] = lambda type_args: get_dtype(types.GenericAlias(AliasedRecord, (type_args[0],)))
                                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...
RecursionError: maximum recursion depth exceeded in comparison

For this particular model, reordering the aliases as shown below can eliminate the error, but this is a confusing limitation for the user. 

AliasedClosedGeneric: AliasedRecord<string>
AliasedOpenGeneric<T>: AliasedRecord<T>
naegelejd commented 8 months ago

The source of recursion (shown in stacktrace above) is at the bottom of the generated types.py:

AliasedRecord = GenericRecord[T]

AliasedOpenGeneric = AliasedRecord[T]

AliasedClosedGeneric = AliasedRecord[str]

def _mk_get_dtype():
    dtype_map: dict[typing.Union[type, types.GenericAlias], typing.Union[np.dtype[typing.Any], typing.Callable[[tuple[type, ...]], np.dtype[typing.Any]]]] = {}
    get_dtype = _dtypes.make_get_dtype_func(dtype_map)

    dtype_map[GenericRecord] = lambda type_args: np.dtype([('v', get_dtype(type_args[0]))], align=True)
    dtype_map[AliasedRecord] = lambda type_args: get_dtype(types.GenericAlias(GenericRecord, (type_args[0],)))
    dtype_map[AliasedOpenGeneric] = lambda type_args: get_dtype(types.GenericAlias(AliasedRecord, (type_args[0],)))
    dtype_map[AliasedClosedGeneric] = get_dtype(types.GenericAlias(AliasedRecord, (str,)))

    return get_dtype

The problem is that AliasedOpenGeneric == AliasedRecord, so we overwrite the dtype_map lambda, recursing without ever resolving to the final dtype of GenericRecord.

I'll try two quick fixes:

  1. Avoid generating a dtype mapping for aliased types (doing so for the base type is sufficient), or
  2. Only generate a dtype mapping if it hasn't already been added to dtype_map