Python 3.9 `typing` raises error on `Generic` usage in `AsyncToSync` class

[pmayostendorp]

Issue

asgiref 3.7.0 raises a TypeError when loading certain components under Python 3.9. On import, the typing module complains in the following manner:

    from asgiref.sync import sync_to_async
File "/usr/local/lib/python3.9/site-packages/asgiref/sync.py", line 130, in <module>
    class AsyncToSync(Generic[_P, _R]):
File "/usr/local/lib/python3.9/typing.py", line 243, in inner
    return func(*args, **kwds)
File "/usr/local/lib/python3.9/typing.py", line 917, in __class_getitem__
    raise TypeError(
TypeError: Parameters to Generic[...] must all be type variables

Steps to Reproduce

1. Open a shell in Python 3.9
2. Try to from asgiref.sync import sync_to_async

https://github.com/django/asgiref/blob/89cfe10fd58eca207c3f9db2bbfee5d7b8bd1108/asgiref/sync.py#L130

[andrewgodwin]

I cannot replicate this on 3.9 - once I correct the typo in your command and type from asgiref.sync import sync_to_async, the import works perfectly fine. (Our test matrix also runs on 3.9 and would have caught an outright error like this!)

Can I get a list of all the packages installed in the environment you are trying to install it in, the logs from the installation process, and an idea of where your Python installation is sourced from?

[pmayostendorp]

Sure, thanks for the help and yeah, sorry for autocorrect. Wants to turn asgiref -> aspired. We resolved our issue by pinning asgiref to v6, but imagine others are going to encounter this one.

The base image here is python:3.9.0-buster. This is in the context of a Django app with Channels 4.0.

Here is a pip list from the container running our unit test pipeline. I was able to reproduce locally as well derived from nearly identical pip requirements.

aiofiles                         23.1.0
aiohttp                          3.8.3
aiosignal                        1.3.1
aiosqlite                        0.19.0
amqp                             5.1.1
anyio                            3.2.1
appnope                          0.1.2
argon2-cffi                      20.1.0
asgiref                          3.7.0
async-generator                  1.10
async-timeout                    4.0.2
asyncio                          3.4.3
asyncua                          0.9.98
attrs                            21.3.0
autobahn                         23.1.2
Automat                          22.10.0
Babel                            2.9.1
backcall                         0.2.0
billiard                         3.6.4.0
bleach                           3.3.0
Brotli                           1.0.9
celery                           5.2.3
certifi                          2021.5.30
cffi                             1.14.5
channels                         4.0.0
channels-redis                   4.0.0
chardet                          4.0.0
charset-normalizer               2.1.1
click                            8.1.3
click-didyoumean                 0.3.0
click-plugins                    1.1.1
click-repl                       0.2.0
cligj                            0.7.2
constantly                       15.1.0
coverage                         7.2.5
crispy-bootstrap5                0.6
cryptography                     37.0.2
cycler                           0.10.0
daphne                           4.0.0
decorator                        5.0.9
defusedxml                       0.7.1
descartes                        1.1.0
Django                           3.2.6
django-bootstrap-datepicker-plus 4.0.0
django-celery-beat               2.2.1
django-cors-headers              3.7.0
django-crispy-forms              1.14.0
django-environ                   0.7.0
django-extensions                3.1.3
django-filter                    21.1
django-json-widget               1.1.1
django-permissions-auditor       1.1.0
django-redis                     5.0.0
django-timezone-field            4.2.3
django-vectortiles               0.1.0
djangorestframework              3.12.2
djangorestframework-gis          0.17
drf-spectacular                  0.13.2
entrypoints                      0.3
exceptiongroup                   1.1.1
Fiona                            1.8.20
frozenlist                       1.3.3
future                           0.18.3
geojson                          2.5.0
gunicorn                         19.9.0
hardhat                          22.36.1
hyperlink                        21.0.0
idna                             2.10
importlib-metadata               4.5.0
incremental                      22.10.0
inflection                       0.5.1
iniconfig                        1.1.1
ipykernel                        5.5.5
ipython                          7.24.1
ipython-genutils                 0.2.0
ipywidgets                       7.6.3
jedi                             0.18.0
Jinja2                           3.0.1
joblib                           1.0.1
json5                            0.9.6
jsonschema                       3.2.0
jupyter                          1.0.0
jupyter-client                   6.1.12
jupyter-console                  6.4.0
jupyter-core                     4.7.1
jupyter-server                   1.9.0
jupyterlab                       3.0.16
jupyterlab-pygments              0.1.2
jupyterlab-server                2.6.0
jupyterlab-widgets               1.0.0
kiwisolver                       1.3.1
kombu                            5.2.4
lxml                             4.9.2
MarkupSafe                       2.0.1
matplotlib                       3.4.2
matplotlib-inline                0.1.2
mercantile                       1.2.1
mistune                          2.0.1
msgpack                          1.0.5
multidict                        6.0.4
munch                            2.5.0
nbclassic                        0.3.1
nbclient                         0.5.3
nbconvert                        5.5.0
nbformat                         5.1.3
nest-asyncio                     1.5.1
networkx                         2.7
notebook                         6.4.0
numpy                            1.22.0
opcua                            0.98.13
packaging                        20.9
pandas                           1.2.5
pandocfilters                    1.4.3
parso                            0.8.2
pexpect                          4.8.0
pickleshare                      0.7.5
Pillow                           8.2.0
pip                              21.3.1
pluggy                           0.13.1
ply                              3.11
progressbar                      2.5
prometheus-client                0.11.0
prompt-toolkit                   3.0.19
psycopg2                         2.9.1
ptyprocess                       0.7.0
py                               1.10.0
pyasn1                           0.5.0
pyasn1-modules                   0.3.0
pycparser                        2.20
Pygments                         2.9.0
PyJWT                            2.4.0
Pyomo                            6.0.1
pyOpenSSL                        22.0.0
pyparsing                        2.4.7
pyproj                           3.1.0
pyrsistent                       0.17.3
pytest                           7.2.0
pytest-asyncio                   0.20.1
pytest-cov                       4.0.0
pytest-django                    4.5.2
python-crontab                   2.7.1
python-dateutil                  2.8.1
pytz                             2022.1
PyYAML                           6.0
pyzmq                            22.1.0
qtconsole                        5.1.0
QtPy                             1.9.0
redis                            4.5.5
requests                         2.25.1
requests-unixsocket              0.2.0
scikit-learn                     0.24.2
scipy                            1.7.0
seaborn                          0.11.1
Send2Trash                       1.7.1
service-identity                 21.1.0
setuptools                       59.6.0
Shapely                          1.7.1
simpy                            4.0.1
six                              1.16.0
sklearn                          0.0
sniffio                          1.2.0
sortedcontainers                 2.4.0
sqlparse                         0.4.1
terminado                        0.10.1
testpath                         0.5.0
threadpoolctl                    2.2.0
toml                             0.10.2
tomli                            2.0.1
tornado                          6.1
traitlets                        5.0.5
transitions                      0.8.11
Twisted                          22.10.0
txaio                            23.1.1
typing-extensions                3.10.0.0
uritemplate                      4.1.1
urllib3                          1.26.6
vine                             5.0.0
wcwidth                          0.2.5
webencodings                     0.5.1
websocket-client                 1.1.0
wheel                            0.36.0
whitenoise                       6.2.0
widgetsnbextension               3.5.1
yarl                             1.9.2
zipp                             3.4.1
zope.interface                   6.0

Full stack trace from our testing pipeline is:

pytest --cov --ignore=mes/tests/ --junitxml=test-results/report.xml
Traceback (most recent call last):
  File "/usr/local/bin/pytest", line 8, in <module>
    sys.exit(console_main())
  File "/usr/local/lib/python3.9/site-packages/_pytest/config/__init__.py", line 190, in console_main
    code = main()
  File "/usr/local/lib/python3.9/site-packages/_pytest/config/__init__.py", line 148, in main
    config = _prepareconfig(args, plugins)
  File "/usr/local/lib/python3.9/site-packages/_pytest/config/__init__.py", line 329, in _prepareconfig
    config = pluginmanager.hook.pytest_cmdline_parse(
  File "/usr/local/lib/python3.9/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/usr/local/lib/python3.9/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/usr/local/lib/python3.9/site-packages/pluggy/manager.py", line 84, in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
  File "/usr/local/lib/python3.9/site-packages/pluggy/callers.py", line 203, in _multicall
    gen.send(outcome)
  File "/usr/local/lib/python3.9/site-packages/_pytest/helpconfig.py", line 103, in pytest_cmdline_parse
    config: Config = outcome.get_result()
  File "/usr/local/lib/python3.9/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/usr/local/lib/python3.9/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/usr/local/lib/python3.9/site-packages/_pytest/config/__init__.py", line 1058, in pytest_cmdline_parse
    self.parse(args)
  File "/usr/local/lib/python3.9/site-packages/_pytest/config/__init__.py", line 1346, in parse
    self._preparse(args, addopts=addopts)
  File "/usr/local/lib/python3.9/site-packages/_pytest/config/__init__.py", line 1248, in _preparse
    self.hook.pytest_load_initial_conftests(
  File "/usr/local/lib/python3.9/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/usr/local/lib/python3.9/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/usr/local/lib/python3.9/site-packages/pluggy/manager.py", line 84, in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
  File "/usr/local/lib/python3.9/site-packages/pluggy/callers.py", line 208, in _multicall
    return outcome.get_result()
  File "/usr/local/lib/python3.9/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/usr/local/lib/python3.9/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/usr/local/lib/python3.9/site-packages/pytest_django/plugin.py", line 348, in pytest_load_initial_conftests
    from django.conf import settings as dj_settings
  File "/usr/local/lib/python3.9/site-packages/django/conf/__init__.py", line 19, in <module>
    from django.utils.deprecation import RemovedInDjango40Warning
  File "/usr/local/lib/python3.9/site-packages/django/utils/deprecation.py", line 5, in <module>
    from asgiref.sync import sync_to_async
  File "/usr/local/lib/python3.9/site-packages/asgiref/sync.py", line 130, in <module>
    class AsyncToSync(Generic[_P, _R]):
  File "/usr/local/lib/python3.9/typing.py", line 243, in inner
    return func(*args, **kwds)
  File "/usr/local/lib/python3.9/typing.py", line 917, in __class_getitem__
    raise TypeError(
TypeError: Parameters to Generic[...] must all be type variables

In my most recent testing on my local, the error crops up via Django manage.py command (I apologize, the original post was inaccurate as I realize I was importing from inside a Django-initiated shell_plus session):

Traceback (most recent call last):
   File "/construction_automation/./manage.py", line 21, in <module>
     main()
   File "/construction_automation/./manage.py", line 10, in main
     from django.core.management import execute_from_command_line
   File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 13, in <module>
     from django.apps import apps
   File "/usr/local/lib/python3.9/site-packages/django/apps/__init__.py", line 1, in <module>
     from .config import AppConfig
   File "/usr/local/lib/python3.9/site-packages/django/apps/config.py", line 7, in <module>
     from django.utils.deprecation import RemovedInDjango41Warning
   File "/usr/local/lib/python3.9/site-packages/django/utils/deprecation.py", line 5, in <module>
     from asgiref.sync import sync_to_async
   File "/usr/local/lib/python3.9/site-packages/asgiref/sync.py", line 130, in <module>
     class AsyncToSync(Generic[_P, _R]):
   File "/usr/local/lib/python3.9/typing.py", line 243, in inner
     return func(*args, **kwds)
   File "/usr/local/lib/python3.9/typing.py", line 917, in __class_getitem__
     raise TypeError(
 TypeError: Parameters to Generic[...] must all be type variables

It appears that the traces both go through django.utils.deprecation.

[andrewgodwin]

Found it - you're using an old typing_extensions, this problem is fixed in 4.0 and above. I'll update the package metadata and push a patch release to ensure that constraint is applied.

[andrewgodwin]

Fixed in db2e53a66be92cb4b073408e4faf37db709699e9 and released as 3.7.1.

[pmayostendorp]

Beautiful! Many thanks @andrewgodwin ! 👏