Closed coredumperror closed 1 year ago
Can confirm this was a very confusing bug that hit us today, and we tracked it down to an update of importlib-metadata
from 4.12.0
to 5.0.0
. From a quick check it seems to be this line, https://github.com/python/importlib_metadata/compare/v4.13.0...v5.0.0#diff-bf79a43449f7a7e1e76063e303fbdd35bec7eb50f2e9ddba26e3048def32ed06R856, but it seems odd this would impact importing celery.
Same issue.
Yep same here! Breaks Python 3.7 compatibility. Took a long time to figure out....
This broke every single airflow cluster that did a pull :-(
See pinned issue #409. @jaraco can you confirm if this really is Python 3.7 specific problem or the problem is in the packages we use that need updating?
See https://github.com/celery/celery/issues/7783 where celery has been tracking the issue.
confirm if this really is Python 3.7 specific problem or the problem is in the packages we use that need updating?
The issue isn't specific to Python 3.7. It will affect any consumers of importlib_metadata
5 or later that use entry_points()
and rely on the dict-like interface it previously returned. In the case of Celery, it did only affect Python 3.7 and earlier because it relied on stdlib's importlib.metadata
on Python 3.8 and later.
There are many projects that only rely on importlib_metadata
for Python 3.7 and earlier and rely on importlib.metadata
in the stdlib for 3.8 and later. These projects are also at risk, because in Python 3.12, importlib.metadata
gets the same behavior. Only on importlib_metadata 3.6 and Python 3.10 or later is the SelectableGroups
compatibility interface supplied, so anyone using importlib.metadata
on Python 3.8 or 3.9 must use the deprecated interface.
That's why I recommend to either:
Other options that I don't recommend include:
entry_points
function entirely and use more raw interfaces on Distribution
objects.A temporary workaround is to pin to importlib_metadata < 5
(or force-install that version).
I recognize that as consumers of Celery, this issue feels out of your control, which is why I'm glad to see that Celery has addressed the issue (though best I can tell, the fix isn't released yet).
Reliance on this deprecated behavior has been marked as Deprecated for over a year. I'm curious to see - is there something unique to celery (or other projects) that made it difficult for these downstream projects to detect and respond to the warnings?
I can confirm that in at least one run, celery did emit the warnings:
=============================== warnings summary ===============================
.tox/3.7-integration-redis/lib/python3.7/site-packages/kombu/utils/compat.py:85
/home/runner/work/celery/celery/.tox/3.7-integration-redis/lib/python3.7/site-packages/kombu/utils/compat.py:85: DeprecationWarning: SelectableGroups dict interface is deprecated. Use select.
entry_points = importlib_metadata.entry_points().get(namespace, [])
t/integration/test_canvas.py: 150 warnings
t/integration/test_inspect.py: 5 warnings
t/integration/test_security.py: 1 warning
t/integration/test_tasks.py: 40 warnings
/home/runner/work/celery/celery/celery/utils/imports.py:144: DeprecationWarning: SelectableGroups dict interface is deprecated. Use select.
for ep in entry_points().get(namespace, []):
Things that celery and other projects can do to detect these incoming incompatible changes:
I regret the disruption caused here. If there's something more that importlib_metadata can do, please let me know.
@jaraco Thanks for the details.
Just wanted to let you know that in our runs, we haven't seen anything.
For example a run where I have set importlib_metadata<5.0.0
.
The problem for us was in tavern
which relies on stevedore
.
@jaraco Thanks for the details. Just wanted to let you know that in our runs, we haven't seen anything. For example a run where I have set
importlib_metadata<5.0.0
. The problem for us was intavern
which relies onstevedore
.
I looked briefly and I concur - it appears as if no warning was issued for sabnzbd/sabnzbd
and I don't see anything that would have disabled the warnings (e.g. -W ignore
). I think it would be a useful exercise to investigate why the tests failed to report the deprecation warning, as it may help me understand a condition where such a warning doesn't reach the intended audience, or it may help projects like yours avoid this situation for other deprecations.
Is it possible that stevedore has updated the underlying usage and that's why the warnings aren't emitted any longer? Are you able to run with -W error
to force any deprecation warnings to raise as Exceptions to find a traceback showing the codepath that's reached?
I suspect the way tavern.run invokes pytest.main directly may be implicated in masking any warnings that occur within that run.
Hi @jaraco
I failed to detect this issue with the Chaos Toolkit but an user reported it recently.
import click
import importlib_metadata
from click_plugins import with_plugins
@click.group
def cli():
pass
with_plugins(importlib_metadata.entry_points().get("chaostoolkit.cli_plugins"))(cli)
This throws an error with importlib_metadata >= 5
So I switched to:
import click
import importlib_metadata
from click_plugins import with_plugins
@click.group
def cli():
pass
with_plugins(importlib_metadata.entry_points(group="chaostoolkit.cli_plugins"))(cli)
This now works.
But I'm confused by your statement above. What's the right path forward? Should I use this fix and force importlib_metadata >= 6 ? Or should I stick to older versions as other projects will likely not accept this newer version when installing?
Basically, I'm not clear what's the official position.
Cheers,
You don't need to force importlib_metadata 6. >=3.6
is the minimum version for selectable groups (the latter usage). If you can't rely on that version (released Feb 21), there is backports.entry_points_selectable that provides the preferred interface for older importlib_metadata and Python releases.
I can't run the import statement
from celery import Celery
in Python 3.7 with importlib-metadata 5.0.0 installed. I get:This makes no sense. That file totally has a
Celery
object in it... kindof. It's brought in by an unusual construction at the bottom of https://github.com/celery/celery/blob/master/celery/__init__.py, which may be what's confusing importlib-metadata 5.Minimally necessary install to trigger this is to make a virtualenv from Python 3.7.10, activate it, and run
pip install celery
. This will automatically install importlib-metadata 5.0.0.Then just start
python
and runfrom celery import Celery
. It'll crash with the above traceback. Downgrade toimportlib-metadata==4.13.0
and it won't crash.This also affects 3.7.13, so it probably affects all other point releases of 3.7. We use 3.7.10 because that's what the AmazonLinux docker image provides.
Oddly enough, installing celery on Python 3.8 doesn't result in importlib-metadata being installed automatically. The import works fine without it installed. And manually installing 5.0 doesn't break the import like it does in 3.7. So this seems to be a Python 3.7-specific issue.