Closed mikepqr closed 2 years ago
Note this is also affecting GitHub actions on macOS and Windows (but not Ubuntu): https://github.com/mikepqr/resume.md/actions/runs/2678077124.
What version of importlib-metadata
are you using, we can see here that Python Markdown requires: importlib-metadata>=4.4
for Python < 3.10.
These are installations into fresh virtual environments with pip install python-markdown
. I therefore get whatever packages markdown declares as dependencies.
As you can see above, that results in:
Successfully installed importlib-metadata-4.12.0 markdown-3.4 zipp-3.8.1
in python 3.9, and just
Successfully installed markdown-3.4
in python 3.10.
Note also that I get the same error on python3.10, which has no importlib-metadata dependency.
I wonder if something changed in 4.12.0...I'm on 4.11.3 on Python 3.10 and I wasn't experiencing any issues. Tests were run on Py3.10 as well with no issues: https://github.com/Python-Markdown/markdown/runs/7359376261?check_suite_focus=true. If you downgrade importlib-metadata to 4.11.X do you still get the issue?
Away from test environment now, will check later. But downgrading importlib-metadata on 3.9 won't help on python 3.10. It seems like the current release of python-markdown is broken out of the box on 3.10 on macOS and Windows. Looks like the CI suite only covers Linux?
Hmm, I'm running macOS and Python 3.10 importlib-metadata 4.12.0 and Python Markdown 3.4 with no issues. I cannot seem to reproduce...
Is this running with a brew python or an official python distribution?
Locally it happens to be running in pyenv-installed fresh builds of official python 3.9.6 and 3.10.3, and fresh virtual environments. brew is not involved.
But please note I am seeing the exact same error in Github actions on macOS and Windows.
I understand, but there is something else going on here. I should be able to reproduce this issue on macOS...I'll have to spend some time this evening and see isolate things in a virtual environment. I should note that I'm on Python 3.10.4. Not sure if that makes a difference either.
Thanks! I will also try to narrow this down more later.
The code which is raising this error has not changed in two years and was part of the 3.3.0 release. There were no changes in any release from 3.3.1 through 3.3.7. If it works in any of those but not in 3.4.0, then the issue is something other than our code.
When you do pip install "markdown<3.4"
, which version do you get?
3.3.7.
Make an empty repo with workflow to attempt to bisect.
Here's what happens with pip install markdown
(i.e. import markdown
@ 3.4.0 fails on macOS and Windows with AttributeError: module 'importlib' has no attribute 'util'
).
Here's the same action with python-markdown pinned to 3.3.7 (works on all tested platforms).
I also have plenty of Windows actions run with Python 3.9 and 3.10 with the version right before Markdown 3.4: https://github.com/facelessuser/pymdown-extensions/actions/runs/2652028392. So I am hesitant to think there is an issue.
we can see here that Python Markdown requires:
importlib-metadata>=4.4
for Python < 3.10.
Actually, we don't require importlib-metadata
at all for Python 3.10, only for any previous Python version. It was my understanding that by default, Python 3.10 was supposed to match importlib-metadata
version 4.4 or later. Perhaps that is not true in some cases. Although, it seems strange that an import path for a standard lib would only exist on some platforms.
Sorry, I may have stated it backward.
@mikepqr if you install importlib-metadata>=4.4
in python 3.10 does that resolve the issue for you?
I can confirm on macOS that util
does exist. I just don't have 3.9 on this machine anymore to test that.
Python 3.10.4 (v3.10.4:9d38120e33, Mar 23 2022, 17:29:05) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import importlib
>>> importlib.util
<module 'importlib.util' from '/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/importlib/util.py'>
>>>
I also have plenty of Windows actions run with Python 3.9 and 3.10 with the version right before Markdown 3.4
Do these work with markdown 3.4 though?
@mikepqr if you install importlib-metadata>=4.4 in python 3.10 does that resolve the issue for you?
I can check later, but I feel like 3.9 vs. 3.10 is probably a red herring, given the error is the same on both versions.
Do these work with markdown 3.4 though?
We'll know shortly: https://github.com/facelessuser/pymdown-extensions/actions/runs/2678476922
is this relevant, i.e. are you trying to access importlib.util
as an attribute of importlib
, rather than by importing importlib.util
?
It's a submodule, nothing is wrong with that:
Python 3.10.4 (v3.10.4:9d38120e33, Mar 23 2022, 17:29:05) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import importlib
>>> importlib.util.find_spec
<function find_spec at 0x108703d00>
Actions passed on py3.9 and py3.10 on Windows. I don't test in CI on mac, but as I stated, I cannot reproduce locally on my mac.
π€·π»
Configured the action to just access importlib.util.find_spec. Same error.
And locally in fresh environments:
$ python
Python 3.9.6 (default, Jul 5 2021, 13:23:11)
[Clang 12.0.5 (clang-1205.0.22.9)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import importlib
>>> importlib.util.find_spec
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: module 'importlib' has no attribute 'util'
$ python
Python 3.10.3 (main, Mar 25 2022, 22:16:41) [Clang 12.0.5 (clang-1205.0.22.9)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import importlib
>>> importlib.util.find_spec
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: module 'importlib' has no attribute 'util'
There is something very unique about your environment and how you are doing things. I have many, many projects that rely on Markdown for their documentation. I've demonstrated that PY3.10 does in fact access util
just fine along with current actions indicating that to be the case as well.
I do not deny you are having an issue, but I cannot reproduce it in any way. You appear to be an outlier. Maybe try using actions/setup-python@v2
for Python actions π€·π».
It's a submodule, nothing is wrong with that
I don't think that's generally true. It depends on how importlib/__init__.py
is set up. You get importlib.util
in your namespace iff that __init__.py
explicitly gives it to you:
$ tree
.
βββ package
βββ __init__.py
βββ util.py
2 directories, 3 files
$ cat package/__init__.py
$ python3
Python 3.9.12 (main, Mar 26 2022, 15:51:15)
[Clang 13.1.6 (clang-1316.0.21.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import package
>>> package.util
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: module 'package' has no attribute 'util'
>>> import package.util
>>> package.util
<module 'package.util' from '/Users/mleewilliams/tmp/package/package/util.py'>
vs.
$ cat package/__init__.py
import package.util
$ python3
Python 3.9.12 (main, Mar 26 2022, 15:51:15)
[Clang 13.1.6 (clang-1316.0.21.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import package
>>> package.util
<module 'package.util' from '/Users/mleewilliams/tmp/package/package/util.py'>
>>>
I'm wondering if a change in importlib/__init__.py
explains what we're seeing?
There is something very unique about your environment and how you are doing things.
eh, I'm validating this behavior by doing pip install markdown; python -c "import markdown"
with the default version of python provided in the Windows and macOS Github CI runners, and I found out about this problem because a user reported it to me.
Maybe try using actions/setup-python@v2 for Python actions
I will try later.
Following the advice on https://bugs.python.org/issue41958 that describes import importlib; importlib.util.find_spec
as user error:
$ git clone https://github.com/Python-Markdown/markdown/
Cloning into 'markdown'...
remote: Enumerating objects: 10709, done.
remote: Counting objects: 100% (507/507), done.
remote: Compressing objects: 100% (237/237), done.
remote: Total 10709 (delta 357), reused 360 (delta 265), pack-reused 10202
Receiving objects: 100% (10709/10709), 3.41 MiB | 15.57 MiB/s, done.
Resolving deltas: 100% (7001/7001), done.
$ cd markdown/
$ python3 -m venv venv
$ source venv/bin/activate
$ python
Python 3.9.12 (main, Mar 26 2022, 15:51:15)
[Clang 13.1.6 (clang-1316.0.21.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import markdown
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/mleewilliams/tmp/markdown/markdown/__init__.py", line 22, in <module>
from .core import Markdown, markdown, markdownFromFile
File "/Users/mleewilliams/tmp/markdown/markdown/core.py", line 27, in <module>
from .preprocessors import build_preprocessors
File "/Users/mleewilliams/tmp/markdown/markdown/preprocessors.py", line 29, in <module>
from .htmlparser import HTMLExtractor
File "/Users/mleewilliams/tmp/markdown/markdown/htmlparser.py", line 29, in <module>
spec = importlib.util.find_spec('html.parser')
AttributeError: module 'importlib' has no attribute 'util'
>>>
$ vim markdown/htmlparser.py
$ git diff
diff --git a/markdown/htmlparser.py b/markdown/htmlparser.py
index 7ca858e..3512d1a 100644
--- a/markdown/htmlparser.py
+++ b/markdown/htmlparser.py
@@ -20,7 +20,7 @@ License: BSD (see LICENSE.md for details).
"""
import re
-import importlib
+import importlib.util
import sys
$ python
Python 3.9.12 (main, Mar 26 2022, 15:51:15)
[Clang 13.1.6 (clang-1316.0.21.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import markdown
>>>
Okay, I can reproduce the error sometimes, but not every time.
>>> import importlib
>>> importlib.util
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: module 'importlib' has no attribute 'util'
>>> from importlib import util
>>> importlib.util
<module 'importlib.util' from 'C:\\Users\\wlimberg\\AppData\\Local\\Programs\\Python\\Python38\\lib\\importlib\\util.py'>
A few things to note:
util
is imported directly, then it can be referenced from importlib.util
, but not before.However, I can do this with no errors:
from importlib.util import find_spec, module_from_spec
@mikepqr I'm wondering if this change will work for you:
diff --git a/markdown/htmlparser.py b/markdown/htmlparser.py
index 7ca858e..cfdc49c 100644
--- a/markdown/htmlparser.py
+++ b/markdown/htmlparser.py
@@ -20,14 +20,14 @@ License: BSD (see LICENSE.md for details).
"""
import re
-import importlib
import sys
+from importlib.util import find_spec, module_from_spec
# Import a copy of the html.parser lib as `htmlparser` so we can monkeypatch it.
# Users can still do `from html import parser` and get the default behavior.
-spec = importlib.util.find_spec('html.parser')
-htmlparser = importlib.util.module_from_spec(spec)
+spec = find_spec('html.parser')
+htmlparser = module_from_spec(spec)
spec.loader.exec_module(htmlparser)
sys.modules['htmlparser'] = htmlparser
It seems to work for me.
I can't understand why this makes a difference. And weirdly, now I suddenly can't get Markdown to work without the change when I import it from within a Python session. However, when I call Markdown from the command line (python -m markdown
), I get no error.
Okay, I can reproduce the error sometimes, but not every time.
I think the difference you're seeing is because import importlib
imports importlib/__init__.py
(which does not (always?) add util
to its namespace) and from importlib import util
imports importlib/util.py
. See this comment for more.
The error only occurs if I do the imports in a certain order. Once util is imported directly, then it can be referenced from importlib.util, but not before.
The order is not the issue. It's whether or not you do import importlib.util
(or, equivalently, from importlib import util
).
importlib/__init__.py
is very, very complicated, and I can't figure out what it's doing at all. It's certainly doing some weird stuff. But importing importlib.util
directly circumvents this complexity entirely so yes, that diff would fix it for me.
So would the diff I posted, which matches what Markdown already does in its setup.py.
I suggest you add that change, assuming it doesn't break already working environments?! (Although I totally understand if you want to get to the root cause of this before landing.)
diff --git a/markdown/htmlparser.py b/markdown/htmlparser.py
index 7ca858e..3512d1a 100644
--- a/markdown/htmlparser.py
+++ b/markdown/htmlparser.py
@@ -20,7 +20,7 @@ License: BSD (see LICENSE.md for details).
"""
import re
-import importlib
+import importlib.util
import sys
That also resolves it for me. There is some weird funny-business going on here. Why do I need the fix from the Python prompt, but not when calling a script from the command line? And why did this suddenly start happening today, but I've never encountered it before. And why does downgrading Markdown make a difference when there is no difference to the offending code? So weird.
So would the diff I posted, which matches what Markdown already does in its setup.py.
Well, I guess that should be the route we take to fix it then. I still have no idea why the error didn't appear before now or why it suddenly started appearing now.
Me neither!
Well, this has been a weird and interesting bug. Nice to see a resolution.
Well, I can't promise the fix doesn't break currently working environments, given that it seems like none of us understand the problem! But the fix unblocks me, so I'm all for it ;-)
Given that Python documentation has an example of importing importlib.util
, I don't think it can break any currently working environment. Of course, unless someone relied on us raising AttributeError
:)
My builds are green again. Thanks folks!
With python3.9 on macOS:
With python3.10 on macOS:
pip install "markdown<3.4"
works, so this is perhaps a regression in the 3.4 release?