twine broke entirely: any command causes TypeError: expected string or bytes-like object

[Fogapod]

Your Environment

OS: Arch Linux

$ python --version
Python 3.9.5

$ pip show twine
Name: twine
Version: 3.4.1
Summary: Collection of utilities for publishing packages on PyPI
Home-page: https://twine.readthedocs.io/
Author: Donald Stufft and individual contributors
Author-email: donald@stufft.io
License: UNKNOWN
Location: /home/eugene/.local/lib/python3.9/site-packages
Requires: tqdm, rfc3986, keyring, colorama, importlib-metadata, readme-renderer, requests, pkginfo, requests-toolbelt
Required-by:

twine was installed using pip and worked before. I don't remember if I updated it before it broke.

The Issue

Nothing works. Any command raises exception:

$ twine
Traceback (most recent call last):
  File "/home/eugene/.local/bin//twine", line 8, in <module>
    sys.exit(main())
  File "/home/eugene/.local/lib/python3.9/site-packages/twine/__main__.py", line 28, in main
    result = cli.dispatch(sys.argv[1:])
  File "/home/eugene/.local/lib/python3.9/site-packages/twine/cli.py", line 43, in dispatch
    registered_commands = entry_points(group="twine.registered_commands")
  File "/usr/lib/python3.9/site-packages/importlib_metadata/__init__.py", line 981, in entry_points
    return SelectableGroups.load(eps).select(**params)
  File "/usr/lib/python3.9/site-packages/importlib_metadata/__init__.py", line 434, in load
    ordered = sorted(eps, key=by_group)
  File "/usr/lib/python3.9/site-packages/importlib_metadata/__init__.py", line 978, in <genexpr>
    eps = itertools.chain.from_iterable(
  File "/usr/lib/python3.9/site-packages/importlib_metadata/_itertools.py", line 16, in unique_everseen
    k = key(element)
  File "/usr/lib/python3.9/site-packages/importlib_metadata/__init__.py", line 913, in _normalized_name
    return self._name_from_stem(stem) or super()._normalized_name
  File "/usr/lib/python3.9/site-packages/importlib_metadata/__init__.py", line 597, in _normalized_name
    return Prepared.normalize(self.name)
  File "/usr/lib/python3.9/site-packages/importlib_metadata/__init__.py", line 837, in normalize
    return re.sub(r"[-_.]+", "-", name).lower().replace('-', '_')
  File "/usr/lib/python3.9/re.py", line 210, in sub
    return _compile(pattern, flags).sub(repl, string, count)
TypeError: expected string or bytes-like object

[Fogapod]

Downgrading to twine 3.3.0 worked (random older version)

[bhrutledge]

Can you show the output of twine --version?

Here's mine (which is working on macOS):

% twine --version
twine version 3.4.1 (importlib_metadata: 3.7.3, pkginfo: 1.7.0, requests: 2.25.1, requests-toolbelt: 0.9.1, tqdm:
4.59.0)

[Fogapod]

Can you show the output of twine --version?

No, as I said, no commands work, not even twine --version. Same error each time.

[bhrutledge]

Ah, right. How about the output of python -m pip freeze?

[Fogapod]

$ python -m pip freeze

``` aiohttp==3.6.3 aiosqlite==0.17.0 alabaster==0.7.12 anytree==2.8.0 appdirs==1.4.4 argcomplete==1.12.3 asn1crypto==1.4.0 async-cleverbot==0.2.2 async-timeout==3.0.1 attrs==21.2.0 Babel==2.9.1 bcrypt==3.2.0 black==21.5b2 bleach==3.2.1 btrfsutil==5.12.1 CacheControl==0.12.6 cached-property==1.5.2 cachy==0.3.0 certifi==2020.12.5 cffi==1.14.5 cfgv==3.2.0 chardet==3.0.4 cleo==0.8.1 click==8.0.1 clikit==0.6.2 colorama==0.4.4 coverage==5.5 crashtest==0.3.1 cryptography==3.4.7 daemonize==2.5.0 discord==1.7.3 discord.py==1.7.3 distlib==0.3.1 distro==1.5.0 dnspython==1.16.0 docker==5.0.0 docker-compose==1.29.2 docker-pycreds==0.4.0 dockerpty==0.4.1 docopt==0.6.2 docutils==0.17.1 edgedb==0.15.0 evdev==1.4.0 filelock==3.0.12 flake8==3.9.2 flake8-bugbear==21.4.3 googletrans==4.0.0rc1 gplaycli==3.29 h11==0.9.0 h2==3.2.0 hpack==3.0.0 hstspreload==2020.12.22 html5lib==1.1 httpcore==0.9.1 httpx==0.13.3 hyperframe==5.2.0 identify==1.5.10 idna==2.10 imagesize==1.2.0 importlib-metadata==4.5.0 iniconfig==1.1.1 iotop==0.6 isc==2.0 isort==5.8.0 jeepney==0.6.0 Jinja2==3.0.1 jsonschema==3.2.0 keyring==21.8.0 lensfun==0.3.95 libtorrent===1.2.14-build-libtorrent-rasterbar-src-libtorrent-rasterbar-1.2.14-bindings-python lit==12.0.0.dev0 lockfile==0.12.2 louis==3.18.0 lutris==0.5.8.3 lxml==4.6.3 Markdown==3.3.4 MarkupSafe==2.0.1 matlink-gpapi==0.4.4.5 mccabe==0.6.1 meson==0.58.1 more-itertools==8.7.0 msgpack==1.0.2 multidict==4.7.6 mypy==0.910 mypy-extensions==0.4.3 nodeenv==1.5.0 notify2==0.3.1 numpy==1.20.3 openrazer==3.0.1 openrazer-daemon==3.0.1 ordered-set==4.0.2 packaging==20.9 paramiko==2.7.2 pastel==0.2.1 pathspec==0.8.1 pexpect==4.8.0 Pillow==8.3.1 pink-accents==0.0.3 pipx==0.16.3 pkginfo==1.6.1 pluggy==0.13.1 ply==3.11 podman-compose==0.1.5 poetry==1.1.4 poetry-core==1.0.0 pre-commit==2.13.0 protobuf==3.14.0 psutil==5.8.0 ptyprocess==0.7.0 py==1.10.0 pyaxmlparser==0.3.24 pybind11==2.6.2 pycairo==1.20.0 pycodestyle==2.7.0 pycparser==2.20 pyflakes==2.3.1 Pygments==2.9.0 PyGObject==3.40.1 pylev==1.3.0 PyNaCl==1.4.0 pyparsing==2.4.7 pyrsistent==0.18.0 pytest==6.2.4 pytest-cov==2.12.1 python-dotenv==0.18.0 python-xlib==0.31 pytz==2021.1 pyudev==0.22.0.dev20201112 PyYAML==5.4.1 ranger-fm==1.9.3 readme-renderer==28.0 regex==2020.11.13 requests==2.25.1 requests-toolbelt==0.9.1 rfc3986==1.4.0 SecretStorage==3.3.0 selinux==3.2 sentry-sdk==1.3.0 setproctitle==1.2.2 shellingham==1.3.2 six==1.16.0 sniffio==1.2.0 snowballstemmer==2.1.0 spark-parser==1.8.9 Sphinx==4.0.2 sphinxcontrib-applehelp==1.0.2 sphinxcontrib-devhelp==1.0.2 sphinxcontrib-htmlhelp==2.0.0 sphinxcontrib-jsmath==1.0.1 sphinxcontrib-qthelp==1.0.3 sphinxcontrib-serializinghtml==1.1.5 team==1.0 texttable==1.6.3 toml==0.10.2 tomlkit==0.7.0 tqdm==4.55.0 travitia-talk==0.0.1 twine==3.4.1 typed-ast==1.4.1 typing-extensions==3.7.4.3 ueberzug==18.1.9 urllib3==1.26.5 userpath==1.6.0 virtualenv==20.2.2 webencodings==0.5.1 websocket-client==0.59.0 xdis==5.0.7 yarl==1.5.1 youtube-dl==2021.6.6 zipp==3.4.1 ```

[bhrutledge]

This is reminding me of https://github.com/pypa/twine/issues/760, where there was some odd behavior around multiple installations of importlib_metadata. More troubleshooting commands from that issue:

type -a twine

type -a python

python -c "import sys; print(sys.executable)"

python -c "import importlib_metadata; print(importlib_metadata.version('importlib-metadata'))"

Have you tried installing Twine in a fresh virtual environment? That worked for me on macOS, using the same versions that you have installed:

% python3 -m venv venv-twine-774

% source venv-twine-774/bin/activate

% python3 -m pip install importlib_metadata==4.5.0 twine==3.4.1

% rehash

% type -a twine
twine is /private/tmp/venv-twine-774/bin/twine
twine is /Users/bhrutledge/.local/bin/twine

% twine --version
twine version 3.4.1 (importlib_metadata: 4.5.0, pkginfo: 1.7.1, requests: 2.25.1, requests-toolbelt: 0.9.1, tqdm:
4.61.2)

[Fogapod]

I will report back after Monday

[Fogapod]

debugging

$ type -a twine
twine is /home/eugene/.local/bin//twine

$ type -a python
python is /usr/bin/python

$ python -c "import sys; print(sys.executable)"
/usr/bin/python

$ python -c "import importlib_metadata; print(importlib_metadata.version('importlib-metadata'))"
4.6.0

venv

$ type -a twine
twine is /home/eugene/venv-twine-774/bin/twine
twine is /home/eugene/.local/bin//twine

$ twine --version
twine version 3.4.1 (importlib_metadata: 4.5.0, pkginfo: 1.7.1, requests: 2.26.0, requests-toolbelt: 0.9.1, tqdm: 4.61.2)

so it works inside venv, but refuses to work outside even after installing older version and reinstalling

EDIT: just noticed that my PATH was messed up, but after fixing it nothing changed:

$ type -a twine
twine is /home/eugene/.local/bin/twine

[bhrutledge]

I'm stumped. I don't see anything out of the ordinary in the debugging info. I suspect there's an obscure conflict in your global Python environment, but I don't know how to track it down. @jaraco might have a suggestion, as a maintainer of Twine and importlib_metadata.

You might try:

python -m pip freeze > global-requirements.txt

python -m venv /tmp/test-venv

source /tmp/test/venv-bin/activate

pip install -r global-requirements.txt

And see if Twine still fails. If it does, you could try a binary chop of the requirements file to isolate the conflict.

Or, you could add import pdb; pdb.set_trace() to /usr/lib/python3.9/site-packages/importlib_metadata/__init__.py, above line 837 (which is the last third-party code to execute before the failure), look at the local variables, and trace it back up the stack.

However: I think this all points to why it's generally discouraged to install packages into your global Python environment. According to the PyPA's guide to installing stand alone command line tools:

Usually you want to be able to access these from anywhere, but installing packages and their dependencies to the same global environment can cause version conflicts and break dependencies the operating system has on Python packages.

Following the recommendation in that guide, you could pipx install twine. Another common pattern is to install Twine as a development dependency in project virtual environments, along with other tools like flake8, black, etc

[jaraco]

The error message indicates that importlib_metadata is attempting to load the entry points for every package on the system, but encountering a package whose name is not text (probably None). Thinking I'd seen this message before, I searched the importlib_metadata project for the error message, but I came up empty.

I suspect you have a package with invalid metadata (missing or corrupted).

I'd do this:

>>> import importlib_metadata as md
>>> dists = md.distributions()
>>> broken = [dist for dist in dists if dist.name is None]
>>> for dist in broken:
...     print(dist._path)
...     print('metadata length', len(dist.metadata()))

I'd then inspect the metadata in each path that's printed.

[jaraco]

The issue isn't a twine issue. It just happens to affect twine because twine switched to using this newer importlib_metadata that happens to trip up on this environment. It's possible importlib_metadata should be more lenient in this case, but let's defer that consideration until we determine the cause.

[Fogapod]

>>> import importlib_metadata as md
>>> dists = md.distributions()
>>> broken = [dist for dist in dists if dist.name is None]
>>> for dist in broken:
...     print(dist._path)
...     print('metadata length', len(dist.metadata()))

This produces error:

/home/eugene/.local/lib/python3.9/site-packages/-ravitia_talk-0.0.1.dist-info
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
TypeError: 'Message' object is not callable

Removing 2nd line, I get 2 broken packages:

/home/eugene/.local/lib/python3.9/site-packages/-ravitia_talk-0.0.1.dist-info
/home/eugene/.local/lib/python3.9/site-packages/-ip-19.0.dist-info

First is mine with broken name for some reason, 2nd one is pip with broken name as well. Deleting them fixed twine. I have no idea when and why 1st letter of these packages got replaced by -.

[rachtsingh]

For future searchers, I had a similar related issue with scipy==1.9.3 that I solved with help from this thread. The following code pointed it out to me:

import importlib_metadata as md
for dist in md.distributions():
    try:
        _ = dist._normalized_name
    except:
        print(dist._path)

Still unsure why scipy was buggy, but I just downgraded to 1.9.2 and it works fine. Hope that helps someone.

[umarwar]

Downgrading twine to 3.3.0 worked for me.