Python 3.8 - "TypeError: expected string or bytes-like object" with poetry install

[azmeuk]

• [x] I am on the latest Poetry version.

• [x] I have searched the issues of this repo and believe that this is not a duplicate.

• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: gitlab python3.8 docker image

• Poetry version: 1.0.3

• Link of a Gist with the contents of your pyproject.toml file: gitlab-ci report

Issue

I have a project using poetry that use Gitlab-CI and tox to run tests in several environments. The very same code works in python 3.5, 3.6 and 3.7, but produce the following error with python 3.8. The error message is not very clear so I am not sure what is going on.

You can view all the successful executions here and the failing one here. If it matters, you can see the tox configuration here and the poetry configuration here.

py38 run-test: commands[0] | poetry install -vvv --extras all
Using virtualenv: /builds/yaal/sheraf/.tox/py38
[TypeError]
expected string or bytes-like object
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/clikit/console_application.py", line 131, in run
    status_code = command.handle(parsed_args, io)
  File "/usr/local/lib/python3.8/site-packages/clikit/api/command/command.py", line 120, in handle
    status_code = self._do_handle(args, io)
  File "/usr/local/lib/python3.8/site-packages/clikit/api/command/command.py", line 171, in _do_handle
    return getattr(handler, handler_method)(args, io, self)
  File "/usr/local/lib/python3.8/site-packages/cleo/commands/command.py", line 92, in wrap_handle
    return self.handle()
  File "/usr/local/lib/python3.8/site-packages/poetry/console/commands/install.py", line 47, in handle
    installer = Installer(
  File "/usr/local/lib/python3.8/site-packages/poetry/installation/installer.py", line 55, in __init__
    installed = self._get_installed()
  File "/usr/local/lib/python3.8/site-packages/poetry/installation/installer.py", line 488, in _get_installed
    return InstalledRepository.load(self._env)
  File "/usr/local/lib/python3.8/site-packages/poetry/repositories/installed_repository.py", line 26, in load
    package = Package(name, version, version)
  File "/usr/local/lib/python3.8/site-packages/poetry/packages/package.py", line 42, in __init__
    self._name = canonicalize_name(name)
  File "/usr/local/lib/python3.8/site-packages/poetry/utils/helpers.py", line 26, in canonicalize_name
    return _canonicalize_regex.sub("-", name).lower()
ERROR: InvocationError for command /usr/local/bin/poetry install -vvv --extras all (exited with code 1)

Thank you for your help

[toymachine]

Hi, I experience the same issue, and eventually traced it to this:

class InstalledRepository(Repository):
    @classmethod
    def load(cls, env):  # type: (Env) -> InstalledRepository
        """
        Load installed packages.

        For now, it uses the pip "freeze" command.
        """
        repo = cls()
        seen = set()

        print(env.sys_path)
        for entry in env.sys_path:
            if not entry:
                continue
            print('entry', repr(entry))

e.g. at this point env.sys_path contains an empty string as the first item, which somehow eventually leads to 'name' passed to canonicalize_name being None skipping the empty entry as i did here seems to fix it for now

[jonadaly]

Seeing the exact same issue, but on Python 3.7.2 and poetry 1.0.3. Downgrading to poetry 1.0.2 made no difference.

[finswimmer]

I think this is caused by a bug in virtualenv like mentioned in #1953. This is fixed in the latest release of virtualenv.

[cordery]

This will also happen if you somehow end up with an .egg-info file (presumably malformatted) in your project directory. I think I created one unintentionally while experimenting with dephell, but regardless of the source this is what you can expect to see:

myproject % touch myproject.egg-info
myproject % poetry check
All set!
myproject % poetry show

[TypeError]
expected string or bytes-like object
myproject % rm myproject.egg-info
myproject % poetry show
aiohttp                        3.6.2     Async http client/server framework (asyncio)
....

[Evgenus]

@cordery thank you, man. that was sick.

[Vozf]

For me it was also due to activated conda environment. So it was fixed by conda deactivate

[trim21]

I met this bug again in poetry==1.1.0 on windows.

[abn]

@Trim21 can you raise a new issue with some additional details please? Maybe a pyproject.toml, -vvv logs and how python was installed? I am particularly curious to see if the use of PYTHONUSERBASE has an impact on this as well.

[trim21]

@abn OK, I'll create a repo with github actions to reproduce it.

[adam-grant-hendry]

Same issue

[adam-grant-hendry]

@Trim21 Is there a link to your repo?

[adam-grant-hendry]

@cordery Also confirm this was due to a bad egg for me. I deleted the .egg-info folder, recreated my venv, and reran poetry install with my existing toml and it worked.

[abn]

@adam-grant-hendry; yes I think this is because we moved to a in-tree editable installer (1.1) rather than relying on setuptools (1.0). The old egg-info takes precedence when python path is built i think.

[trim21]

Not yet, being busy these days.

------------------ Original ------------------ From: "adam-grant-hendry"<notifications@github.com>; Date: 2020年10月7日(星期三) 凌晨4:06 To: "python-poetry/poetry"<poetry@noreply.github.com>; Cc: "Trim21"<github@trim21.me>; "Mention"<mention@noreply.github.com>; Subject: Re: [python-poetry/poetry] Python 3.8 - "TypeError: expected string or bytes-like object" with poetry install (#2027)

@Trim21 Is there a link to your repo?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[AmmarlOsama]

I met this bug again in poetry==1.1.0 on windows.

did u solve it ?

[trim21]

@AmmarlOsama It has been fixed in latest poetry.

[repnop]

Hitting this on a fresh project, with 1.1.4 latest release installed via pip

edit: I commented out a git dep and now it let me poetry install, not sure if that's related or useful info

[sinoroc]

@repnop It would be more helpful for you if you communicated meaningful information, such as the pyproject.toml, the name of the git dependency that causes the issue, the full console output showing the error message.

[repnop]

@sinoroc its an internal work project so the git dependency is not public, but its a simple

dep = { git = "git://a.url/dep" }

the full error is exactly the same as above, and I did hit it when trying to install aiomysql or sanic before that (but it hasn't happened with them since I fixed? whatever the problem was there). I tried deleting the venv multiple times and reinstalling the dependencies.

[bersace]

Same here. I can't get to workaround this with a git dependency.

[shaunc]

Also the same (git dependency) -- made sure I have most recent virtualenv, and deleted the old virtualenv, but still didn't work with reported error.

Pyproject.toml as follows:

[tool]
[tool.poetry]
authors = ["Shaun Cutts <shauncutts@factfiber.com>"]
description = "swapdp port for pattern discovery"
name = "emm"
version = "0.1.0"

packages = [
  {include = "emm", from = "src"},
]

# project configuration for emm
[tool.poetry.scripts]
# <script> = '<dottedname>:<function>'

[tool.poetry.dependencies]
antegraph = {
  git = "git+ssh://git@gitlab.factfiber.com/factfiber/antegraph.git"}
dacite = "^1.5.1"
h5netcdf = "^0.8.1"
hdbscan = "^0.8.26"
ipykernel = "^5.3.4"
loguru = "^0.5.3"
numba = "^0.51.2"
numexpr = "^2.7.1"
numpy = "^1.19.2"
pandas = "^1.1.3"
pyarrow = "^1.0.1"
python = "^3.8"
python-dotenv = "^0.14.0"
xarray = "^0.16.1"
yamale = "^3.0.4"

[tool.poetry.dev-dependencies]
black = "^20.8b1"
dephell = "^0.8.3"
flake8 = "*"
matplotlib = "^3.3.2"
mypy = "*"
pdoc3 = "^0.9.1"
pre-commit = "^2.7.1"
pylint = "*"
pyls-mypy = "^0.1.8"
pytest = "^6.1.1"
pytest-cov = "^2.10.1"
pytest-snapshot = "^0.4.2"
python-language-server = "*"
rope = "==0.17.*,>=0.17.0"

[tool.pytest.ini_options]
required_plugins = "pytest-snapshot"

[tool.black]
exclude = '''
/(
    \.git
  | \.hg
  | \.mypy_cache
  | \.tox
  | \.devspace
  | \.venv
  | \.vscode
  | _build
  | buck-out
  | build
  | dist
)/
'''
include = '\.pyi?$'
line-length = 72

[tool.dephell.main]
from = {format = "poetry", path = "pyproject.toml"}
to = {format = "setuppy", path = "setup.py"}

[build-system]
build-backend = "poetry.masonry.api"
requires = ["poetry>=0.12"]

[zvolsky]

My idea is that this error will occur allways if the name of installed package is missing for poetry.

I want install some package shared between my projects on the development machine (something like -e mode of pip). I create empty mypackage/ and inside mypackage/mypackage/ with code.

Now: Both poetry add mypackage/ and poetry add mypackage/mypackage/ will fail with the error above.

Then I create mypackage/setup.py with setup(name='mypackage', .....). And now:poetry add mypackage/` will install the editable package without problems.

Minimum setup.py content for this goal is:

from setuptools import setup
setup(name='mypackage')

Maybe this will help to identify the reasons more exactly?

[djackson-exo-inc]

What is the solution to this? This ticket is closed, but I am still getting the issue. I have read every comment here and did not see what allowed this to be closed. If this is fixed in some version of poetry, can someone please be explicit about which specific version of poetry this is fixed in? Thank you.

[gregorgabrovsek]

On my machine, this was fixed when I installed Poetry version 1.1.5.

[RashiqAzhan]

Don't know why this issue is closed but still exists in version 1.1.5.

[YoshihikoFuruhashi]

In my case, I moved pyproject.toml to the root directory of the git repository and that fixed the problem. But the error occurred if pyproject.toml was placed in the sub directory.

OK

git repository root/ 　├ pyproject.toml 　├ src/

NG

git repository root/ 　├ src/ 　 　└ pyproject.toml

(repository = GitHub repository of poetry package)

[PaulBenn-UnlikelyAI]

For people looking for an open issue related to this, the discussion seems to continue here: https://github.com/python-poetry/poetry/issues/3628

[jaepil-choi]

This issue still exists in the latest 1.1.7

[Chiang97912]

I solved this problem by deleting the virtual environment. First you can get ENV_NAME by typing poetry env list, and then typing poetry env remove ENV_NAME to delete this virtual environment.

[mlissner]

Nuking and recreating the env did it for me too. I did hit CTRL+C in the middle of an install before this problem began. I guess I'm naive to think that'd be safe.

[RobertTownley]

I was encountering this issue within the build phase of a dockerized Django web app. I made a few changes to my Dockerfile which fixed this issue for me. Unfortunately I'm not fully clear on which change fixed it. I suspect it's that in poetry v1.x.x the lack of virtualenvs is configured via POETRY_VIRTUALENVS_CREATE=false rather thank configuring it with RUN poetry config virtualenvs.create. Nonetheless, I run both in this dockerfile (probably redundantly) and the issue has fixed itself.

FROM python:3
ENV PYTHONUNBUFFERED 1
EXPOSE 8000

ENV PYTHONFAULTHANDLER=1 \
  PYTHONUNBUFFERED=1 \
  PYTHONHASHSEED=random \
  PIP_NO_CACHE_DIR=off \
  PIP_DISABLE_PIP_VERSION_CHECK=on \
  PIP_DEFAULT_TIMEOUT=100 \
  POETRY_VIRTUALENVS_CREATE=false \
  POETRY_VERSION=1.1.11

RUN pip install --upgrade pip
RUN pip install poetry
COPY pyproject.toml .
COPY poetry.lock .

RUN poetry config virtualenvs.create false \
  && poetry install $(test "$YOUR_ENV" == production && echo "--no-dev") --no-interaction --no-ansi

CMD /entrypoint.sh

[douwevandermeij]

For everyone still having issues with this, also with Python 3.9, this could also work (worked for me):

poetry export --without-hashes > requirements.txt
pip install -r requirements.txt

Or in Docker:

...

RUN pip install --upgrade pip
RUN pip install poetry
COPY pyproject.toml .
COPY poetry.lock .

RUN poetry export --without-hashes > requirements.txt \
    && pip install -r requirements.txt

[bw2]

Deleting the single *.egg from my .../python3.*/site-packages/ directory fixed this error.

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.