poetry publish fails when tool.poetry.source is specified throwing "a repository with name was already added"

[Jakobhenningjensen]

Description

When releasing some code we have a system which step 1 is to run a unittest. If that passes, the code is published to our internal packages such that we can use them in production. The code for the unittest/publish is

set -e
pip install poetry==1.8.3 keyring==24.0.0 keyrings.google-artifactregistry-auth==1.1.1

poetry install
poetry run pytest ./tests/unit_tests

poetry config repositories.my_repo https://my-package/packages"

poetry publish --build -r my_repo

The issue is that the publish part fails when I want to include pytorch with CUDA, thus I have to specify where to get the CUDA-torch from using the tool.poetry.source block. The publish part throws the warning/error

A repository with name torch-gpu-linux was already added.

and exists with status 1, making the entire pipeline failing since the package cannot be published.

It seems like the repo specified in tool.poetry.source is being added twice, thus failing on the second one.

Workarounds

Not in the .toml-file.

The way we work around it later on, is to skip the torch-part completely and then install it manually later on when the package is downloaded (which easily can lead to bugs).

Poetry Installation Method

pip

Operating System

It is run in a docker-container

Poetry Version

1.8.3

Poetry Configuration

cache-dir = "/root/.cache/pypoetry"
experimental.system-git-client = false
installer.max-workers = null
installer.modern-installation = true
installer.no-binary = null
installer.parallel = true
repositories.dinero.url = "https://europe-west4-python.pkg.dev/test-dineroci-cm/ai-packages/"
repositories.torch-gpu-linux.url = "https://download.pytorch.org/whl/cu118"
virtualenvs.create = true
virtualenvs.in-project = null
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.no-setuptools = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}/virtualenvs"  # /root/.cache/pypoetry/virtualenvs
virtualenvs.prefer-active-python = false
virtualenvs.prompt = "{project_name}-py{python_version}"

Python Sysconfig

No response

Example pyproject.toml

The (minimalistic) toml-file is

[tool.poetry]
name = "Foo"
version = "0"
description = ""
authors = ["Me"]
readme = "README.md"

[tool.poetry.dependencies]
python = "~3.11"
sentence-transformers = "^2.7.0"
torch = [
    { version = "^2.3.0", markers = "sys_platform == 'darwin'" },   # macOS, no CUDA
    { version = "^2.3.0+cu118", markers = "sys_platform == 'linux'", source="torch-gpu-linux" }  # Linux with CUDA
]

[tool.poetry.dev-dependencies]
pytest = "^7.2.0"

[[tool.poetry.source]]
name = "my_repo"
url = "https://my-package/packages"
priority = "primary"

[[tool.poetry.source]]
name = "PyPI"
priority = "supplemental"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

[[tool.poetry.source]]
name = "torch-gpu-linux"
url = "https://download.pytorch.org/whl/cu118"
priority = "explicit"

Poetry Runtime Logs

poetry publish --build -r my_repo
Loading configuration file /root/.config/pypoetry/config.toml
[11:37:55][Step 1/1] Adding repository repo (https://my-package/packages) and setting it as primary
[11:37:55][Step 1/1] Adding repository PyPI (https://pypi.org/simple/) and setting it as supplemental
[11:37:55][Step 1/1] Adding repository torch-gpu-linux (https://download.pytorch.org/whl/cu118) and setting it as explicit
[11:37:55][Step 1/1] Adding repository torch-gpu-linux (https://download.pytorch.org/whl/cu118) and setting it as explicit
[11:37:56][Step 1/1] 
[11:37:56][Step 1/1]   Stack trace:
[11:37:56][Step 1/1] 
[11:37:56][Step 1/1]   12  /usr/local/lib/python3.11/site-packages/cleo/application.py:327 in run
[11:37:56][Step 1/1]        325│ 
[11:37:56][Step 1/1]        326│             try:
[11:37:56][Step 1/1]      → 327│                 exit_code = self._run(io)
[11:37:56][Step 1/1]        328│             except BrokenPipeError:
[11:37:56][Step 1/1]        329│                 # If we are piped to another process, it may close early and send a
[11:37:56][Step 1/1] 
[11:37:56][Step 1/1]   11  /usr/local/lib/python3.11/site-packages/poetry/console/application.py:190 in _run
[11:37:56][Step 1/1]        188│         self._load_plugins(io)
[11:37:56][Step 1/1]        189│ 
[11:37:56][Step 1/1]      → 190│         exit_code: int = super()._run(io)
[11:37:56][Step 1/1]        191│         return exit_code
[11:37:56][Step 1/1]        192│ 
[11:37:56][Step 1/1] 
[11:37:56][Step 1/1]   10  /usr/local/lib/python3.11/site-packages/cleo/application.py:431 in _run
[11:37:56][Step 1/1]        429│             io.input.interactive(interactive)
[11:37:56][Step 1/1]        430│ 
[11:37:56][Step 1/1]      → 431│         exit_code = self._run_command(command, io)
[11:37:56][Step 1/1]        432│         self._running_command = None
[11:37:56][Step 1/1]        433│ 
[11:37:56][Step 1/1] 
[11:37:56][Step 1/1]    9  /usr/local/lib/python3.11/site-packages/cleo/application.py:473 in _run_command
[11:37:56][Step 1/1]        471│ 
[11:37:56][Step 1/1]        472│         if error is not None:
[11:37:56][Step 1/1]      → 473│             raise error
[11:37:56][Step 1/1]        474│ 
[11:37:56][Step 1/1]        475│         return terminate_event.exit_code
[11:37:56][Step 1/1] 
[11:37:56][Step 1/1]    8  /usr/local/lib/python3.11/site-packages/cleo/application.py:457 in _run_command
[11:37:56][Step 1/1]        455│ 
[11:37:56][Step 1/1]        456│             if command_event.command_should_run():
[11:37:56][Step 1/1]      → 457│                 exit_code = command.run(io)
[11:37:56][Step 1/1]        458│             else:
[11:37:56][Step 1/1]        459│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED
[11:37:56][Step 1/1] 
[11:37:56][Step 1/1]    7  /usr/local/lib/python3.11/site-packages/cleo/commands/base_command.py:117 in run
[11:37:56][Step 1/1]        115│         io.input.validate()
[11:37:56][Step 1/1]        116│ 
[11:37:56][Step 1/1]      → 117│         return self.execute(io) or 0
[11:37:56][Step 1/1]        118│ 
[11:37:56][Step 1/1]        119│     def merge_application_definition(self, merge_args: bool = True) -> None:
[11:37:56][Step 1/1] 
[11:37:56][Step 1/1]    6  /usr/local/lib/python3.11/site-packages/cleo/commands/command.py:61 in execute
[11:37:56][Step 1/1]         59│ 
[11:37:56][Step 1/1]         60│         try:
[11:37:56][Step 1/1]      →  61│             return self.handle()
[11:37:56][Step 1/1]         62│         except KeyboardInterrupt:
[11:37:56][Step 1/1]         63│             return 1
[11:37:56][Step 1/1] 
[11:37:56][Step 1/1]    5  /usr/local/lib/python3.11/site-packages/poetry/console/commands/publish.py:52 in handle
[11:37:56][Step 1/1]         50│         from poetry.publishing.publisher import Publisher
[11:37:56][Step 1/1]         51│ 
[11:37:56][Step 1/1]      →  52│         publisher = Publisher(self.poetry, self.io)
[11:37:56][Step 1/1]         53│ 
[11:37:56][Step 1/1]         54│         # Building package first, if told
[11:37:56][Step 1/1] 
[11:37:56][Step 1/1]    4  /usr/local/lib/python3.11/site-packages/poetry/console/commands/command.py:23 in poetry
[11:37:56][Step 1/1]         21│     def poetry(self) -> Poetry:
[11:37:56][Step 1/1]         22│         if self._poetry is None:
[11:37:56][Step 1/1]      →  23│             return self.get_application().poetry
[11:37:56][Step 1/1]         24│ 
[11:37:56][Step 1/1]         25│         return self._poetry
[11:37:56][Step 1/1] 
[11:37:56][Step 1/1]    3  /usr/local/lib/python3.11/site-packages/poetry/console/application.py:129 in poetry
[11:37:56][Step 1/1]        127│             project_path = self._io.input.option("directory")
[11:37:56][Step 1/1]        128│ 
[11:37:56][Step 1/1]      → 129│         self._poetry = Factory().create_poetry(
[11:37:56][Step 1/1]        130│             cwd=project_path,
[11:37:56][Step 1/1]        131│             io=self._io,
[11:37:56][Step 1/1] 
[11:37:56][Step 1/1]    2  /usr/local/lib/python3.11/site-packages/poetry/factory.py:95 in create_poetry
[11:37:56][Step 1/1]         93│ 
[11:37:56][Step 1/1]         94│         poetry.set_pool(
[11:37:56][Step 1/1]      →  95│             self.create_pool(
[11:37:56][Step 1/1]         96│                 config,
[11:37:56][Step 1/1]         97│                 poetry.local_config.get("source", []),
[11:37:56][Step 1/1] 
[11:37:56][Step 1/1]    1  /usr/local/lib/python3.11/site-packages/poetry/factory.py:172 in create_pool
[11:37:56][Step 1/1]        170│                 io.write_line(message)
[11:37:56][Step 1/1]        171│ 
[11:37:56][Step 1/1]      → 172│             pool.add_repository(repository, priority=priority)
[11:37:56][Step 1/1]        173│             if repository.name.lower() == "pypi":
[11:37:56][Step 1/1]        174│                 explicit_pypi = True
[11:37:56][Step 1/1] 
[11:37:56][Step 1/1]   ValueError
[11:37:56][Step 1/1] 
[11:37:56][Step 1/1]   A repository with name torch-gpu-linux was already added.
[11:37:56][Step 1/1] 
[11:37:56][Step 1/1]   at /usr/local/lib/python3.11/site-packages/poetry/repositories/repository_pool.py:132 in add_repository
[11:37:56][Step 1/1]       128│         Adds a repository to the pool.
[11:37:56][Step 1/1]       129│         """
[11:37:56][Step 1/1]       130│         repository_name = repository.name.lower()
[11:37:56][Step 1/1]       131│         if self.has_repository(repository_name):
[11:37:56][Step 1/1]     → 132│             raise ValueError(
[11:37:56][Step 1/1]       133│                 f"A repository with name {repository_name} was already added."
[11:37:56][Step 1/1]       134│             )
[11:37:56][Step 1/1]       135│ 
[11:37:56][Step 1/1]       136│         if default or secondary:
[11:37:56][Step 1/1] Process exited with code 1

[dimbleby]

does not reproduce - as expected I see an error about connecting to "my_repo" but no sign of the error that you are reporting

$ poetry publish --build -r my_repo
Building Foo (0)
  - Building sdist
  - Built foo-0.tar.gz
  - Building wheel
  - Built foo-0-py3-none-any.whl

Publishing Foo (0) to my_repo
 - Uploading foo-0-py3-none-any.whl FAILED

Connection Error: We were unable to connect to the repository, ensure the url is correct and can be reached.

please provide a way to reproduce

[Jakobhenningjensen]

I don't know how else I should reproduce apart from the code above, since it's the one causing the error.

If I remove the torch-gpu-linux source (and just have torch="2.3.0") it works fine

[dimbleby]

maybe provide a dockerfile in which the error happens, then we should get past any "works on my computer" differences

(still works on my computer, though!)

[Jakobhenningjensen]

Just being curious; if you use the "-vvv" flag for publish, is "pytorch-gpu" only being added once?

[Jakobhenningjensen]

For some odd reason, changing the version from "0" to "1" in my .toml file seemed to work. I also added "PyPI" as source for the "darwin" source, thus I don't know which of them did the trick but .. 🤷

[dimbleby]

perhaps you had previously made changes but forgot to hit "save"...

well anyway: I guess either provide a repro or close this out, please

[bmarroquin]

It looks like it is adding torch-gpu-linux twice because it is configured in the poetry config as a repository and the pyproject.toml as a source. Poetry merges these 2 at run time and that is probably why you are running into this specific issue. Remove it from the poetry config and try again.

[Jakobhenningjensen]

It looks like it is adding torch-gpu-linux twice because it is configured in the poetry config as a repository and the pyproject.toml as a source. Poetry merges these 2 at run time and that is probably why you are running into this specific issue. Remove it from the poetry config and try again.

Where is the poetry config? I have only modified the pyproject.toml

[bmarroquin]

I always forget where it is but it doesn't matter, you can modify and see the contents using the 'poetry config' command.

'poetry config --unset repositories.' should do it I think.