Git repository and package name mismatch

[antoineeudes]

• Poetry version: 1.2.2
• Python version: 3.8.2
• OS version and name: macOS Monterey
• pyproject.toml: https://gist.github.com/antoineeudes/dfccd21bb13dd5d55e8bf434b7d59cdd

• [x] I am on the latest stable Poetry version, installed using a recommended method.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] I have consulted the FAQ and blog for any relevant entries or release notes.
• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

Hello Poetry team!

I am trying to install a dependency from a private Gitlab repository. Let's call the repository foo and the library bar. In my pyproject.toml, I added the following line:

bar = {git = "ssh://git@gitlab.com/xxx/foo.git", develop = true, tag = "v0.13.1"}

After executing poetry install, I see that the library is successfully installed:

~ poetry run python
Python 3.8.2 (v3.8.2:7b3ab5921f, Feb 24 2020, 17:52:18)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import bar
>>> bar.__version__
'v0.13.1'

But, when I try to install another dependency, I get an error:

~ poetry add pandas

No git repository was found at /Users/me/Library/Caches/pypoetry/virtualenvs/xxxxxxxxx-py3.8/src/bar

It appears the git repository is not found in the virtual environment under the path src/bar. However, I can see the repo is there under the path src/foo. I guess there could be a confusion between the name of the library and the name of the git repository.

~ ls /Users/me/Library/Caches/pypoetry/virtualenvs/xxxxxxxxx-py3.8/src/
foo

Here is the complete log:

~  poetry -vvv add pandas
Loading configuration file /Users/me/Library/Preferences/pypoetry/config.toml
Using virtualenv: /Users/me/Library/Caches/pypoetry/virtualenvs/xxxxxxxxx-py3.8

  Stack trace:

  18  ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/cleo/application.py:329 in run
       327│
       328│             try:
     → 329│                 exit_code = self._run(io)
       330│             except Exception as e:
       331│                 if not self._catch_exceptions:

  17  ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/poetry/console/application.py:185 in _run
       183│         self._load_plugins(io)
       184│
     → 185│         exit_code: int = super()._run(io)
       186│         return exit_code
       187│

  16  ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/cleo/application.py:423 in _run
       421│             io.input.set_stream(stream)
       422│
     → 423│         exit_code = self._run_command(command, io)
       424│         self._running_command = None
       425│

  15  ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/cleo/application.py:465 in _run_command
       463│
       464│         if error is not None:
     → 465│             raise error
       466│
       467│         return event.exit_code

  14  ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/cleo/application.py:446 in _run_command
       444│
       445│         try:
     → 446│             self._event_dispatcher.dispatch(event, COMMAND)
       447│
       448│             if event.command_should_run():

  13  ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/cleo/events/event_dispatcher.py:23 in dispatch
        21│
        22│         if listeners:
     →  23│             self._do_dispatch(listeners, event_name, event)
        24│
        25│         return event

  12  ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/cleo/events/event_dispatcher.py:84 in _do_dispatch
        82│                 break
        83│
     →  84│             listener(event, event_name, self)
        85│
        86│     def _sort_listeners(self, event_name: str) -> None:

  11  ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/poetry/console/application.py:316 in configure_installer_for_event
       314│             return
       315│
     → 316│         cls.configure_installer_for_command(command, event.io)
       317│
       318│     @staticmethod

  10  ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/poetry/console/application.py:323 in configure_installer_for_command
       321│
       322│         poetry = command.poetry
     → 323│         installer = Installer(
       324│             io,
       325│             command.env,

   9  ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/poetry/installation/installer.py:79 in __init__
        77│         self._installer = self._get_installer()
        78│         if installed is None:
     →  79│             installed = self._get_installed()
        80│
        81│         self._installed_repository = installed

   8  ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/poetry/installation/installer.py:573 in _get_installed
       571│
       572│     def _get_installed(self) -> InstalledRepository:
     → 573│         return InstalledRepository.load(self._env)
       574│

   7  ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/poetry/repositories/installed_repository.py:291 in load
       289│                     continue
       290│
     → 291│                 package = cls.create_package_from_distribution(distribution, env)
       292│
       293│                 if with_dependencies:

   6  ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/poetry/repositories/installed_repository.py:158 in create_package_from_distribution
       156│                 source_url,
       157│                 source_reference,
     → 158│             ) = cls.get_package_vcs_properties_from_path(
       159│                 env.path / "src" / canonicalize_name(distribution.metadata["name"])
       160│             )

   5  ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/poetry/repositories/installed_repository.py:89 in get_package_vcs_properties_from_path
        87│         from poetry.vcs.git import Git
        88│
     →  89│         info = Git.info(repo=src)
        90│         return "git", info.origin, info.revision
        91│

   4  ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/poetry/vcs/git/backend.py:174 in info
       172│     @classmethod
       173│     def info(cls, repo: Repo | Path | str) -> GitRepoLocalInfo:
     → 174│         return GitRepoLocalInfo(repo=repo)
       175│
       176│     @staticmethod

   3  <string>:3 in __init__
         1│

   2  ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/poetry/vcs/git/backend.py:144 in __post_init__
       142│
       143│     def __post_init__(self, repo: Repo | Path | str) -> None:
     → 144│         repo = Git.as_repo(repo=repo) if not isinstance(repo, Repo) else repo
       145│         self.origin = Git.get_remote_url(repo=repo, remote="origin")
       146│         self.revision = Git.get_revision(repo=repo)

   1  ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/poetry/vcs/git/backend.py:152 in as_repo
       150│     @staticmethod
       151│     def as_repo(repo: Path | str) -> Repo:
     → 152│         return Repo(str(repo))
       153│
       154│     @staticmethod

  NotGitRepository

  No git repository was found at /Users/me/Library/Caches/pypoetry/virtualenvs/xxxxxxxxx-py3.8/src/bar

  at ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/dulwich/repo.py:1126 in __init__
      1122│             elif (os.path.isdir(os.path.join(root, OBJECTDIR))
      1123│                     and os.path.isdir(os.path.join(root, REFSDIR))):
      1124│                 bare = True
      1125│             else:
    → 1126│                 raise NotGitRepository(
      1127│                     "No git repository was found at %(path)s" % dict(path=root)
      1128│                 )
      1129│
      1130│         self.bare = bare

Thank you !

[neersighted]

I updated the title to describe what is actually going on; overall I think our VCS support breaks when subdirectories or mismatched names are involved, and this may be a duplicate of an existing issue. Not sure who to ping on VCS issues, but this entire subject could use some eyes, so I'll try to bother the other maintainers about it.

[JanMalte]

This got broken with some version between 1.1.14 and 1.2.2

Maybe by introducing canonicalize_name with 1.2.0: https://github.com/python-poetry/poetry/issues/4770#issuecomment-1179560514

The same happens with the dash/underscore transformation/mapping:

poetry 1.1.14

# pyproject.toml
abas-erp-api-client = {git = "ssh://git@gitlab.XXXX/abas_erp_api_client.git", develop = true, rev = "main"}

$ poetry install

$ ls venv/src
abas-erp-api-client

$ poetry update
No dependencies to install or update

poetry 1.2.2

# pyproject.toml
abas-erp-api-client = {git = "ssh://git@gitlab.XXXX/abas_erp_api_client.git", develop = true, rev = "main"}

$ poetry install

$ ls venv/src
abas_erp_api_client

$ poetry update
No git repository was found at /Users/maltegerth/test-poetry-1.2.2/venv/src/abas-erp-api-client

[neersighted]

This got broken with some version between 1.1.14 and 1.2.2

I don't think this assertion is correct per-se; mismatched names have always been an issue. We've added more features that can trigger this, and made our name handling code more robust. But I think this has always been fundamentally naive/broken, and recent changes have just increased the surface area.

[neersighted]

Related: #6958

[PhilMarsh]

Here's a possible workaround for anyone else hitting this.

I ran into this while trying to upgrade from Poetry 1.1 and I found poetry install only fails when the dependency has develop = true. So I removed develop = true in pyproject.toml, manually updated develop = false in poetry.lock, and poetry env remove'd the broken env. Now poetry install works every time for Poetry 1.2+.

This does mean we can't use develop = true. For anyone that needs to develop locally, this is definitely not ideal. But it should be fine if you're just pinning to git commits to work around release issues (like me), because you're not actually working on the dependencies locally. (I don't actually know why we committed develop = true to our repo in the first place. Persisting a local dev flag seems like a bug. Unless I'm missing something about git deps?)

[kkroening]

All four of these issues appear to be the same issue (AFAICT):

• https://github.com/python-poetry/poetry/issues/6958
• https://github.com/python-poetry/poetry/issues/7113 (this issue)
• https://github.com/python-poetry/poetry/issues/7181
• https://github.com/python-poetry/poetry/issues/7949

Of the four, this issue (#7113) seems to get to the essence of the fundamental problem, nicely summarized by @antoineeudes :

I guess there could be a confusion between the name of the library and the name of the git repository.

I would guess that this could be solved with the "Potential fix" described here: https://github.com/python-poetry/poetry/issues/7181#issuecomment-1427120155

(... but I haven't worked on Poetry, so perhaps that's naive)

I'm guessing that a significant number of users are running into this problem since there are so many different issues about it, and it likely prevents anyone from using Poetry with any sort of private git monorepos (or even just repos with more than one package).

[ralbertazzi]

I can't seem to reproduce it on Poetry 1.5.1. Here's my pyproject.toml

[tool.poetry]
name = "poetry-git"
version = "0.1.0"
description = ""
authors = ["Riccardo Albertazzi <foo@bar.com>"]
readme = "README.md"
packages = [{include = "poetry_git"}]

[tool.poetry.dependencies]
python = "^3.9"

# A plain simple fork of requests
requests = {git = "https://github.com/ralbertazzi/rfork.git", develop = true}

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

every command succeeds.

❯ poetry install
Creating virtualenv poetry-git in /<path-to-project>/.venv
Installing dependencies from lock file

Package operations: 5 installs, 0 updates, 0 removals

  • Installing certifi (2023.5.7)
  • Installing charset-normalizer (3.1.0)
  • Installing idna (3.4)
  • Installing urllib3 (2.0.2)
  • Installing requests (2.31.0 6e5b15d)

Installing the current project: poetry-git (0.1.0)

❯ poetry update
Updating dependencies
Resolving dependencies... (1.6s)

Package operations: 0 installs, 1 update, 0 removals

  • Updating requests (2.31.0 /<path-to-project>/.venv/src/rfork -> 2.31.0 6e5b15d)

❯ poetry add redis
Using version ^4.5.5 for redis

Updating dependencies
Resolving dependencies... (0.1s)

Package operations: 2 installs, 1 update, 0 removals

  • Installing async-timeout (4.0.2)
  • Installing redis (4.5.5)
  • Updating requests (2.31.0 /<path-to-project>/.venv/src/rfork -> 2.31.0 6e5b15d)

Here's the poetry.lock entry for reference:

[[package]]
name = "requests"
version = "2.31.0"
description = "Python HTTP for Humans."
optional = false
python-versions = ">=3.7"
files = []
develop = true

[package.dependencies]
certifi = ">=2017.4.17"
charset_normalizer = ">=2,<4"
idna = ">=2.5,<4"
urllib3 = ">=1.21.1,<3"

[package.extras]
socks = ["PySocks (>=1.5.6,!=1.5.7)"]
use-chardet-on-py3 = ["chardet (>=3.0.2,<6)"]

[package.source]
type = "git"
url = "https://github.com/ralbertazzi/rfork.git"
reference = "HEAD"
resolved_reference = "6e5b15d542a4e85945fd72066bb6cecbc3a82191"

I'd be great if you could provide a way to reproduce the issue on the latest Poetry

[ralbertazzi]

For doubt's sake I also tried the ssh specification and that also works:

requests = {git = "ssh://git@github.com/ralbertazzi/rfork.git", develop = true}

[JonathanRayner]

@ralbertazzi can you try poetry install with this repo https://github.com/JonathanRayner/some_other_repo ? Here I have 5 packages that are each slower clones, hopefully more consistently triggering the issue.