Resolution of dependencies in directories fails to respect explicit source

[pcapel]

Description

When using a mono-repo with dependencies that are sub-directories, installation works fine, but when adding/removing deps, the resolution process fails because an internal dependency can't be found on PyPi. This happens because the source information isn't propagated correctly from the Provider.

This is admittedly a pretty niche issue for our project structure. As best as I can tell, the explicit nature of the repository isn't pulled into the _explicit_sources in the provider from the DirectoryPackage that actually holds the definition.

Workarounds

I was able to mitigate by modifying the source file directly for Provider, changing from this:

https://github.com/python-poetry/poetry/blob/a74d3a79028535baf48073c969d327978d87d6c9/src/poetry/puzzle/provider.py#L708

To this:

if explicit_source := self._explicit_sources.get(dependency.name) or package.source_reference:

This has the effect (as far as I can tell) that the source pulled from the lock file will always be respected, so I don't know if it's a valid long term fix.

Poetry Installation Method

other

Operating System

Ubuntu 22.04.4

Poetry Version

1.8.3

Poetry Configuration

cache-dir = "/home/Philip.Capel/.cache/pypoetry"                                                                                                                                                          
experimental.system-git-client = false                                                                                                                                                                    
installer.max-workers = null                                                                                                                                                                              
installer.modern-installation = true                                                                                                                                                                      
installer.no-binary = null                                                                                                                                                                                
installer.parallel = true                                                                                                                                                                                 
keyring.enabled = true                                                                                                                                                                                    
repositories.internal.url = "https://pypi.dev.internal.tools"                                                                                                                                               
solver.lazy-wheel = true                                                                                                                                                                                  
virtualenvs.create = true                                                                                                                                                                                 
virtualenvs.in-project = null                                                                                                                                                                             
virtualenvs.options.always-copy = false                                                                                                                                                                   
virtualenvs.options.no-pip = false                                                                                                                                                                        
virtualenvs.options.no-setuptools = false                                                                                                                                                                 
virtualenvs.options.system-site-packages = false                                                                                                                                                          
virtualenvs.path = "{cache-dir}/virtualenvs"  # /home/Philip.Capel/.cache/pypoetry/virtualenvs                                                                                                            
virtualenvs.prefer-active-python = true                                                                                                                                                                   
virtualenvs.prompt = "{project_name}-py{python_version}"                                                                                                                                                  
warnings.export = true

Python Sysconfig

No response

Example pyproject.toml

You will need to have a mono-repo style project, where there is a top level package defined by the following:

[tool.poetry.dependencies]                                                                                                                                                                                                                                              
python = ">=3.10,<3.11"                                                                                                                                                                            
some-sub-package = {path = "./components/some-sub-package", develop = true, optional = true}

Within the some-sub-package if you define the internal source and point to it explicitly in the dependency you could have something like:

[tool.poetry.dependencies]                                                                                                                                                                                                                    
dep_that_breaks = {version = "^1.12.0", source = "internal"}                                                                                                                                                                                                                                                                                                                                                                   

[[tool.poetry.source]]                                                                                                                                                                                                                                                   
name = "internal"                                                                                                                                                                                                                                                         
url = "https://pypi.dev.internal.tools"                                                                                                                                                                                                                                   
priority = "explicit"     

Poetry Runtime Logs

Loading configuration file /home/Philip.Capel/.config/pypoetry/config.toml
Adding repository evozyne (https://pypi.dev.internal.tools) and setting it as explicit
Using virtualenv: /home/Philip.Capel/projects/mew/.venv
Updating dependencies
Resolving dependencies...
# redacted
Source (PyPI): Getting info for dep_that_breaks (0.4) from PyPI
[urllib3:urllib3.connectionpool] https://pypi.org:443 "GET /pypi/dep_that_breaks/0.4/json HTTP/11" 404 24
Falling back to installed packages to discover metadata for dep_that_breaks
Found 0 compatible packages for dep_that_breaks
   1: Version solving took 0.500 seconds.
   1: Tried 1 solutions.

[dimbleby]

there is no way to pass source information through standard metadata, it is expected that a parent project knows nothing about sources defined in a dependency.