search

python-poetry / poetry

Python packaging and dependency management made easy
https://python-poetry.org
MIT License
30.93k stars 2.25k forks source link

Dependency resolution problem with non-pypi repository #706

Closed ojii closed 5 years ago

ojii commented 5 years ago

Version: Poetry 0.12.10

I have a private PyPI server for company internal packages. Something in poetry seems to get confused when that private package has public dependencies. Specifically this case:

private-package depends on public-package which itself has further (public) dependencies.

  1. poetry add private-package
  2. poetry add public-package fails with [PackageNotFound]: Package [<private-package>] not found.

However poetry add works for public packages not depended on by private-package and also works for public packages which have no dependencies on their own, even if depended on by private-package.

ojii commented 5 years ago

Here's the -vvvvv log (company-pypi is a private pypi server, private-package depends on aiohttp):

$ poetry add -vvvvv  aiohttp
Using virtualenv: /Users/jonas.obrist/Library/Caches/pypoetry/virtualenvs/repro-py3.7
company-pypi: 146 packages found for aiohttp *
Using version ^3.4 for aiohttp

Updating dependencies
Resolving dependencies...
   1: fact: repro is 0.1.0
   1: derived: repro
   1: fact: repro depends on private-package (^0.0.11)
   1: fact: repro depends on attrs (^18.2)
   1: fact: repro depends on requests (^2.20)
   1: fact: repro depends on aiohttp (^3.4)
   1: selecting repro (0.1.0)
   1: derived: aiohttp (^3.4)
   1: derived: requests (^2.20)
   1: derived: attrs (^18.2)
   1: derived: private-package (^0.0.11)
company-pypi: 5 packages found for aiohttp >=3.4,<4.0
company-pypi: The cache for aiohttp 3.4.4 is outdated. Refreshing.
company-pypi: Downloading sdist: aiohttp-3.4.4.tar.gz
   1: fact: aiohttp (3.4.4) depends on attrs (>=17.3.0)
   1: fact: aiohttp (3.4.4) depends on chardet (>=2.0,<4.0)
   1: fact: aiohttp (3.4.4) depends on multidict (>=4.0,<5.0)
   1: fact: aiohttp (3.4.4) depends on async_timeout (>=3.0,<4.0)
   1: fact: aiohttp (3.4.4) depends on yarl (>=1.0,<2.0)
   1: selecting aiohttp (3.4.4)
   1: derived: yarl (>=1.0,<2.0)
   1: derived: async_timeout (>=3.0,<4.0)
   1: derived: multidict (>=4.0,<5.0)
   1: derived: chardet (>=2.0,<4.0)
company-pypi: The cache for requests 2.20.1 is outdated. Refreshing.
company-pypi: Downloading wheel: requests-2.20.1-py2.py3-none-any.whl
   1: fact: requests (2.20.1) depends on chardet (>=3.0.2,<3.1.0)
   1: fact: requests (2.20.1) depends on idna (>=2.5,<2.8)
   1: fact: requests (2.20.1) depends on urllib3 (>=1.21.1,<1.25)
   1: fact: requests (2.20.1) depends on certifi (>=2017.4.17)
   1: selecting requests (2.20.1)
   1: derived: certifi (>=2017.4.17)
   1: derived: urllib3 (>=1.21.1,<1.25)
   1: derived: idna (>=2.5,<2.8)
   1: derived: chardet (>=3.0.2,<3.1.0)
company-pypi: The cache for attrs 18.2.0 is outdated. Refreshing.
company-pypi: Downloading wheel: attrs-18.2.0-py2.py3-none-any.whl
   1: selecting attrs (18.2.0)
company-pypi: The cache for private-package 0.0.11 is outdated. Refreshing.
company-pypi: Downloading wheel: private-package-0.0.11-py3-none-any.whl
   1: fact: private-package (0.0.11) depends on attrs (*)
   1: fact: private-package (0.0.11) depends on aiobotocore (*)
   1: fact: private-package (0.0.11) depends on boto3 (<=1.5)
   1: fact: private-package (0.0.11) depends on aiohttp (>3,<=3.0.9)
   1: derived: not private-package (0.0.11)
company-pypi: 0 packages found for private-package >0.0.11,<0.0.12
   1: Version solving took 8.155 seconds.
   1: Tried 1 solutions.

[PackageNotFound]  
Package [private-package] not found.           

Exception trace:
 /Users/jonas.obrist/.poetry/lib/poetry/_vendor/py3.7/cleo/application.py in run() at line 94
   status_code = self.do_run(input_, output_)
 /Users/jonas.obrist/.poetry/lib/poetry/console/application.py in do_run() at line 88
   return super(Application, self).do_run(i, o)
 /Users/jonas.obrist/.poetry/lib/poetry/_vendor/py3.7/cleo/application.py in do_run() at line 197
   status_code = command.run(input_, output_)
 /Users/jonas.obrist/.poetry/lib/poetry/console/commands/command.py in run() at line 77
   return super(BaseCommand, self).run(i, o)
 /Users/jonas.obrist/.poetry/lib/poetry/_vendor/py3.7/cleo/commands/base_command.py in run() at line 146
   status_code = self.execute(input_, output_)
 /Users/jonas.obrist/.poetry/lib/poetry/_vendor/py3.7/cleo/commands/command.py in execute() at line 107
   return self.handle()
 /Users/jonas.obrist/.poetry/lib/poetry/console/commands/add.py in handle() at line 139
   status = installer.run()
 /Users/jonas.obrist/.poetry/lib/poetry/installation/installer.py in run() at line 76
   self._do_install(local_repo)
 /Users/jonas.obrist/.poetry/lib/poetry/installation/installer.py in _do_install() at line 158
   ops = solver.solve(use_latest=self._whitelist)
 /Users/jonas.obrist/.poetry/lib/poetry/puzzle/solver.py in solve() at line 38
   packages, depths = self._solve(use_latest=use_latest)
 /Users/jonas.obrist/.poetry/lib/poetry/puzzle/solver.py in _solve() at line 171
   self._package, self._provider, locked=locked, use_latest=use_latest
 /Users/jonas.obrist/.poetry/lib/poetry/mixology/__init__.py in resolve_version() at line 7
   return solver.solve()
 /Users/jonas.obrist/.poetry/lib/poetry/mixology/version_solver.py in solve() at line 79
   next = self._choose_package_version()
 /Users/jonas.obrist/.poetry/lib/poetry/mixology/version_solver.py in _choose_package_version() at line 354
   packages = self._provider.search_for(dependency)
 /Users/jonas.obrist/.poetry/lib/poetry/puzzle/provider.py in search_for() at line 146
   allow_prereleases=dependency.allows_prereleases(),
 /Users/jonas.obrist/.poetry/lib/poetry/repositories/pool.py in find_packages() at line 65
   name, constraint, extras=extras, allow_prereleases=allow_prereleases
 /Users/jonas.obrist/.poetry/lib/poetry/repositories/pypi_repository.py in find_packages() at line 104
   info = self.get_package_info(name)
 /Users/jonas.obrist/.poetry/lib/poetry/repositories/pypi_repository.py in get_package_info() at line 228
   name, lambda: self._get_package_info(name)
 /Users/jonas.obrist/.poetry/lib/poetry/_vendor/py3.7/cachy/repository.py in remember_forever() at line 174
   val = value(callback)
 /Users/jonas.obrist/.poetry/lib/poetry/_vendor/py3.7/cachy/helpers.py in value() at line 6
   return val()
 /Users/jonas.obrist/.poetry/lib/poetry/repositories/pypi_repository.py in <lambda>() at line 228
   name, lambda: self._get_package_info(name)
 /Users/jonas.obrist/.poetry/lib/poetry/repositories/pypi_repository.py in _get_package_info() at line 234
   raise PackageNotFound("Package [{}] not found.".format(name))

add [-D|--dev] [--git GIT] [--path PATH] [-E|--extras EXTRAS] [--optional] [--python PYTHON] [--platform PLATFORM] [--allow-prereleases] [--dry-run] [--] <name> (<name>)...
ojii commented 5 years ago

Is there a way to make private repositories behave like --extra-index-url rather than --index-url (in pip)? Ideally I'd like poetry to try pypi.org first, then fall back to the company pypi.

AzMoo commented 5 years ago

Hi, we're getting this same issue with a private repo.

@ojii Did you find a workaround?

Virtualenv <parent-package>-py3.6 already exists.
Using virtualenv: /Users/matt/Library/Caches/pypoetry/virtualenvs/<parent-package>-py3.6
Updating dependencies
Resolving dependencies...
   1: fact: <parent-package> is 1.3.3
   1: derived: <parent-package>
   1: fact: <parent-package> depends on django-cms (=3.5.1)
   1: fact: <parent-package> depends on django (=1.11)
   1: fact: <parent-package> depends on django-filer (=1.3.0)
   1: fact: <parent-package> depends on djangocms-text-ckeditor (=3.5.3)
   1: fact: <parent-package> depends on djangocms-link (=2.1.2)
   1: fact: <parent-package> depends on djangocms-file (=2.0.2)
   1: fact: <parent-package> depends on djangocms-picture (=2.0.6)
   1: fact: <parent-package> depends on djangocms-video (=2.0.4)
   1: fact: <parent-package> depends on djangocms-googlemap (=1.1.1)
   1: fact: <parent-package> depends on djangocms-snippet (=2.0.0)
   1: fact: <parent-package> depends on djangocms-style (=2.0.2)
   1: fact: <parent-package> depends on djangocms-column (=1.0.2)
   1: fact: <parent-package> depends on dj-database-url (=0.5.0)
   1: fact: <parent-package> depends on django-csp (=3.3)
   1: fact: <parent-package> depends on psycopg2-binary (^2.7)
   1: fact: <parent-package> depends on python-decouple (=3.1)
   1: fact: <parent-package> depends on waitress (=1.1.0)
   1: fact: <parent-package> depends on easy-thumbnails (=2.4.2)
   1: fact: <parent-package> depends on boto3 (^1.6)
   1: fact: <parent-package> depends on cmsplugin-contact (=1.1.3)
   1: fact: <parent-package> depends on django-csp-admin (=0.1.0)
   1: fact: <parent-package> depends on django-storages (=1.6.5)
   1: fact: <parent-package> depends on djangocms_forms (=0.2.5)
   1: fact: <parent-package> depends on django-compressor (=2.2)
   1: fact: <parent-package> depends on <private-package> (=1.0.2)
   1: fact: <parent-package> depends on raven (^6.9)
   1: fact: <parent-package> depends on whitenoise (^4.1)1
   1: fact: <parent-package> depends on python-decouple-aws (^0.2.3)
   1: fact: <parent-package> depends on django-alive (^1.0)
   1: fact: <parent-package> depends on pre-commit (^1.12)
   1: fact: <parent-package> depends on prospector (^1.1)
   1: fact: <parent-package> depends on pytest (^3.9)
   1: fact: <parent-package> depends on pytest-django (^3.4)
   1: fact: <parent-package> depends on pytest-pythonpath (^0.7.3)
   1: selecting <parent-package> (1.3.3)
   1: derived: pytest-pythonpath (^0.7.3)
   1: derived: pytest-django (^3.4)
   1: derived: pytest (^3.9)
   1: derived: prospector (^1.1)
   1: derived: pre-commit (^1.12)
   1: derived: django-alive (^1.0)
   1: derived: python-decouple-aws (^0.2.3)
   1: derived: whitenoise (^4.1)
   1: derived: raven (^6.9)
   1: derived: <private-package> (=1.0.2)
   1: derived: django-compressor (=2.2)
   1: derived: djangocms_forms (=0.2.5)
   1: derived: django-storages (=1.6.5)
   1: derived: django-csp-admin (=0.1.0)
   1: derived: cmsplugin-contact (=1.1.3)
   1: derived: boto3 (^1.6)
   1: derived: easy-thumbnails (=2.4.2)
   1: derived: waitress (=1.1.0)
   1: derived: python-decouple (=3.1)
   1: derived: psycopg2-binary (^2.7)
   1: derived: django-csp (=3.3)
   1: derived: dj-database-url (=0.5.0)
   1: derived: djangocms-column (=1.0.2)
   1: derived: djangocms-style (=2.0.2)
   1: derived: djangocms-snippet (=2.0.0)
   1: derived: djangocms-googlemap (=1.1.1)
   1: derived: djangocms-video (=2.0.4)
   1: derived: djangocms-picture (=2.0.6)
   1: derived: djangocms-file (=2.0.2)
   1: derived: djangocms-link (=2.1.2)
   1: derived: djangocms-text-ckeditor (=3.5.3)
   1: derived: django-filer (=1.3.0)
   1: derived: django (=1.11)
   1: derived: django-cms (=3.5.1)
cmv-pkg-shed: 0 packages found for pytest-pythonpath >=0.7.3,<0.8.0
PyPI: No release information found for pytest-pythonpath-0.4, skipping
PyPI: 1 packages found for pytest-pythonpath >=0.7.3,<0.8.0
   1: fact: pytest-pythonpath (0.7.3) depends on pytest (>=2.5.2)
   1: selecting pytest-pythonpath (0.7.3)
cmv-pkg-shed: 0 packages found for pytest-django >=3.4,<4.0
PyPI: 5 packages found for pytest-django >=3.4,<4.0
   1: fact: pytest-django (3.4.5) depends on pytest (>=3.6)
   1: selecting pytest-django (3.4.5)
cmv-pkg-shed: 0 packages found for pytest >=3.9,<4.0
PyPI: 5 packages found for pytest >=3.9,<4.0
   1: fact: pytest (3.10.1) depends on py (>=1.5.0)
   1: fact: pytest (3.10.1) depends on six (>=1.10.0)
   1: fact: pytest (3.10.1) depends on attrs (>=17.4.0)
   1: fact: pytest (3.10.1) depends on more-itertools (>=4.0.0)
   1: fact: pytest (3.10.1) depends on atomicwrites (>=1.0)
   1: fact: pytest (3.10.1) depends on pluggy (>=0.7)
   1: fact: pytest (3.10.1) depends on pathlib2 (>=2.2.0)
   1: fact: pytest (3.10.1) depends on colorama (*)
   1: selecting pytest (3.10.1)
   1: derived: colorama (*)
   1: derived: pathlib2 (>=2.2.0)
   1: derived: pluggy (>=0.7)
   1: derived: atomicwrites (>=1.0)
   1: derived: more-itertools (>=4.0.0)
   1: derived: attrs (>=17.4.0)
   1: derived: six (>=1.10.0)
   1: derived: py (>=1.5.0)
cmv-pkg-shed: 0 packages found for prospector >=1.1,<2.0
PyPI: 8 packages found for prospector >=1.1,<2.0
   1: fact: prospector (1.1.6.2) depends on pylint-plugin-utils (>=0.2.6)
   1: fact: prospector (1.1.6.2) depends on pylint-celery (0.3)
   1: fact: prospector (1.1.6.2) depends on pylint-flask (0.5)
   1: fact: prospector (1.1.6.2) depends on requirements-detector (>=0.6)
   1: fact: prospector (1.1.6.2) depends on setoptconf (>=0.2.0)
   1: fact: prospector (1.1.6.2) depends on dodgy (>=0.1.9)
   1: fact: prospector (1.1.6.2) depends on pyyaml (*)
   1: fact: prospector (1.1.6.2) depends on mccabe (>=0.5.0)
   1: fact: prospector (1.1.6.2) depends on pyflakes (>=0.8.1,<2.0.0)
   1: fact: prospector (1.1.6.2) depends on pycodestyle (>=2.0.0,<=2.4.0)
   1: fact: prospector (1.1.6.2) depends on pep8-naming (>=0.3.3,<=0.4.1)
   1: fact: prospector (1.1.6.2) depends on pydocstyle (>=2.0.0)
   1: fact: prospector (1.1.6.2) depends on pylint (2.1.1)
   1: fact: prospector (1.1.6.2) depends on pylint-django (2.0.2)
   1: fact: prospector (1.1.6.2) depends on astroid (2.0.4)
   1: selecting prospector (1.1.6.2)
   1: derived: astroid (2.0.4)
   1: derived: pylint-django (2.0.2)
   1: derived: pylint (2.1.1)
   1: derived: pydocstyle (>=2.0.0)
   1: derived: pep8-naming (>=0.3.3,<=0.4.1)
   1: derived: pycodestyle (>=2.0.0,<=2.4.0)
   1: derived: pyflakes (>=0.8.1,<2.0.0)
   1: derived: mccabe (>=0.5.0)
   1: derived: pyyaml (*)
   1: derived: dodgy (>=0.1.9)
   1: derived: setoptconf (>=0.2.0)
   1: derived: requirements-detector (>=0.6)
   1: derived: pylint-flask (0.5)
   1: derived: pylint-celery (0.3)
   1: derived: pylint-plugin-utils (>=0.2.6)
cmv-pkg-shed: 0 packages found for pylint-django 2.0.2
PyPI: 1 packages found for pylint-django 2.0.2
cmv-pkg-shed: 0 packages found for pylint-flask 0.5
PyPI: 1 packages found for pylint-flask 0.5
cmv-pkg-shed: 0 packages found for pylint-celery 0.3
PyPI: 1 packages found for pylint-celery 0.3
cmv-pkg-shed: 0 packages found for pre-commit >=1.12,<2.0
PyPI: 4 packages found for pre-commit >=1.12,<2.0
   1: fact: pre-commit (1.14.1) depends on aspy.yaml (*)
   1: fact: pre-commit (1.14.1) depends on cfgv (>=1.4.0)
   1: fact: pre-commit (1.14.1) depends on identify (>=1.0.0)
   1: fact: pre-commit (1.14.1) depends on importlib-metadata (*)
   1: fact: pre-commit (1.14.1) depends on nodeenv (>=0.11.1)
   1: fact: pre-commit (1.14.1) depends on pyyaml (*)
   1: fact: pre-commit (1.14.1) depends on six (*)
   1: fact: pre-commit (1.14.1) depends on toml (*)
   1: fact: pre-commit (1.14.1) depends on virtualenv (*)
   1: fact: pre-commit (1.14.1) depends on importlib-resources (*)
   1: selecting pre-commit (1.14.1)
   1: derived: importlib-resources (*)
   1: derived: virtualenv (*)
   1: derived: toml (*)
   1: derived: nodeenv (>=0.11.1)
   1: derived: importlib-metadata (*)
   1: derived: identify (>=1.0.0)
   1: derived: cfgv (>=1.4.0)
   1: derived: aspy.yaml (*)
cmv-pkg-shed: 0 packages found for django-alive >=1.0,<2.0
PyPI: 2 packages found for django-alive >=1.0,<2.0
   1: fact: django-alive (1.0.1) depends on django (*)
   1: selecting django-alive (1.0.1)
cmv-pkg-shed: 0 packages found for python-decouple-aws >=0.2.3,<0.3.0
PyPI: 1 packages found for python-decouple-aws >=0.2.3,<0.3.0
   1: fact: python-decouple-aws (0.2.3) depends on python-decouple (>=3.1,<4.0)
   1: fact: python-decouple-aws (0.2.3) depends on boto3 (>=1.7,<2.0)
   1: selecting python-decouple-aws (0.2.3)
   1: derived: boto3 (>=1.7,<2.0)
cmv-pkg-shed: 0 packages found for whitenoise >=4.1,<5.0
PyPI: 3 packages found for whitenoise >=4.1,<5.0
   1: selecting whitenoise (4.1.2)
cmv-pkg-shed: 0 packages found for raven >=6.9,<7.0
PyPI: 2 packages found for raven >=6.9,<7.0
   1: selecting raven (6.10.0)
cmv-pkg-shed: 0 packages found for <private-package> 1.0.2
   1: Version solving took 2.479 seconds.
   1: Tried 1 solutions.

[PackageNotFound]
Package [<private-package>] not found.

Exception trace:
 /Users/matt/.poetry/lib/poetry/_vendor/py3.6/cleo/application.py in run() at line 94
   status_code = self.do_run(input_, output_)
 /Users/matt/.poetry/lib/poetry/console/application.py in do_run() at line 88
   return super(Application, self).do_run(i, o)
 /Users/matt/.poetry/lib/poetry/_vendor/py3.6/cleo/application.py in do_run() at line 197
   status_code = command.run(input_, output_)
 /Users/matt/.poetry/lib/poetry/console/commands/command.py in run() at line 77
   return super(BaseCommand, self).run(i, o)
 /Users/matt/.poetry/lib/poetry/_vendor/py3.6/cleo/commands/base_command.py in run() at line 146
   status_code = self.execute(input_, output_)
 /Users/matt/.poetry/lib/poetry/_vendor/py3.6/cleo/commands/command.py in execute() at line 107
   return self.handle()
 /Users/matt/.poetry/lib/poetry/console/commands/update.py in handle() at line 41
   return installer.run()
 /Users/matt/.poetry/lib/poetry/installation/installer.py in run() at line 76
   self._do_install(local_repo)
 /Users/matt/.poetry/lib/poetry/installation/installer.py in _do_install() at line 158
   ops = solver.solve(use_latest=self._whitelist)
 /Users/matt/.poetry/lib/poetry/puzzle/solver.py in solve() at line 38
   packages, depths = self._solve(use_latest=use_latest)
 /Users/matt/.poetry/lib/poetry/puzzle/solver.py in _solve() at line 171
   self._package, self._provider, locked=locked, use_latest=use_latest
 /Users/matt/.poetry/lib/poetry/mixology/__init__.py in resolve_version() at line 7
   return solver.solve()
 /Users/matt/.poetry/lib/poetry/mixology/version_solver.py in solve() at line 79
   next = self._choose_package_version()
 /Users/matt/.poetry/lib/poetry/mixology/version_solver.py in _choose_package_version() at line 354
   packages = self._provider.search_for(dependency)
 /Users/matt/.poetry/lib/poetry/puzzle/provider.py in search_for() at line 146
   allow_prereleases=dependency.allows_prereleases(),
 /Users/matt/.poetry/lib/poetry/repositories/pool.py in find_packages() at line 65
   name, constraint, extras=extras, allow_prereleases=allow_prereleases
 /Users/matt/.poetry/lib/poetry/repositories/pypi_repository.py in find_packages() at line 104
   info = self.get_package_info(name)
 /Users/matt/.poetry/lib/poetry/repositories/pypi_repository.py in get_package_info() at line 228
   name, lambda: self._get_package_info(name)
 /Users/matt/.poetry/lib/poetry/_vendor/py3.6/cachy/repository.py in remember_forever() at line 174
   val = value(callback)
 /Users/matt/.poetry/lib/poetry/_vendor/py3.6/cachy/helpers.py in value() at line 6
   return val()
 /Users/matt/.poetry/lib/poetry/repositories/pypi_repository.py in <lambda>() at line 228
   name, lambda: self._get_package_info(name)
 /Users/matt/.poetry/lib/poetry/repositories/pypi_repository.py in _get_package_info() at line 234
   raise PackageNotFound("Package [{}] not found.".format(name))

update [--no-dev] [--dry-run] [--lock] [--] [<packages>]...
ojii commented 5 years ago

@AzMoo sadly my "workaround" for now is "not use poetry in projects that have non-pypi dependencies".

cipriantarta commented 5 years ago

Any news on this? Is there a good argument not to use --extra-index-url rather than --index-url?

scottyHH commented 5 years ago

Any update on this. I just spend quite some time on migrating the whole project to poetry and now i am stuck at this point.

ojii commented 5 years ago

Any news on this? Is there a good argument not to use --extra-index-url rather than --index-url?

I've actually thought about a good reason not to use --extra-index-url. Say you have a private package privpkg on your private repo. If someone now uploads privpkg to the public repo, using --extra-index-url would pull that in.

Maybe a better solution would be something like:

[tool.poetry.dependencies]
privpkg = { version = "^2.13.0", repo = "privaterepo" }

Though this raises the question how private and public dependencies of privpkg are handled, I guess the lockfile could take care of that?

scottyHH commented 5 years ago

My Problem is rather that I cannot install any private packages that I would install with pip like this: pip install some-package --extra-index-url=...

Even those that don't have any dependencies.

sdispater commented 5 years ago

@ojii I think this is fixed in the latest 1.0.0 prerelease (1.0.0a2). I changed a lot of things in the dependency resolution process. Can you tell me if that fixes your issue?

sdispater commented 5 years ago

Also can you test the 0.12.11 release?

scottyHH commented 5 years ago

I think I have the 0.12.11 release. But I will check it out tomorrow and come back to you. I will also test the prerelease.

ojii commented 5 years ago

@sdispater

Also can you test the 0.12.11 release?

It almost works. The problem I'm facing now is that it tries the private repo first, then the public one. Since our private repo is not a perfect mirror of pypi but rather a place where we upload our packages, some things have trouble installing (in my case cryptography since the private repo doesn't have wheels for some reason).

Also, I had to run poetry run pip install -U pip since the first attempt to poetry add <all my dependencies> failed. The second time it failed trying to build a wheel ("Could not build wheels for {} which use PEP 517 and cannot be installed directly ").

@ojii I think this is fixed in the latest 1.0.0 prerelease (1.0.0a2). I changed a lot of things in the dependency resolution process. Can you tell me if that fixes your issue?

Again the old pip screwed me over at first. (I really wished poetry updated pip after creating a venv) and then the same issue, it failed to get a wheel of cryptography from the private repo (only tar.gz) then failed to build it because I'm on OS X.

scottyHH commented 5 years ago

@sdispater

I am currently using the 0.12.11 release. There it is not working. I've updated to the prerelease you've mentioned and i get the same error message as before. When I install a public package from my own index-repository it works like a charm, when installing something last is private it can't find it anymore. (I've tested the URL through normal pip and pipenv and it works there).

Here is my traceback in a minimal configuration:

Using virtualenv: /home/cwittlinger/.cache/pypoetry/virtualenvs/repo-a09xFGWp-py3.6
Updating dependencies
Resolving dependencies...
   1: fact: repo is 0.1.0
   1: derived: repo
   1: fact: repo depends on priv_repo (*)
   1: selecting repo (0.1.0)
   1: derived: priv_repo (*)
   1: Version solving took 0.510 seconds.
   1: Tried 1 solutions.

[PackageNotFound]
Package [priv_repo] not found.

Traceback (most recent call last):
  File "/home/cwittlinger/.poetry/lib/poetry/_vendor/py3.6/clikit/console_application.py", line 131, in run
    status_code = command.handle(parsed_args, io)
  File "/home/cwittlinger/.poetry/lib/poetry/_vendor/py3.6/clikit/api/command/command.py", line 112, in handle
    status_code = self._do_handle(args, io)
  File "/home/cwittlinger/.poetry/lib/poetry/_vendor/py3.6/clikit/api/command/command.py", line 160, in _do_handle
    return getattr(handler, handler_method)(args, io, self)
  File "/home/cwittlinger/.poetry/lib/poetry/_vendor/py3.6/cleo/commands/command.py", line 92, in wrap_handle
    return self.handle()
  File "/home/cwittlinger/.poetry/lib/poetry/console/commands/update.py", line 37, in handle
    return installer.run()
  File "/home/cwittlinger/.poetry/lib/poetry/installation/installer.py", line 74, in run
    self._do_install(local_repo)
  File "/home/cwittlinger/.poetry/lib/poetry/installation/installer.py", line 156, in _do_install
    ops = solver.solve(use_latest=self._whitelist)
  File "/home/cwittlinger/.poetry/lib/poetry/puzzle/solver.py", line 38, in solve
    packages, depths = self._solve(use_latest=use_latest)
  File "/home/cwittlinger/.poetry/lib/poetry/puzzle/solver.py", line 171, in _solve
    self._package, self._provider, locked=locked, use_latest=use_latest
  File "/home/cwittlinger/.poetry/lib/poetry/mixology/__init__.py", line 7, in resolve_version
    return solver.solve()
  File "/home/cwittlinger/.poetry/lib/poetry/mixology/version_solver.py", line 79, in solve
    next = self._choose_package_version()
  File "/home/cwittlinger/.poetry/lib/poetry/mixology/version_solver.py", line 354, in _choose_package_version
    packages = self._provider.search_for(dependency)
  File "/home/cwittlinger/.poetry/lib/poetry/puzzle/provider.py", line 132, in search_for
    allow_prereleases=dependency.allows_prereleases(),
  File "/home/cwittlinger/.poetry/lib/poetry/repositories/pool.py", line 65, in find_packages
    name, constraint, extras=extras, allow_prereleases=allow_prereleases
  File "/home/cwittlinger/.poetry/lib/poetry/repositories/pypi_repository.py", line 105, in find_packages
    info = self.get_package_info(name)
  File "/home/cwittlinger/.poetry/lib/poetry/repositories/pypi_repository.py", line 238, in get_package_info
    name, lambda: self._get_package_info(name)
  File "/home/cwittlinger/.poetry/lib/poetry/_vendor/py3.6/cachy/repository.py", line 174, in remember_forever
    val = value(callback)
  File "/home/cwittlinger/.poetry/lib/poetry/_vendor/py3.6/cachy/helpers.py", line 6, in value
    return val()
  File "/home/cwittlinger/.poetry/lib/poetry/repositories/pypi_repository.py", line 238, in <lambda>
    name, lambda: self._get_package_info(name)
  File "/home/cwittlinger/.poetry/lib/poetry/repositories/pypi_repository.py", line 244, in _get_package_info
    raise PackageNotFound('Package [{}] not found.'.format(name))
scottyHH commented 5 years ago

I just figured out that even with --index-url= the normal pip install works, if I installed all the other dependencies manually. For some reasons poetry is still not able to find the package though.

My private repository is hosted on gemfury if that helps at all.

dusktreader commented 5 years ago

[EDIT] forgot to include pyproject.toml. now added below

@sdispater I ran into this issue today with a local devpi instance. I tried with poetry versions 0.12.11, 0.12.15, and 1.0.0a2. In each instance, when I tried to add my local package, poetry reports that it could not find the package. When searching for the package, it was not found either.

I jumped into the underlying virtual environment and tried using pip to install the package which seemed to work fine.

I'm able to publish to the local devpi repo using poetry without issue.

Here is some example output:

$ cat pyproject.toml
[tool.poetry]
name = "pa-mockup"
version = "0.1.0"
description = ""
authors = ["Tucker Beck <tucker.beck@-------.com>"]

[tool.poetry.dependencies]
python = "^3.7"
pendulum = "^2.0"
flask-apispec = "^0.8.0"
flask = "^1.0"
flask-buzz = "^0.1.14"
flask-jwt-extended = "^3.18"
flask-cors = "^3.0"
flask-mail = "^0.9.1"
flask-redis = "^0.3.0"

[tool.poetry.dev-dependencies]
pytest = "^3.0"
ipdb = "^0.12.0"

[[tool.poetry.source]]
name = "pa-devpi"
url = "http://---.---.--.--:3141/pa-team/pa/"

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"
$ poetry --version
Poetry 0.12.15
$ poetry add fetch-config --verbose=3
Using virtualenv: /home/tbeck/.cache/pypoetry/virtualenvs/dummy-py3.7
pa-devpi: 0 packages found for fetch-config *

[PackageNotFound]     
Package [fetch-config] not found.  

Exception trace:
 /home/tbeck/.poetry/lib/poetry/_vendor/py3.7/cleo/application.py in run() at line 94
   status_code = self.do_run(input_, output_)
 /home/tbeck/.poetry/lib/poetry/console/application.py in do_run() at line 88
   return super(Application, self).do_run(i, o)
 /home/tbeck/.poetry/lib/poetry/_vendor/py3.7/cleo/application.py in do_run() at line 197
   status_code = command.run(input_, output_)
 /home/tbeck/.poetry/lib/poetry/console/commands/command.py in run() at line 77
   return super(BaseCommand, self).run(i, o)
 /home/tbeck/.poetry/lib/poetry/_vendor/py3.7/cleo/commands/base_command.py in run() at line 146
   status_code = self.execute(input_, output_)
 /home/tbeck/.poetry/lib/poetry/_vendor/py3.7/cleo/commands/command.py in execute() at line 107
   return self.handle()
 /home/tbeck/.poetry/lib/poetry/console/commands/add.py in handle() at line 69
   packages, allow_prereleases=self.option("allow-prereleases")
 /home/tbeck/.poetry/lib/poetry/console/commands/init.py in _determine_requirements() at line 230
   requirement["name"], allow_prereleases=allow_prereleases
 /home/tbeck/.poetry/lib/poetry/console/commands/init.py in _find_best_version_for_package() at line 260
   name, required_version, allow_prereleases=allow_prereleases
 /home/tbeck/.poetry/lib/poetry/version/version_selector.py in find_best_candidate() at line 29
   package_name, constraint, allow_prereleases=allow_prereleases
 /home/tbeck/.poetry/lib/poetry/repositories/pool.py in find_packages() at line 65
   name, constraint, extras=extras, allow_prereleases=allow_prereleases
 /home/tbeck/.poetry/lib/poetry/repositories/pypi_repository.py in find_packages() at line 108
   info = self.get_package_info(name)
 /home/tbeck/.poetry/lib/poetry/repositories/pypi_repository.py in get_package_info() at line 241
   name, lambda: self._get_package_info(name)
 /home/tbeck/.poetry/lib/poetry/_vendor/py3.7/cachy/repository.py in remember_forever() at line 174
   val = value(callback)
 /home/tbeck/.poetry/lib/poetry/_vendor/py3.7/cachy/helpers.py in value() at line 6
   return val()
 /home/tbeck/.poetry/lib/poetry/repositories/pypi_repository.py in <lambda>() at line 241
   name, lambda: self._get_package_info(name)
 /home/tbeck/.poetry/lib/poetry/repositories/pypi_repository.py in _get_package_info() at line 247
   raise PackageNotFound("Package [{}] not found.".format(name))

add [-D|--dev] [--git GIT] [--path PATH] [-E|--extras EXTRAS] [--optional] [--python PYTHON] [--platform PLATFORM] [--allow-prereleases] [--dry-run] [--] <name> (<name>)...
$ poetry debug:info

Poetry
======

 * Version: 0.12.15
 * Python:  3.7.2

Virtualenv
==========

 * Python:         3.7.2
 * Implementation: CPython
 * Path:           /home/tbeck/.cache/pypoetry/virtualenvs/dummy-py3.7
 * Valid:          True

System
======

 * Platform: linux
 * OS:       posix
 * Python:   /home/tbeck/.pyenv/versions/3.7.2
$ cd /home/tbeck/.cache/pypoetry/virtualenvs/dummy-py3.7
total 16
drwxrwxr-x 2 tbeck tbeck 4096 May  6 16:59 bin
drwxrwxr-x 2 tbeck tbeck 4096 May  6 16:59 include
drwxrwxr-x 3 tbeck tbeck 4096 May  6 16:59 lib
lrwxrwxrwx 1 tbeck tbeck    3 May  6 16:59 lib64 -> lib
-rw-rw-r-- 1 tbeck tbeck   98 May  6 16:59 pyvenv.cfg
$ source bin/activate
/venv:dummy-py3.7/ $ pip install fetch-config
Looking in indexes: http://---.---.--.--:3141/pa-team/pa/
Collecting fetch-config
  Downloading http://---.---.--.--:3141/pa-team/pa/+f/7b6/2123e4cd6a015/fetch_config-0.1.3-py3-none-any.whl
Collecting py-buzz<0.4.0,>=0.3.7 (from fetch-config)
  Downloading http://---.---.--.--:3141/root/pypi/+f/4b7/a89bebad927c6/py_buzz-0.3.7-py3-none-any.whl
Installing collected packages: py-buzz, fetch-config
Successfully installed fetch-config-0.1.3 py-buzz-0.3.7
You are using pip version 18.1, however version 19.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
dusktreader commented 5 years ago

@sdispater After some debugging, I figured out the issue. It mostly on my end. The URL I used to specify my index is the same as I use for my pip configuration (pip.conf). However, it does not carry the /+simple path that poetry needed. Apparently, native pip adds that on automatically when it is contacting an index, but poetry needed it expicitly.

Once I ammended my pyproject.toml file accordingly, it worked great.

Thanks again for the great tool!

working pyproject.toml:


[tool.poetry]
name = "pa-mockup"
version = "0.1.0"
description = ""
authors = ["Tucker Beck <tucker.beck@---.com>"]

[tool.poetry.dependencies]
python = "^3.7"
pendulum = "^2.0"
flask-apispec = "^0.8.0"
flask = "^1.0"
flask-buzz = "^0.1.14"
flask-jwt-extended = "^3.18"
flask-cors = "^3.0"
flask-mail = "^0.9.1"
flask-redis = "^0.3.0"
fetch-config = "^0.1.3"

[tool.poetry.dev-dependencies]
pytest = "^3.0"
ipdb = "^0.12.0"

[[tool.poetry.source]]
name = "pa-devpi"
url = "http://---.---.--.--:3141/pa-team/pa/+simple"

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"
sdispater commented 5 years ago

@ojii Could you test your use case with the 1.0.0a3 which improves support for private repositories?

You can check out #908 to see what changed but basically you can declare repositories as secondary to make pypi.org preferred.

rmasters commented 5 years ago

@sdispater Not exactly the same issue; but I was having a similar error in a project that depends on a private github repo and 1.0.0a3 fixed that.

ojii commented 5 years ago

@ojii Could you test your use case with the 1.0.0a3 which improves support for private repositories?

@sdispater I'm sorry but we simply stopped using private packages as our "solution" to this so I no longer have an easy setup to test this. Looking at the PR though that looks like a good solution.

github-actions[bot] commented 5 months ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.