403 error when using private myget repository

voney commented 2 years ago

[ ] I am on the latest Poetry version.
[x] I am on the most recent pre-release version 1.2.0b1
[x] I have searched the issues of this repo and believe that this is not a duplicate.
[x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).
OS version and name: Fedora Workstation 35
Poetry version: 1.2.0b1
Link of a Gist with the contents of your pyproject.toml file: pyproject.toml

Issue

Myget uses basic authentication for the repository but hands back CDN download links that include the authentication as part of the URL.

When poetry attempts to download the file it's still using the same session it used to hit myget in the first place, which includes the basic auth headers. This causes the CDN to return a 403 error, similar to the below:

403 Client Error: Forbidden for url: https://az774065.vo.msecnd.net/redacted/redacted-2.0.6-py3-none-any.whl?sv=2015-12-11&sr=b&sig=5xCEJnyQXzUqK0ECzoABMH1vdVxPRCW0Yqzv7VqE8Yg%3D&st=2022-05-20T03%3A55%3A00Z&se=2022-05-20T05%3A05%3A00Z&sp=r&rscc=max-age%3D3600&rsct=binary%2Foctet-stream&rscd=filename%3Dredacted-2.0.6-py3-none-any.whl

The same repository works just fine with pip so it should be working in poetry.

I can think of two solutions to this:

Have poetry transparently catch the first 403 and re-try without the auth headers. This is hacky but works (I tried it locally). The downside being that it may cause undesired behaviour for other repos.
Add a flag to the repo in the pyproject.toml file that instructs poetry to not use the basic auth headers to download the packages. (I had a crack at this one, but the semantics of how poetry parses the pyproject.toml file is a bit beyond me)

abn commented 2 years ago

With recent changes this might have already been resolved. Can you test the version from master?

See these issues for more information: #5518

Install Poetry from VCS

pipx install --force --suffix=@git 'poetry @ git+https://github.com/python-poetry/poetry.git'

voney commented 2 years ago

Apologies for the delay in testing.

The git version still doesn't work, but in a new way.

(.venv) [daniel@dv-xps-fedora Davis_state_consumer]$ poetry@git add --source myget 'tymetrics'
Using version ^2.0.6 for redacted

Updating dependencies
Resolving dependencies... (4.2s)

  HTTPError

  404 Client Error: Not Found for url: https://pypi.org/pypi/redacted/json/

  at ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/requests/models.py:960 in raise_for_status
      956│         elif 500 <= self.status_code < 600:
      957│             http_error_msg = u'%s Server Error: %s for url: %s' % (self.status_code, reason, self.url)
      958│ 
      959│         if http_error_msg:
    → 960│             raise HTTPError(http_error_msg, response=self)
      961│ 
      962│     def close(self):
      963│         """Releases the connection back to the pool. Once this method has been
      964│         called the underlying ``raw`` object must not be accessed again.

That's after adding the myget repo using:

poetry@git source add myget https://www.myget.org/F/redacted/python/

abn commented 2 years ago

@voney looks like your source was searched, nothing found and then PyPI was looked at. You can try with -vvv for more information. To see if you actually got redirected to the correct package. It could also be that the credentials are not set correctly perhaps for your main repo?

voney commented 2 years ago

Ok so it looks like maybe the dependency solver isn't correctly getting additional packages from the private repo. Take a look.

(.venv) [daniel@dv-xps-fedora my_project]$ poetry@git add --source myget 'redacted' -vvv
Loading configuration file /home/daniel/.config/pypoetry/config.toml
Loading configuration file /home/daniel/.config/pypoetry/auth.toml
Adding repository myget (https://www.myget.org/F/myrepo/python)
Using virtualenv: /home/daniel/myrepo/my_project/.venv
Project environment contains an empty path in sys_path, ignoring.
[keyring.backend] Loading KWallet
[keyring.backend] Loading SecretService
[keyring.backend] Loading Windows
[keyring.backend] Loading chainer
[keyring.backend] Loading libsecret
[keyring.backend] Loading macOS
Creating new session for www.myget.org
[urllib3.connectionpool] Starting new HTTPS connection (1): www.myget.org:443
[urllib3.connectionpool] https://www.myget.org:443 "GET /F/myrepo/python/redacted/ HTTP/1.1" 200 1357
Source (myget): 11 packages found for redacted *
Using version ^2.0.6 for redacted

Updating dependencies
Resolving dependencies...
   1: fact: my-project is 0.1.0
   1: derived: my-project
   1: fact: my-project depends on redacted (^2.0.6)
   1: selecting my-project (0.1.0)
   1: derived: redacted (>=2.0.6,<3.0.0)
[urllib3.connectionpool] https://www.myget.org:443 "GET /F/myrepo/python/redacted/ HTTP/1.1" 200 1357
Source (myget): 1 packages found for redacted >=2.0.6,<3.0.0
   1: fact: redacted (2.0.6) depends on nose (*)
   1: fact: redacted (2.0.6) depends on azure-eventhub (>=5.0.0)
   1: fact: redacted (2.0.6) depends on redacted_dependency (*)
   1: selecting redacted (2.0.6)
   1: derived: redacted_dependency
   1: derived: azure-eventhub (>=5.0.0)
   1: derived: nose
[urllib3.connectionpool] https://www.myget.org:443 "GET /F/myrepo/python/redacted_dependency/ HTTP/1.1" 200 1030
Source (myget): 3 packages found for redacted_dependency *
Creating new session for pypi.org
[urllib3.connectionpool] Starting new HTTPS connection (1): pypi.org:443
[urllib3.connectionpool] https://pypi.org:443 "GET /pypi/redacted_dependency/json/ HTTP/1.1" 404 1980
   1: Version solving took 1.010 seconds.
   1: Tried 1 solutions.

  Stack trace:

  28  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/cleo/application.py:330 in run
       328│ 
       329│             try:
     → 330│                 exit_code = self._run(io)
       331│             except Exception as e:
       332│                 if not self._catch_exceptions:

  27  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/poetry/console/application.py:182 in _run
       180│         self._load_plugins(io)
       181│ 
     → 182│         exit_code: int = super()._run(io)
       183│         return exit_code
       184│ 

  26  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/cleo/application.py:425 in _run
       423│                 io.set_input(ArgvInput(argv))
       424│ 
     → 425│         exit_code = self._run_command(command, io)
       426│         self._running_command = None
       427│ 

  25  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/cleo/application.py:467 in _run_command
       465│ 
       466│         if error is not None:
     → 467│             raise error
       468│ 
       469│         return event.exit_code

  24  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/cleo/application.py:451 in _run_command
       449│ 
       450│             if event.command_should_run():
     → 451│                 exit_code = command.run(io)
       452│             else:
       453│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

  23  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/cleo/commands/base_command.py:118 in run
       116│         io.input.validate()
       117│ 
     → 118│         status_code = self.execute(io)
       119│ 
       120│         if status_code is None:

  22  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/cleo/commands/command.py:85 in execute
        83│ 
        84│         try:
     →  85│             return self.handle()
        86│         except KeyboardInterrupt:
        87│             return 1

  21  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/poetry/console/commands/add.py:256 in handle
       254│         self._installer.whitelist([cast(str, r["name"]) for r in requirements])
       255│ 
     → 256│         status = self._installer.run()
       257│ 
       258│         if status == 0 and not self.option("dry-run"):

  20  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/poetry/installation/installer.py:112 in run
       110│         local_repo = Repository()
       111│ 
     → 112│         return self._do_install(local_repo)
       113│ 
       114│     def dry_run(self, dry_run: bool = True) -> Installer:

  19  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/poetry/installation/installer.py:245 in _do_install
       243│                 source_root=self._env.path.joinpath("src")
       244│             ):
     → 245│                 ops = solver.solve(use_latest=self._whitelist).calculate_operations()
       246│         else:
       247│             self._io.write_line("Installing dependencies from lock file")

  18  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/poetry/puzzle/solver.py:80 in solve
        78│         with self._provider.progress():
        79│             start = time.time()
     →  80│             packages, depths = self._solve(use_latest=use_latest)
        81│             end = time.time()
        82│ 

  17  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/poetry/puzzle/solver.py:149 in _solve
       147│ 
       148│         try:
     → 149│             result = resolve_version(
       150│                 self._package, self._provider, locked=locked, use_latest=use_latest
       151│             )

  16  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/poetry/mixology/__init__.py:24 in resolve_version
        22│     solver = VersionSolver(root, provider, locked=locked, use_latest=use_latest)
        23│ 
     →  24│     return solver.solve()
        25│ 

  15  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/poetry/mixology/version_solver.py:125 in solve
       123│             while next is not None:
       124│                 self._propagate(next)
     → 125│                 next = self._choose_package_version()
       126│ 
       127│             return self._result()

  14  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/poetry/mixology/version_solver.py:409 in _choose_package_version
       407│             dependency = unsatisfied[0]
       408│         else:
     → 409│             dependency = min(*unsatisfied, key=_get_min)
       410│ 
       411│         locked = self._get_locked(dependency)

  13  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/poetry/mixology/version_solver.py:401 in _get_min
       399│                 return (
       400│                     not dependency.marker.is_any(),
     → 401│                     len(self._dependency_cache.search_for(dependency)),
       402│                 )
       403│             except ValueError:

  12  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/poetry/mixology/version_solver.py:62 in _search_for
        60│         packages = self.cache.get(key)
        61│         if packages is None:
     →  62│             packages = self.provider.search_for(dependency)
        63│         else:
        64│             packages = [p for p in packages if dependency.constraint.allows(p.version)]

  11  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/poetry/puzzle/provider.py:220 in search_for
       218│             packages = self.search_for_url(dependency)
       219│         else:
     → 220│             packages = self._pool.find_packages(dependency)
       221│ 
       222│             packages.sort(

  10  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/poetry/repositories/pool.py:169 in find_packages
       167│         packages = []
       168│         for repo in self._repositories:
     → 169│             packages += repo.find_packages(dependency)
       170│ 
       171│         return packages

   9  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/poetry/repositories/pypi_repository.py:54 in find_packages
        52│ 
        53│         try:
     →  54│             info = self.get_package_info(dependency.name)
        55│         except PackageNotFound:
        56│             self._log(

   8  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/poetry/repositories/pypi_repository.py:155 in get_package_info
       153│             return self._get_package_info(name)
       154│ 
     → 155│         package_info: dict[str, Any] = self._cache.store("packages").remember_forever(
       156│             name, lambda: self._get_package_info(name)
       157│         )

   7  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/cachy/repository.py:174 in remember_forever
       172│             return val
       173│ 
     → 174│         val = value(callback)
       175│ 
       176│         self.forever(key, val)

   6  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/cachy/helpers.py:6 in value
         4│ def value(val):
         5│     if callable(val):
     →   6│         return val()
         7│ 
         8│     return val

   5  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/poetry/repositories/pypi_repository.py:156 in <lambda>
       154│ 
       155│         package_info: dict[str, Any] = self._cache.store("packages").remember_forever(
     → 156│             name, lambda: self._get_package_info(name)
       157│         )
       158│         return package_info

   4  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/poetry/repositories/pypi_repository.py:161 in _get_package_info
       159│ 
       160│     def _get_package_info(self, name: str) -> dict[str, Any]:
     → 161│         data = self._get(f"pypi/{name}/json")
       162│         if data is None:
       163│             raise PackageNotFound(f"Package [{name}] not found.")

   3  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/poetry/repositories/pypi_repository.py:246 in _get
       244│     def _get(self, endpoint: str) -> dict[str, Any] | None:
       245│         try:
     → 246│             json_response = self.session.get(self._base_url + endpoint)
       247│         except requests.exceptions.TooManyRedirects:
       248│             # Cache control redirect loop.

   2  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/poetry/utils/authenticator.py:235 in get
       233│ 
       234│     def get(self, url: str, **kwargs: Any) -> requests.Response:
     → 235│         return self.request("get", url, **kwargs)
       236│ 
       237│     def post(self, url: str, **kwargs: Any) -> requests.Response:

   1  ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/poetry/utils/authenticator.py:221 in request
       219│                 if resp.status_code not in [502, 503, 504] or is_last_attempt:
       220│                     if raise_for_status:
     → 221│                         resp.raise_for_status()
       222│                     return resp
       223│ 

  HTTPError

  404 Client Error: Not Found for url: https://pypi.org/pypi/redacted_dependency/json/

  at ~/.local/pipx/venvs/poetry@git/lib64/python3.10/site-packages/requests/models.py:960 in raise_for_status
      956│         elif 500 <= self.status_code < 600:
      957│             http_error_msg = u'%s Server Error: %s for url: %s' % (self.status_code, reason, self.url)
      958│ 
      959│         if http_error_msg:
    → 960│             raise HTTPError(http_error_msg, response=self)
      961│ 
      962│     def close(self):
      963│         """Releases the connection back to the pool. Once this method has been
      964│         called the underlying ``raw`` object must not be accessed again.

For context here, I'm installing redacted which repends on redacted_dependency which is also in the myget repo.

abn commented 2 years ago

Conversation moved to https://github.com/python-poetry/poetry/issues/5682.

github-actions[bot] commented 8 months ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

python-poetry / poetry

403 error when using private myget repository #5651

Issue

Install Poetry from VCS