search

python-poetry / poetry

Python packaging and dependency management made easy
https://python-poetry.org
MIT License
31.08k stars 2.26k forks source link

Dulwich does not support Dumb Git transport #6999

Closed rruiter87 closed 3 months ago

rruiter87 commented 1 year ago

Issue

I'm unable to add git dependencies both via https and ssh. If I just do a regular git clone <repo> it works for ssh.

HTTPS

❯ poetry add git+https://github.com:mbuesch/pyprofibus.git -vvv
Using virtualenv: C:\Users\rruiter\Miniconda3\envs\profinet310
[keyring.backend] Loading KWallet
[keyring.backend] Loading SecretService
[keyring.backend] Loading Windows
[keyring.backend] Loading chainer
[keyring.backend] Loading libsecret
[keyring.backend] Loading macOS
[urllib3.connectionpool] Starting new HTTPS connection (1): github.com:443
[urllib3.connectionpool] https://github.com:443 "GET /mbuesch/pyprofibus.git/info/refs?service=git-upload-pack HTTP/1.1" 200 None
Cloning https://github.com/mbuesch/pyprofibus.git at 'HEAD' to C:\Users\rruiter\Miniconda3\envs\profinet310\src\pyprofibus
[urllib3.connectionpool] Starting new HTTPS connection (1): git.bues.ch:443
[urllib3.connectionpool] https://git.bues.ch:443 "GET /git/crcgen.git/info/refs?service=git-upload-pack HTTP/1.1" 200 817

  Stack trace:

  23  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cleo\application.py:329 in run
       327│
       328│             try:
     → 329│                 exit_code = self._run(io)
       330│             except Exception as e:
       331│                 if not self._catch_exceptions:

  22  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\console\application.py:185 in _run
       183│         self._load_plugins(io)
       184│
     → 185│         exit_code: int = super()._run(io)
       186│         return exit_code
       187│

  21  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cleo\application.py:423 in _run
       421│             io.input.set_stream(stream)
       422│
     → 423│         exit_code = self._run_command(command, io)
       424│         self._running_command = None
       425│

  20  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cleo\application.py:465 in _run_command
       463│
       464│         if error is not None:
     → 465│             raise error
       466│
       467│         return event.exit_code

  19  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cleo\application.py:449 in _run_command
       447│
       448│             if event.command_should_run():
     → 449│                 exit_code = command.run(io)
       450│             else:
       451│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

  18  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cleo\commands\base_command.py:119 in run
       117│         io.input.validate()
       118│
     → 119│         status_code = self.execute(io)
       120│
       121│         if status_code is None:

  17  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cleo\commands\command.py:83 in execute
        81│
        82│         try:
     →  83│             return self.handle()
        84│         except KeyboardInterrupt:
        85│             return 1

  16  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\console\commands\add.py:158 in handle
       156│             return 0
       157│
     → 158│         requirements = self._determine_requirements(
       159│             packages,
       160│             allow_prereleases=self.option("allow-prereleases"),

  15  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\console\commands\init.py:357 in _determine_requirements
       355│
       356│         result = []
     → 357│         for requirement in self._parse_requirements(requires):
       358│             if "git" in requirement or "url" in requirement or "path" in requirement:
       359│                 result.append(requirement)

  14  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\console\commands\init.py:416 in _parse_requirements
       414│             cwd = Path.cwd()
       415│
     → 416│         return [
       417│             parse_dependency_specification(
       418│                 requirement=requirement,

  13  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\console\commands\init.py:417 in <listcomp>
       415│
       416│         return [
     → 417│             parse_dependency_specification(
       418│                 requirement=requirement,
       419│                 env=self.env if isinstance(self, EnvCommand) else None,

  12  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\utils\dependency_specification.py:216 in parse_dependency_specification
       214│
       215│     specification = (
     → 216│         _parse_dependency_specification_url(requirement, env=env)
       217│         or _parse_dependency_specification_path(requirement, cwd=cwd)
       218│         or _parse_dependency_specification_simple(requirement)

  11  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\utils\dependency_specification.py:68 in _parse_dependency_specification_url
        66│
        67│     if url_parsed.scheme in ["git+https", "git+ssh"]:
     →  68│         return _parse_dependency_specification_git_url(requirement, env)
        69│
        70│     if url_parsed.scheme in ["http", "https"]:

  10  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\utils\dependency_specification.py:49 in _parse_dependency_specification_git_url
        47│
        48│     source_root = env.path.joinpath("src") if env else None
     →  49│     package = Provider.get_package_from_vcs(
        50│         "git",
        51│         url=url.url,

   9  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\puzzle\provider.py:341 in get_package_from_vcs
       339│             raise ValueError(f"Unsupported VCS dependency {vcs}")
       340│
     → 341│         return _get_package_from_git(
       342│             url=url,
       343│             branch=branch,

   8  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\puzzle\provider.py:95 in _get_package_from_git
        93│     source_root: Path | None = None,
        94│ ) -> Package:
     →  95│     source = Git.clone(
        96│         url=url,
        97│         source_root=source_root,

   7  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\vcs\git\backend.py:427 in clone
       425│             if not cls.is_using_legacy_client():
       426│                 local = cls._clone(url=url, refspec=refspec, target=target)
     → 427│                 cls._clone_submodules(repo=local)
       428│                 return local
       429│         except HTTPUnauthorized:

   6  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\vcs\git\backend.py:356 in _clone_submodules
       354│                         continue
       355│
     → 356│                 cls.clone(
       357│                     url=url.decode("utf-8"),
       358│                     source_root=source_root,

   5  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\vcs\git\backend.py:426 in clone
       424│         try:
       425│             if not cls.is_using_legacy_client():
     → 426│                 local = cls._clone(url=url, refspec=refspec, target=target)
       427│                 cls._clone_submodules(repo=local)
       428│                 return local

   4  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\vcs\git\backend.py:258 in _clone
       256│             local = Repo(str(target))
       257│
     → 258│         remote_refs = cls._fetch_remote_refs(url=url, local=local)
       259│
       260│         logger.debug(

   3  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\vcs\git\backend.py:201 in _fetch_remote_refs
       199│
       200│         with local:
     → 201│             result: FetchPackResult = client.fetch(
       202│                 path,
       203│                 local,

   2  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\dulwich\client.py:837 in fetch
        835│             f, commit, abort = target.object_store.add_pack()
        836│         try:
     →  837│             result = self.fetch_pack(
        838│                 path,
        839│                 determine_wants,

   1  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\dulwich\client.py:2076 in fetch_pack
       2074│         """
       2075│         url = self._get_url(path)
     → 2076│         refs, server_capabilities, url = self._discover_references(
       2077│             b"git-upload-pack", url
       2078│         )

  AttributeError

  'NoneType' object has no attribute 'startswith'

  at ~\AppData\Roaming\pypoetry\venv\lib\site-packages\dulwich\client.py:1946 in _discover_references
      1942│                 )
      1943│             base_url = resp.redirect_location[: -len(tail)]
      1944│
      1945│         try:
    → 1946│             self.dumb = not resp.content_type.startswith("application/x-git-")
      1947│             if not self.dumb:
      1948│                 proto = Protocol(read, None)
      1949│                 # The first line should mention the service
      1950│                 try:

SSH

❯ poetry add git+ssh://github.com:mbuesch/pyprofibus.git -vvv
Using virtualenv: C:\Users\rruiter\Miniconda3\envs\profinet310

  Stack trace:

  3  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\dulwich\client.py:1149 in fetch_pack
      1147│         with proto:
      1148│             try:
    → 1149│                 refs, server_capabilities = read_pkt_refs(proto.read_pkt_seq())
      1150│             except HangupException as exc:
      1151│                 raise _remote_error_from_stderr(stderr) from exc

  2  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\dulwich\client.py:253 in read_pkt_refs
       251│     refs = {}
       252│     # Receive refs from server
    →  253│     for pkt in pkt_seq:
       254│         (sha, ref) = pkt.rstrip(b"\n").split(None, 1)
       255│         if sha == b"ERR":

  1  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\dulwich\protocol.py:277 in read_pkt_seq
      275│             flush-pkt.
      276│         """
    → 277│         pkt = self.read_pkt_line()
      278│         while pkt:
      279│             yield pkt

  HangupException

  The remote server unexpectedly closed the connection.

  at ~\AppData\Roaming\pypoetry\venv\lib\site-packages\dulwich\protocol.py:220 in read_pkt_line
      216│
      217│         try:
      218│             sizestr = read(4)
      219│             if not sizestr:
    → 220│                 raise HangupException()
      221│             size = int(sizestr, 16)
      222│             if size == 0:
      223│                 if self.report_activity:
      224│                     self.report_activity(4, "read")

The following error occurred when trying to handle this error:

  Stack trace:

  20  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cleo\application.py:329 in run
       327│
       328│             try:
     → 329│                 exit_code = self._run(io)
       330│             except Exception as e:
       331│                 if not self._catch_exceptions:

  19  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\console\application.py:185 in _run
       183│         self._load_plugins(io)
       184│
     → 185│         exit_code: int = super()._run(io)
       186│         return exit_code
       187│

  18  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cleo\application.py:423 in _run
       421│             io.input.set_stream(stream)
       422│
     → 423│         exit_code = self._run_command(command, io)
       424│         self._running_command = None
       425│

  17  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cleo\application.py:465 in _run_command
       463│
       464│         if error is not None:
     → 465│             raise error
       466│
       467│         return event.exit_code

  16  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cleo\application.py:449 in _run_command
       447│
       448│             if event.command_should_run():
     → 449│                 exit_code = command.run(io)
       450│             else:
       451│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

  15  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cleo\commands\base_command.py:119 in run
       117│         io.input.validate()
       118│
     → 119│         status_code = self.execute(io)
       120│
       121│         if status_code is None:

  14  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cleo\commands\command.py:83 in execute
        81│
        82│         try:
     →  83│             return self.handle()
        84│         except KeyboardInterrupt:
        85│             return 1

  13  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\console\commands\add.py:158 in handle
       156│             return 0
       157│
     → 158│         requirements = self._determine_requirements(
       159│             packages,
       160│             allow_prereleases=self.option("allow-prereleases"),

  12  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\console\commands\init.py:357 in _determine_requirements
       355│
       356│         result = []
     → 357│         for requirement in self._parse_requirements(requires):
       358│             if "git" in requirement or "url" in requirement or "path" in requirement:
       359│                 result.append(requirement)

  11  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\console\commands\init.py:416 in _parse_requirements
       414│             cwd = Path.cwd()
       415│
     → 416│         return [
       417│             parse_dependency_specification(
       418│                 requirement=requirement,

  10  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\console\commands\init.py:417 in <listcomp>
       415│
       416│         return [
     → 417│             parse_dependency_specification(
       418│                 requirement=requirement,
       419│                 env=self.env if isinstance(self, EnvCommand) else None,

   9  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\utils\dependency_specification.py:216 in parse_dependency_specification
       214│
       215│     specification = (
     → 216│         _parse_dependency_specification_url(requirement, env=env)
       217│         or _parse_dependency_specification_path(requirement, cwd=cwd)
       218│         or _parse_dependency_specification_simple(requirement)

   8  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\utils\dependency_specification.py:68 in _parse_dependency_specification_url
        66│
        67│     if url_parsed.scheme in ["git+https", "git+ssh"]:
     →  68│         return _parse_dependency_specification_git_url(requirement, env)
        69│
        70│     if url_parsed.scheme in ["http", "https"]:

   7  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\utils\dependency_specification.py:49 in _parse_dependency_specification_git_url
        47│
        48│     source_root = env.path.joinpath("src") if env else None
     →  49│     package = Provider.get_package_from_vcs(
        50│         "git",
        51│         url=url.url,

   6  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\puzzle\provider.py:341 in get_package_from_vcs
       339│             raise ValueError(f"Unsupported VCS dependency {vcs}")
       340│
     → 341│         return _get_package_from_git(
       342│             url=url,
       343│             branch=branch,

   5  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\puzzle\provider.py:95 in _get_package_from_git
        93│     source_root: Path | None = None,
        94│ ) -> Package:
     →  95│     source = Git.clone(
        96│         url=url,
        97│         source_root=source_root,

   4  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\vcs\git\backend.py:426 in clone
       424│         try:
       425│             if not cls.is_using_legacy_client():
     → 426│                 local = cls._clone(url=url, refspec=refspec, target=target)
       427│                 cls._clone_submodules(repo=local)
       428│                 return local

   3  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\vcs\git\backend.py:258 in _clone
       256│             local = Repo(str(target))
       257│
     → 258│         remote_refs = cls._fetch_remote_refs(url=url, local=local)
       259│
       260│         logger.debug(

   2  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\vcs\git\backend.py:201 in _fetch_remote_refs
       199│
       200│         with local:
     → 201│             result: FetchPackResult = client.fetch(
       202│                 path,
       203│                 local,

   1  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\dulwich\client.py:837 in fetch
        835│             f, commit, abort = target.object_store.add_pack()
        836│         try:
     →  837│             result = self.fetch_pack(
        838│                 path,
        839│                 determine_wants,

  HangupException

  d\\rruiter@github.com: Permission denied (publickey).

  at ~\AppData\Roaming\pypoetry\venv\lib\site-packages\dulwich\client.py:1151 in fetch_pack
      1147│         with proto:
      1148│             try:
      1149│                 refs, server_capabilities = read_pkt_refs(proto.read_pkt_seq())
      1150│             except HangupException as exc:
    → 1151│                 raise _remote_error_from_stderr(stderr) from exc
      1152│             (
      1153│                 negotiated_capabilities,
      1154│                 symrefs,
      1155│                 agent,
finswimmer commented 1 year ago

Hey @rruiter87,

does it work if you set experimental.system-git-client to true (https://python-poetry.org/docs/configuration/#experimentalsystem-git-client)?

fin swimmer

rruiter87 commented 1 year ago

SSH no, HTTPS yes

I set the option with 

 ❯ poetry config experimental.system-git-client true

SSH 🔴

❯ poetry add git+ssh://github.com:mbuesch/pyprofibus.git -vvv
Loading configuration file C:\Users\rruiter\AppData\Roaming\pypoetry\config.toml
Using virtualenv: C:\Users\rruiter\Miniconda3\envs\profinet310
Cloning 'github.com:mbuesch/pyprofibus.git' using system git client

Failed to clone github.com:mbuesch/pyprofibus.git, check your git configuration and permissions for this repository.

HTTPS 🟢

❯ poetry add git+https://github.com:mbuesch/pyprofibus.git -vvv
Loading configuration file C:\Users\rruiter\AppData\Roaming\pypoetry\config.toml
Using virtualenv: C:\Users\rruiter\Miniconda3\envs\profinet310
Cloning 'https://github.com/mbuesch/pyprofibus.git' using system git client

Updating dependencies
Resolving dependencies...
   1: fact: profinettest is 0.1.0
   1: derived: profinettest
   1: fact: profinettest depends on pyprofibus (1.11)
   1: selecting profinettest (0.1.0)
   1: derived: pyprofibus (1.11) @ git+https://github.com/mbuesch/pyprofibus.git
   1: selecting pyprofibus (1.11 78f06da)
   1: Version solving took 0.004 seconds.
   1: Tried 1 solutions.

Writing lock file

Finding the necessary packages for the current system

Package operations: 1 install, 0 updates, 0 removals

  • Installing pyprofibus (1.11 78f06da)
neersighted commented 1 year ago

It looks like something to do with your network and/or remote Git service; this will be very hard to debug for anyone but you as you have not provided a repo to test against.

cc @jelmer

rruiter87 commented 1 year ago

Hi @neersighted. The repo was local only and still very new. I only did:

> conda create --name profinet310 python=3.10
> conda env export > conda.yml
> poetry init
> git init; git add . ; git commit -m "initial commit"
> poetry add git+ssh://github.com:mbuesch/pyprofibus.git

For completeness

> git --version
git version 2.33.0.windows.2
> conda --version
conda 4.13.0

Might be the university network though. It is not the first time it causes strange issues :p.

neersighted commented 1 year ago

https://git.bues.ch/git/crcgen.git/ is what stood out to me specifically, however, I did not realize it was a public repo; my bad. It looks like this is a submodule using the 'dumb' HTTP transport -- I'm honestly not sure if Dulwich supports dumb HTTP as it's generally considered deprecated, but it helps that everything is publicly visible.

dimbleby commented 1 year ago

the error looks exactly as though it's in the dulwich code that's trying to handle the dumb transport:

AttributeError

  'NoneType' object has no attribute 'startswith'

  at ~\AppData\Roaming\pypoetry\venv\lib\site-packages\dulwich\client.py:1946 in _discover_references
      1942│                 )
      1943│             base_url = resp.redirect_location[: -len(tail)]
      1944│
      1945│         try:
    → 1946│             self.dumb = not resp.content_type.startswith("application/x-git-")
      1947│             if not self.dumb:
      1948│                 proto = Protocol(read, None)
      1949│                 # The first line should mention the service
      1950│                 try:
neersighted commented 1 year ago

Looks like a poorly configured HTTP server that's not returning a content type at all, honestly:

$ curl --head https://git.bues.ch:443/git/crcgen.git/info/refs?service=git-upload-pack
HTTP/1.1 200 OK
Date: Thu, 10 Nov 2022 15:24:40 GMT
Server: Apache/2.4.54
Last-Modified: Fri, 02 Sep 2022 20:08:07 GMT
ETag: "331-5e7b749dcff07"
Accept-Ranges: bytes
Content-Length: 817

As Content-Type is not mandated by HTTP, Dulwich should be handling this gracefully.

As to the system Git client failing over SSH, that smells more like a configuration issue to me, as the invocation is brutally simple:

https://github.com/python-poetry/poetry/blob/def1ee8f3ae00c307ca028da53d2347615c5c32b/src/poetry/vcs/git/system.py#L21

e.g.

$ git clone --recurse-submodules -- git+ssh://github.com:mbuesch/pyprofibus.git C:\Users\rruiter\Miniconda3\envs\profinet310\src\pyprofibus

Could you give it a go manually, @rruiter87?

rruiter87 commented 1 year ago 
❯ git clone --recurse-submodules -- git+ssh://github.com:mbuesch/pyprofibus.git C:\Users\rruiter\Downloads\empty
Cloning into 'C:\Users\rruiter\Downloads\empty'...
ssh: Could not resolve hostname github.com:mbuesch: Name or service not known
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

But this works 

> git clone git@github.com:mbuesch/pyprofibus.git C:\Users\rruiter\Downloads\emptry1
Cloning into 'C:\Users\rruiter\Downloads\emptry1'...
remote: Enumerating objects: 2104, done.
remote: Counting objects: 100% (291/291), done.
remote: Compressing objects: 100% (109/109), done.
remote: Total 2104 (delta 181), reused 291 (delta 181), pack-reused 1813
Receiving objects: 100% (2104/2104), 677.44 KiB | 17.37 MiB/s, done.
Resolving deltas: 100% (1381/1381), done.
neersighted commented 1 year ago

Ah, I see a critical typo everyone missed:

github.com:mbuesch

In all cases, a / should be used instead of : -- : is only used in the SCP/SFTP path notation that Poetry does not use.

Can you try again with a well-formed URL?

rruiter87 commented 1 year ago

SSH failed, HTTPS worked.

❯ poetry add git+ssh://github.com/mbuesch/pyprofibus.git -vvv
Loading configuration file C:\Users\rruiter\AppData\Roaming\pypoetry\config.toml
Using virtualenv: C:\Users\rruiter\Miniconda3\envs\profinet310
Cloning 'ssh://github.com/mbuesch/pyprofibus.git' using system git client

Failed to clone ssh://github.com/mbuesch/pyprofibus.git, check your git configuration and permissions for this repository.
❯ poetry add git+https://github.com/mbuesch/pyprofibus.git -vvv
Loading configuration file C:\Users\rruiter\AppData\Roaming\pypoetry\config.toml
Using virtualenv: C:\Users\rruiter\Miniconda3\envs\profinet310
Cloning 'https://github.com/mbuesch/pyprofibus.git' using system git client
...
neersighted commented 1 year ago

And the example Git clone based on Poetry's invocation?

jelmer commented 1 year ago

Those two commands say "system git" though, implying it's not related to Dulwich?

neersighted commented 1 year ago

Those two commands say "system git" though, implying it's not related to Dulwich?

I was hoping you could look at the unhandled None Content-Type in the issue body; that being said, attempts with the unmangled URL and Dulwich would also be appreciated @rruiter87.

jelmer commented 1 year ago

The None Content-Type is an error handling bug; I agree Dulwich should raise a different exception, but it's not the root cause of the issues @rruiter87 is hitting

rruiter87 commented 1 year ago

Reactivate Dulwich

❯ poetry config experimental.system-git-client false

add with ssh

❯ poetry add git+ssh://github.com/mbuesch/pyprofibus.git -vvv
Loading configuration file C:\Users\rruiter\AppData\Roaming\pypoetry\config.toml
Using virtualenv: C:\Users\rruiter\Miniconda3\envs\profinet310

  Stack trace:

  3  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\dulwich\client.py:1149 in fetch_pack
      1147│         with proto:
      1148│             try:
    → 1149│                 refs, server_capabilities = read_pkt_refs(proto.read_pkt_seq())
      1150│             except HangupException as exc:
      1151│                 raise _remote_error_from_stderr(stderr) from exc

  2  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\dulwich\client.py:253 in read_pkt_refs
       251│     refs = {}
       252│     # Receive refs from server
    →  253│     for pkt in pkt_seq:
       254│         (sha, ref) = pkt.rstrip(b"\n").split(None, 1)
       255│         if sha == b"ERR":

  1  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\dulwich\protocol.py:277 in read_pkt_seq
      275│             flush-pkt.
      276│         """
    → 277│         pkt = self.read_pkt_line()
      278│         while pkt:
      279│             yield pkt

  HangupException

  The remote server unexpectedly closed the connection.

  at ~\AppData\Roaming\pypoetry\venv\lib\site-packages\dulwich\protocol.py:220 in read_pkt_line
      216│
      217│         try:
      218│             sizestr = read(4)
      219│             if not sizestr:
    → 220│                 raise HangupException()
      221│             size = int(sizestr, 16)
      222│             if size == 0:
      223│                 if self.report_activity:
      224│                     self.report_activity(4, "read")

The following error occurred when trying to handle this error:

  Stack trace:

  20  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cleo\application.py:329 in run
       327│
       328│             try:
     → 329│                 exit_code = self._run(io)
       330│             except Exception as e:
       331│                 if not self._catch_exceptions:

  19  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\console\application.py:185 in _run
       183│         self._load_plugins(io)
       184│
     → 185│         exit_code: int = super()._run(io)
       186│         return exit_code
       187│

  18  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cleo\application.py:423 in _run
       421│             io.input.set_stream(stream)
       422│
     → 423│         exit_code = self._run_command(command, io)
       424│         self._running_command = None
       425│

  17  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cleo\application.py:465 in _run_command
       463│
       464│         if error is not None:
     → 465│             raise error
       466│
       467│         return event.exit_code

  16  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cleo\application.py:449 in _run_command
       447│
       448│             if event.command_should_run():
     → 449│                 exit_code = command.run(io)
       450│             else:
       451│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

  15  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cleo\commands\base_command.py:119 in run
       117│         io.input.validate()
       118│
     → 119│         status_code = self.execute(io)
       120│
       121│         if status_code is None:

  14  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cleo\commands\command.py:83 in execute
        81│
        82│         try:
     →  83│             return self.handle()
        84│         except KeyboardInterrupt:
        85│             return 1

  13  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\console\commands\add.py:158 in handle
       156│             return 0
       157│
     → 158│         requirements = self._determine_requirements(
       159│             packages,
       160│             allow_prereleases=self.option("allow-prereleases"),

  12  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\console\commands\init.py:357 in _determine_requirements
       355│
       356│         result = []
     → 357│         for requirement in self._parse_requirements(requires):
       358│             if "git" in requirement or "url" in requirement or "path" in requirement:
       359│                 result.append(requirement)

  11  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\console\commands\init.py:416 in _parse_requirements
       414│             cwd = Path.cwd()
       415│
     → 416│         return [
       417│             parse_dependency_specification(
       418│                 requirement=requirement,

  10  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\console\commands\init.py:417 in <listcomp>
       415│
       416│         return [
     → 417│             parse_dependency_specification(
       418│                 requirement=requirement,
       419│                 env=self.env if isinstance(self, EnvCommand) else None,

   9  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\utils\dependency_specification.py:216 in parse_dependency_specification
       214│
       215│     specification = (
     → 216│         _parse_dependency_specification_url(requirement, env=env)
       217│         or _parse_dependency_specification_path(requirement, cwd=cwd)
       218│         or _parse_dependency_specification_simple(requirement)

   8  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\utils\dependency_specification.py:68 in _parse_dependency_specification_url
        66│
        67│     if url_parsed.scheme in ["git+https", "git+ssh"]:
     →  68│         return _parse_dependency_specification_git_url(requirement, env)
        69│
        70│     if url_parsed.scheme in ["http", "https"]:

   7  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\utils\dependency_specification.py:49 in _parse_dependency_specification_git_url
        47│
        48│     source_root = env.path.joinpath("src") if env else None
     →  49│     package = Provider.get_package_from_vcs(
        50│         "git",
        51│         url=url.url,

   6  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\puzzle\provider.py:341 in get_package_from_vcs
       339│             raise ValueError(f"Unsupported VCS dependency {vcs}")
       340│
     → 341│         return _get_package_from_git(
       342│             url=url,
       343│             branch=branch,

   5  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\puzzle\provider.py:95 in _get_package_from_git
        93│     source_root: Path | None = None,
        94│ ) -> Package:
     →  95│     source = Git.clone(
        96│         url=url,
        97│         source_root=source_root,

   4  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\vcs\git\backend.py:426 in clone
       424│         try:
       425│             if not cls.is_using_legacy_client():
     → 426│                 local = cls._clone(url=url, refspec=refspec, target=target)
       427│                 cls._clone_submodules(repo=local)
       428│                 return local

   3  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\vcs\git\backend.py:258 in _clone
       256│             local = Repo(str(target))
       257│
     → 258│         remote_refs = cls._fetch_remote_refs(url=url, local=local)
       259│
       260│         logger.debug(

   2  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\vcs\git\backend.py:201 in _fetch_remote_refs
       199│
       200│         with local:
     → 201│             result: FetchPackResult = client.fetch(
       202│                 path,
       203│                 local,

   1  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\dulwich\client.py:837 in fetch
        835│             f, commit, abort = target.object_store.add_pack()
        836│         try:
     →  837│             result = self.fetch_pack(
        838│                 path,
        839│                 determine_wants,

  HangupException

  d\\rruiter@github.com: Permission denied (publickey).

  at ~\AppData\Roaming\pypoetry\venv\lib\site-packages\dulwich\client.py:1151 in fetch_pack
      1147│         with proto:
      1148│             try:
      1149│                 refs, server_capabilities = read_pkt_refs(proto.read_pkt_seq())
      1150│             except HangupException as exc:
    → 1151│                 raise _remote_error_from_stderr(stderr) from exc
      1152│             (
      1153│                 negotiated_capabilities,
      1154│                 symrefs,
      1155│                 agent,

add with https

❯ poetry add git+https://github.com/mbuesch/pyprofibus.git -vvv
Loading configuration file C:\Users\rruiter\AppData\Roaming\pypoetry\config.toml
Using virtualenv: C:\Users\rruiter\Miniconda3\envs\profinet310
[keyring.backend] Loading KWallet
[keyring.backend] Loading SecretService
[keyring.backend] Loading Windows
[keyring.backend] Loading chainer
[keyring.backend] Loading libsecret
[keyring.backend] Loading macOS
[urllib3.connectionpool] Starting new HTTPS connection (1): github.com:443
[urllib3.connectionpool] https://github.com:443 "GET /mbuesch/pyprofibus.git/info/refs?service=git-upload-pack HTTP/1.1" 200 None
[urllib3.connectionpool] Starting new HTTPS connection (2): github.com:443
[urllib3.connectionpool] https://github.com:443 "POST /mbuesch/pyprofibus.git/git-upload-pack HTTP/1.1" 200 None
Cloning https://github.com/mbuesch/pyprofibus.git at 'HEAD' to C:\Users\rruiter\Miniconda3\envs\profinet310\src\pyprofibus
[urllib3.connectionpool] Starting new HTTPS connection (1): git.bues.ch:443
[urllib3.connectionpool] https://git.bues.ch:443 "GET /git/crcgen.git/info/refs?service=git-upload-pack HTTP/1.1" 200 817

  Stack trace:

  23  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cleo\application.py:329 in run
       327│
       328│             try:
     → 329│                 exit_code = self._run(io)
       330│             except Exception as e:
       331│                 if not self._catch_exceptions:

  22  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\console\application.py:185 in _run
       183│         self._load_plugins(io)
       184│
     → 185│         exit_code: int = super()._run(io)
       186│         return exit_code
       187│

  21  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cleo\application.py:423 in _run
       421│             io.input.set_stream(stream)
       422│
     → 423│         exit_code = self._run_command(command, io)
       424│         self._running_command = None
       425│

  20  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cleo\application.py:465 in _run_command
       463│
       464│         if error is not None:
     → 465│             raise error
       466│
       467│         return event.exit_code

  19  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cleo\application.py:449 in _run_command
       447│
       448│             if event.command_should_run():
     → 449│                 exit_code = command.run(io)
       450│             else:
       451│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

  18  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cleo\commands\base_command.py:119 in run
       117│         io.input.validate()
       118│
     → 119│         status_code = self.execute(io)
       120│
       121│         if status_code is None:

  17  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\cleo\commands\command.py:83 in execute
        81│
        82│         try:
     →  83│             return self.handle()
        84│         except KeyboardInterrupt:
        85│             return 1

  16  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\console\commands\add.py:158 in handle
       156│             return 0
       157│
     → 158│         requirements = self._determine_requirements(
       159│             packages,
       160│             allow_prereleases=self.option("allow-prereleases"),

  15  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\console\commands\init.py:357 in _determine_requirements
       355│
       356│         result = []
     → 357│         for requirement in self._parse_requirements(requires):
       358│             if "git" in requirement or "url" in requirement or "path" in requirement:
       359│                 result.append(requirement)

  14  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\console\commands\init.py:416 in _parse_requirements
       414│             cwd = Path.cwd()
       415│
     → 416│         return [
       417│             parse_dependency_specification(
       418│                 requirement=requirement,

  13  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\console\commands\init.py:417 in <listcomp>
       415│
       416│         return [
     → 417│             parse_dependency_specification(
       418│                 requirement=requirement,
       419│                 env=self.env if isinstance(self, EnvCommand) else None,

  12  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\utils\dependency_specification.py:216 in parse_dependency_specification
       214│
       215│     specification = (
     → 216│         _parse_dependency_specification_url(requirement, env=env)
       217│         or _parse_dependency_specification_path(requirement, cwd=cwd)
       218│         or _parse_dependency_specification_simple(requirement)

  11  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\utils\dependency_specification.py:68 in _parse_dependency_specification_url
        66│
        67│     if url_parsed.scheme in ["git+https", "git+ssh"]:
     →  68│         return _parse_dependency_specification_git_url(requirement, env)
        69│
        70│     if url_parsed.scheme in ["http", "https"]:

  10  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\utils\dependency_specification.py:49 in _parse_dependency_specification_git_url
        47│
        48│     source_root = env.path.joinpath("src") if env else None
     →  49│     package = Provider.get_package_from_vcs(
        50│         "git",
        51│         url=url.url,

   9  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\puzzle\provider.py:341 in get_package_from_vcs
       339│             raise ValueError(f"Unsupported VCS dependency {vcs}")
       340│
     → 341│         return _get_package_from_git(
       342│             url=url,
       343│             branch=branch,

   8  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\puzzle\provider.py:95 in _get_package_from_git
        93│     source_root: Path | None = None,
        94│ ) -> Package:
     →  95│     source = Git.clone(
        96│         url=url,
        97│         source_root=source_root,

   7  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\vcs\git\backend.py:427 in clone
       425│             if not cls.is_using_legacy_client():
       426│                 local = cls._clone(url=url, refspec=refspec, target=target)
     → 427│                 cls._clone_submodules(repo=local)
       428│                 return local
       429│         except HTTPUnauthorized:

   6  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\vcs\git\backend.py:356 in _clone_submodules
       354│                         continue
       355│
     → 356│                 cls.clone(
       357│                     url=url.decode("utf-8"),
       358│                     source_root=source_root,

   5  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\vcs\git\backend.py:426 in clone
       424│         try:
       425│             if not cls.is_using_legacy_client():
     → 426│                 local = cls._clone(url=url, refspec=refspec, target=target)
       427│                 cls._clone_submodules(repo=local)
       428│                 return local

   4  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\vcs\git\backend.py:258 in _clone
       256│             local = Repo(str(target))
       257│
     → 258│         remote_refs = cls._fetch_remote_refs(url=url, local=local)
       259│
       260│         logger.debug(

   3  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\vcs\git\backend.py:201 in _fetch_remote_refs
       199│
       200│         with local:
     → 201│             result: FetchPackResult = client.fetch(
       202│                 path,
       203│                 local,

   2  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\dulwich\client.py:837 in fetch
        835│             f, commit, abort = target.object_store.add_pack()
        836│         try:
     →  837│             result = self.fetch_pack(
        838│                 path,
        839│                 determine_wants,

   1  ~\AppData\Roaming\pypoetry\venv\lib\site-packages\dulwich\client.py:2076 in fetch_pack
       2074│         """
       2075│         url = self._get_url(path)
     → 2076│         refs, server_capabilities, url = self._discover_references(
       2077│             b"git-upload-pack", url
       2078│         )

  AttributeError

  'NoneType' object has no attribute 'startswith'

  at ~\AppData\Roaming\pypoetry\venv\lib\site-packages\dulwich\client.py:1946 in _discover_references
      1942│                 )
      1943│             base_url = resp.redirect_location[: -len(tail)]
      1944│
      1945│         try:
    → 1946│             self.dumb = not resp.content_type.startswith("application/x-git-")
      1947│             if not self.dumb:
      1948│                 proto = Protocol(read, None)
      1949│                 # The first line should mention the service
      1950│                 try:
neersighted commented 1 year ago

The first one is definitively local config, @rruiter87

jelmer commented 1 year ago

The None Content-Type is an error handling bug; I agree Dulwich should raise a different exception, but it's not the root cause of the issues @rruiter87 is hitting

Ah, I think the number of different combinations of protocols/repositories mentioned here confused me. If it is actually a dumb repository then this might be caused by the missing content type.

jelmer commented 1 year ago

See https://github.com/jelmer/dulwich/issues/1097 for "dumb" remote access support in Dulwich

rruiter87 commented 1 year ago

Thanks for the help people. I'm not going to try and figure it out for now @neersighted, because at least something worked.

jelmer commented 1 year ago

Glancing over the comments here, it looks like the cause was a mix of two things: an invalidly formatted URL and lack of dumb remote repository support in dulwich (masked by incorrect handling of a missing Content-Type header).

abn commented 1 year ago

I can confirm that is the case. The output is with the Content-Type patch added.


$ poetry add git+https://git.bues.ch/git/crcgen.git -v
Using virtualenv: /tmp/foobar/.venv

  NotImplementedError

  <bound method AbstractHttpGitClient.fetch_pack of Urllib3HttpGitClient('https://git.bues.ch/git/crcgen.git/', dumb=True)>

  at python-poetry/poetry/.venv/lib64/python3.10/site-packages/dulwich/client.py:2092 in fetch_pack
      2088│             wants = [cid for cid in wants if cid != ZERO_SHA]
      2089│         if not wants:
      2090│             return FetchPackResult(refs, symrefs, agent)
      2091│         if self.dumb:
    → 2092│             raise NotImplementedError(self.fetch_pack)
      2093│         req_data = BytesIO()
      2094│         req_proto = Protocol(None, req_data.write)
      2095│         (new_shallow, new_unshallow) = _handle_upload_pack_head(
      2096│             req_proto,
Secrus commented 3 months ago

Poetry will support this once Dulwich adds support: jelmer/dulwich#1097

github-actions[bot] commented 2 months ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.