Multiple private packages from GitHub using GitHub Deploy Keys

[mjurkus]

• [x] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).
• OS version and name: Ubuntu 20.04.4 LTS
• Poetry version: 1.2.0b2

Issue

I want to install 2 private packages from the GitHub repository:

// pyproject.toml
...
[tool.poetry.dependencies]
repo-1 = { git = "ssh://git@github.com/my-org/repo-1.git" }
repo-2 = { git = "ssh://git@github.com/my-org/repo-2.git" }

I'm using https://github.com/webfactory/ssh-agent GitHub action to add 2 Deploy Keys. I'm also following these suggestions regarding multiple deploy keys: https://github.com/webfactory/ssh-agent#support-for-github-deploy-keys

      - name: Setup SSH
        uses: webfactory/ssh-agent@v0.5.4
        with:
          ssh-private-key: |
            ${{ secrets.REPO_1_PRVATE_KEY }}
            ${{ secrets.REPO_2_PRIVATE_KEY }}

Keys are successfully added and work when just running a simple git clone git@github.com:my-org/repo-1.git action.

Unfortunately, this setup does not work with poetry install. When installing the first ssh-key will be used and repo-1 will be installed and repo-2 will fail with an error similar to this:

  Repository not found.
  at ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/dulwich/client.py:1123 in fetch_pack
      1119│         with proto:
      1120│             try:
      1121│                 refs, server_capabilities = read_pkt_refs(proto.read_pkt_seq())
      1122│             except HangupException:
    → 1123│                 raise _remote_error_from_stderr(stderr)

When using 1 repository and 1 deploy key - everything works as expected.

[khendrickx]

We have the same issue and found this to be related to git clone and ssh key authentication.

Git clone will open a ssh connection to git@github.com by looping over they private keys and selecting the first one that allows you to log in. However, at that stage, it is unaware of the repo you want to clone.

Consequently, in your example, git clone will always use secrets.REPO_1_PRVATE_KEY which does not provide access to my-org/repo-2.git.

We're currently using this gist as a hack to circumvent the issue: https://gist.github.com/vhermecz/4e2ae9468f2ff7532bf3f8155ac95c74

[dimbleby]

https://github.com/webfactory/ssh-agent#support-for-github-deploy-keys explains that it works by playing around with git's insteadOf configuration.

That might actually work as of poetry 1.2.0b3 per fixes for https://github.com/python-poetry/poetry/issues/5934 - upgrading is worth a try anyway

[aradipe]

I had the same issue, and have worked around it by downgrading poetry:

+++ b/.github/workflows/ci.yml
@@ -16,7 +16,7 @@ jobs:
         uses: AppThreat/sast-scan-action@master

       - name: Install poetry
-        run: pipx install poetry
+        run: pipx install poetry==1.1.14```

[kklecho]

Looks like web factory is now handling this problem by mapping the right key to the right repo using key comment.

[RobinFrcd]

Same problem here, the action is a thing but if we need to handle this in all the CI builds, this quickly becomes unmaintainable :/

[mripani]

Any updates on this?