gaborfekete-greehill commented 1 month ago

Description

We are currently having a problem how poetry uses the private github repository's branch naming for cloning. All the following issues are present when trying to install from a docker image, locally it works. We use a multi-repository style structuring for our codebase and several repositories depend on each other: utils, database and image-processing database depends on utils image-processing depends on database and utils

The issue occured when we tried poetry install from within the image-processing repository.

We have two branches in database: main and develop The only difference between the two is that while main uses the utils repository as a submodule and a local path:

[tool.poetry.dependencies]
utils = { path = "./utils"}  # local path for submodule

develop uses the github URL with branch name:

[tool.poetry.dependencies]
utils = { git = "git@github.com:greehill/utils.git", branch = "main" }  # could be develop, but it doesn't matter

When the poetry install command is called from image-processing's docker build command, we get some logs indicating where the problem happened (see Poetry Runtime Logs) It shows that it tries to run:

git clone --recurse-submodules -- REPOSITORY VENV_LOCATION

but it doesn't specify the branch name which was provided in the pyproject.toml.

Because of this the git clone command will have to provide a username/password which could not be fulfilled due to being in an automated docker build. We also tried providing a github personal access token by injecting it into the docker build as a secret and use the git insteadof in the git config to replace the url to use this token but it still failed because it tried to clone the submodules which required the username/password authentication for some reason.

Proposed solution would be to use the git clone command's -b BRANCH_NAME and preferably the --single-branch options.

Workarounds

I don't know of any workarounds

Poetry Installation Method

install.python-poetry.org

Operating System

Ubuntu 20.04.6 LTS

Poetry Version

Poetry (version 1.8.3)

Poetry Configuration

cache-dir = "/home/gabor/.cache/pypoetry"
certificates.project.cert = false
experimental.system-git-client = true
installer.max-workers = null
installer.modern-installation = true
installer.no-binary = null
installer.parallel = true
keyring.enabled = true
solver.lazy-wheel = true
virtualenvs.create = true
virtualenvs.in-project = true
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.no-setuptools = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}/virtualenvs"  # /home/gabor/.cache/pypoetry/virtualenvs
virtualenvs.prefer-active-python = false
virtualenvs.prompt = "{project_name}-py{python_version}"
warnings.export = true

Python Sysconfig

No response

Example pyproject.toml

No response

Poetry Runtime Logs

#11 15.30   CalledProcessError
#11 15.30 
#11 15.30   Command '['git', 'clone', '--recurse-submodules', '--', 'git@github.com:greehill/database.git', '/root/.cache/pypoetry/virtualenvs/image-processing-9TtSrW0h-py3.12/src/database']' returned non-zero exit status 1.

dimbleby commented 1 month ago

are you claiming that cloning the whole repository requires different credentials than cloning the branch? this surprises me.

6348 looks like it was an attempt at what you are asking for but apparently too simplistic. Perhaps that pull request and the comments on it will point you in the right direction if you would like to contribute.

Anyway this issue is basically duplicate #2412, albeit with a new reason. Please leave a comment there describing your new use case, and close this one out