Closed hongaar closed 2 years ago
viceice
commented
4 years ago As the error states, renovate / composer has no access to the repo.
Can you please provide some more debug log from the renovate dashboard. You can find the link in pr footer.
hongaar
commented
4 years ago @viceice not sure what is relevant to you, so simply copy-pasting a full redacted log from the dashboard here. I trimmed all our public (non-failing) dependencies, and kept references to the failing private dependency.
Let me know if you need anything else.
rarkins
commented
4 years ago @hongaar is it possible for you to reproduce this in a way that we can run debug against? I think:
I'm not sure that you need A to have multiple releases or not - you might just need one other normal dependency in B to need updating in order to trigger a PR and the subsequent failure.
viceice
commented
4 years ago @rarkins why renovate tries to pin the ignored dependency?
rarkins
commented
4 years ago @rarkins why renovate tries to pin the ignored dependency?
It's not necessarily, hence why I want to see a reproduction. I might be pinning one dependency but it's a totally different github one that fails in artifacts.
stale[bot]
commented
4 years ago This issue has been automatically marked as stale because it has not had recent activity. It will be closed soon if no further activity occurs.
hongaar
commented
4 years ago Triggered by a trigger-happy stale bot 😉
@rarkins Created two repositories as suggested, keeping them as minimal as I could and made sure to include the same mechanism for requiring a private repository as in the original case.
I've invited you as a collaborator to both repositories, if you need it.
Hope this helps! Let me know if you need anything else.
danez
commented
4 years ago I have the exact same issue, with the difference that we use bitbucket server and not github. We actually have an auth.json file (https://getcomposer.org/doc/articles/http-basic-authentication.md) next to composer.json, but this file does not seem to be picked up when renovate bot runs.
The file though usually works, because otherwise our producton servers wouldn't be able to connect to our private git server.
{
"http-basic": {
"git.xxx.xxx": {
"username": "composer",
"password": "abcdefghijklmnopq"
}
},
"github-oauth": {
"github.com": "1234567890"
}
}If I understand the code correctly it looks like it creates it's own auth.json file here: https://github.com/renovatebot/renovate/blob/master/lib/manager/composer/artifacts.ts#L101 Would you accept a PR that instead of overwritting the file, merges it with whatever is already there?
rarkins
commented
4 years ago Would you accept a PR that instead of overwritting the file, merges it with whatever is already there?
Absolutely!
zharinov
commented
4 years ago For private repos, can release versions be obtained from Github tags? For now I can't figure out if we should rely on tags/branches. Or maybe, there should be some Composer-specific declarations?
rarkins
commented
4 years ago @hongaar we've been testing a scenario today similar to how you originally described, and it now works. Both repos are private to properly replicate it, but the one that was previously failing but now working looks like this:
{
"name": "test",
"description": "",
"repositories": [
{
"name": "zharinov/renovate-composer-dep",
"type": "vcs",
"url": "git@github.com:zharinov/renovate-composer-dep.git"
}
],
"require": {
"zharinov/renovate-composer-dep": "1.0.2",
"justinrainbow/json-schema": "5.2.8"
}
}Now, a PR to update json-schema works, without Composer complaining that it can't access the github-hosted dep.
Are you able to verify your original use case?
hongaar
commented
4 years ago @rarkins Thanks for looking into this.
I'm still getting an error unfortunately:
[RuntimeException]
Failed to execute git clone --mirror 'https://**redacted**:***@github.com/exivity/octopus.git' '/tmp/renovate-cache/others/composer/vcs/git-github.com-exivity-octopus.git/'
Cloning into bare repository '/tmp/renovate-cache/others/composer/vcs/git-github.com-exivity-octopus.git'...
remote: Invalid username or password.
fatal: Authentication failed for 'https://github.com/exivity/octopus.git/' The relevant sections in composer.json:
{
"repositories": [
{
"name": "exivity/octopus",
"type": "git",
"url": "git@github.com:exivity/octopus.git"
}
],
"require": {
"exivity/octopus": "^4.0"
}
}We have validated an almost identical scenario as working:
{
"name": "test",
"description": "",
"repositories": [
{
"name": "zharinov/renovate-composer-dep",
"type": "vcs",
"url": "git@github.com:zharinov/renovate-composer-dep.git"
}
],
"require": {
"zharinov/renovate-composer-dep": "1.0.2",
"justinrainbow/json-schema": "5.2.8"
}
}I don't know if the git/vcs makes a difference. Also, do these two repos fall under the same org, and they're both private?
hongaar
commented
4 years ago I don't know if the git/vcs makes a difference.
Will test now.
Also, do these two repos fall under the same org, and they're both private?
That's correct.
rarkins
commented
4 years ago Also, both repos need to have Renovate installed, otherwise it has no authorisation to access the second one
hongaar
commented
4 years ago @rarkins changing git to vcs seems to have fixed the problem! 🎉
Problem solved for us, but maybe others will run into this as well. The vcs is a catch-all type and can be narrowed down by the user. These are valid sub-types (don't know how they translate to renovate's internal logic):
git-bitbucket, hg-bitbucket, github, gitlab, perforce, fossil, git, svn, hg
rarkins
commented
4 years ago Lowering this to normal priority. @zharinov can you try to work out if it's something we're doing differently, or is it Composer? e.g. if it's vcs does it auto-detect that it's github and use the github oauth we provide it, while if it's git then it does not? and if the latter, is there some other auth field we should put our github token so that it does work?
jaska120
commented
4 years ago I am facing the same auth issue as stated above while trying to use private repo as composer dependency. I am running Renovate using official Github Action v23.40.0 and the access rights seems ok, since if I define the same repository in renovate.json the Renovate manages to run in the same (private) dependency repo as I try to require with composer - only composer fails to access:
DEBUG: Datasource unknown error (repository=[org]/[reponame])
"datasource": "git-tags",
"lookupName": "git@github.com:[org]/[reponame].git",
"err": {
"task": {
"commands": [
"ls-remote",
"git@github.com:[org]/[reponame].git"
],
"format": "utf-8"
},
"message": "Host key verification failed.\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n",
"stack": "Error: Host key verification failed.\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n\n at GitExecutorChain.onFatalException (/usr/src/app/node_modules/simple-git/src/lib/runners/git-executor-chain.ts:66:77)\n at GitExecutorChain.<anonymous> (/usr/src/app/node_modules/simple-git/src/lib/runners/git-executor-chain.ts:58:21)\n at Generator.throw (<anonymous>)\n at rejected (/usr/src/app/node_modules/simple-git/src/lib/runners/git-executor-chain.js:6:65)\n at runMicrotasks (<anonymous>)\n at processTicksAndRejections (internal/process/task_queues.js:97:5)"Any suggestions how to fix this? The problem persist even if using the Renovate App on Github. (Org and reponames hidden).
@rarkins on which version did you make it work on your tests so that I could try the same version using Github Actions?
rarkins
commented
4 years ago It won't work if the repo is accessed using git/ssh instead of https, because that requires an ssh key to work and neither the app or actions have ssh keys.
jaska120
commented
4 years ago Thanks for the reply,
I understand the problem. However, on your example you have used git@github.com domain successfully (which will use ssh?).
I did try to use https protocol for the dependency but then I am getting error:
"message": "fatal: could not read Username for 'https://github.com': No such device or address\n",Should I define username and password in renovate.json and inject those while running Action or how one could make composer use token instead of ssh or username password combo while requiring dep from Github?
rarkins
commented
4 years ago The action's automatic token has only permissions for its own repo. To access another repo you'll need to use a PAT and add it with hostRules for hostName=api.gitHub.com and hostType=composer.
simonschaufi
commented
3 years ago I'm having the same composer setup like described here https://github.com/renovatebot/renovate/issues/5616#issuecomment-682480731 and @rarkins said that it is working with git@domain.com repository urls. I'm using GitLab and added my private key like documented here: https://docs.gitlab.com/ee/ci/ssh_keys/#ssh-keys-when-using-the-docker-executor but it looks like the private key is then only available in the gitlab runner container and NOT in the renovate docker container, right? Or can I pass the private key somehow to the renovate docker container as well?
It won't work if the repo is accessed using git/ssh instead of https, because that requires an ssh key to work and neither the app or actions have ssh keys.
is that still true in 2021? It would be so easy to just put the private key in an environment variable and assign that to the renovate docker container and everything would work without tokens.
I see so many people having problems with private repositories and I'm also searching for hours for a solution and have not really found it yet how I can access dependent private composer repositories with an ssh key. Please enlighten me :bulb:
mbrodala
commented
3 years ago Also as mentioned in a discussion (https://github.com/renovatebot/renovate/discussions/9295#discussioncomment-1211876) it would also be useful if one could tell Renovate about a running SSH agent.
For our own Docker containers this means 2 things:
SSH_AUTH_SOCK environment variable pointing to that socket in the mounted directoryThe latter would be possible with customEnvVariables AFAIS but I have no idea about the former.
Directly exposing an SSH key using environment variables would be a less secure solution IMO.
mbrodala
commented
3 years ago Also I'm not sure if one should set up hostRules for GitLab with a token which are supposedly forwarded to Composer using COMPOSER_AUTH.
The source seems to suggest so:
https://github.com/renovatebot/renovate/blob/26.4.3/lib/manager/composer/artifacts.ts#L48-L61
hostRules
.findAll({ hostType: PLATFORM_TYPE_GITLAB })
?.forEach((gitlabHostRule) => {
if (gitlabHostRule?.token) {
const host = gitlabHostRule.resolvedHost || 'gitlab.com';
authJson['gitlab-token'] = authJson['gitlab-token'] || {};
authJson['gitlab-token'][host] = gitlabHostRule.token;
// https://getcomposer.org/doc/articles/authentication-for-private-packages.md#gitlab-token
authJson['gitlab-domains'] = [
host,
...(authJson['gitlab-domains'] || []),
];
}
});Also as mentioned in a discussion (#9295 (reply in thread)) it would also be useful if one could tell Renovate about a running SSH agent.
For our own Docker containers this means 2 things:
- Mounting a volume where the SSH agent has created a socket
- Setting the
SSH_AUTH_SOCKenvironment variable pointing to that socket in the mounted directoryThe latter would be possible with
customEnvVariablesAFAIS but I have no idea about the former.Directly exposing an SSH key using environment variables would be a less secure solution IMO.
What about to set the socket to /tmp/renovate/cache/ssh.sock (renovate cacheDir) this is mounted to all child containers at same path
https://docs.renovatebot.com/self-hosted-configuration/#cachedir
mbrodala
commented
3 years ago @viceice Maybe that could work, I'd need to try that. Still that would be bad for concurrency since both the SSH agent as well as Renovate would then need to access a physical path on the host system. So if multiple SSH agents and Renovate containers are running concurrently, they could clash on this path.
Normally we use a named volume instead here which can be safely shared between multiple containers.
viceice
commented
3 years ago Yes, i like to have some configurable volumes for docker mode in future. You could try the runnind docker in docker, so the whole renovate directory can be a volume. This is how our gitlab pipeline templates work.
maybe https://github.com/renovatebot/github-action/issues/571 is helpful for you
simonschaufi
commented
3 years ago @viceice What is the preferred alternative to have private dependency repositories like mentioned here https://github.com/renovatebot/renovate/issues/5616#issuecomment-682480731 required with git@example.com? I'm using ssh keys everywhere to fetch the dependency but since I can't inject the ssh key in renovate, switching to https urls needs an authentication but I haven't found any solution that work.
viceice
commented
3 years ago As current workaround you can use a ssh agent with file based socket pointing to a file inside renovate cacheDir and using customEnvVariables to pass the agent socket info to renovate child container. The cache dir is automatically passed.
@rarkins @HonkingGoose Maybe we can add a FAQ for this to the docs, as i think this should work for most ssh users.
Maybe we can try to automaticall mount the ssh agent socket in a future renovate version, if this works well.
rarkins
commented
3 years ago Would #11470 solve this?
HonkingGoose
commented
3 years ago @rarkins @HonkingGoose Maybe we can add a FAQ for this to the docs, as i think this should work for most ssh users.
Do we need to document this urgently? Or can we wait until we have know if issue #11470 will fix the problem?
Maybe doing the fix in the code is easy and quick, and then we don't need the FAQ entry.
If fixing the problem in the code is hard, we could put a "workaround" in the FAQ to help people until we can fix it properly.
mbrodala
commented
3 years ago @rarkins Can you elaborate on how #11470 could help here? In our case packages are required as dev-<branch> in Composer thus a Git clone is necessary. There is no alternative to cloning from the given source URL. So where would you retrieve the code from with insteadOf?
rarkins
commented
3 years ago It could work if the repository was available via git over https, by using a https git url with token embedded instead of ssh
viceice
commented
3 years ago @rarkins @HonkingGoose Maybe we can add a FAQ for this to the docs, as i think this should work for most ssh users.
Do we need to document this urgently? Or can we wait until we have know if issue #11470 will fix the problem?
Maybe doing the fix in the code is easy and quick, and then we don't need the FAQ entry.
I think this is not easy and fast to fix. And we can't message all ssh urls to https, because we only do this for known hosts.
If fixing the problem in the code is hard, we could put a "workaround" in the FAQ to help people until we can fix it properly.
I would prefer to add that section, as it would be a quick workaround for any future ssh issues we even not yet know about.
HonkingGoose
commented
3 years ago @viceice Can you make a draft PR to add the workaround to the FAQ? I can try to help with the wording once we have a draft of the text.
mbrodala
commented
3 years ago I'll try to test if the suggested workaround actually works. ;-)
viceice
commented
3 years ago I would prefer to first test the workaround and then document it 😉 https://github.com/renovatebot/renovate/issues/5616#issuecomment-903846364
SSH_AUTH_SOCK=/tmp/renovate/cache/ssh.sock (default renovate cacheDir)customEnvVariables: { 'SSH_AUTH_SOCK': process.env.SSH_AUTH_SOCK }This should work for all binarySource modes
simonschaufi
commented
3 years ago @viceice I might do something wrong but it doesn't work for me like you described.
composer.json:
{
"repositories": [
{ "type": "git", "url": "git@example.com:vendor/repo.git" }
],
"require": {
"vendor/repo": "@dev",
"giggsey/libphonenumber-for-php": "^8.12"
}
}locked is version 8.12.30. executed like so:
composer req giggsey/libphonenumber-for-php:8.12.30
composer req giggsey/libphonenumber-for-php:^8.12 --no-updaterenovate.json
{
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"extends": [
"config:base",
"docker:enableMajor",
":disableMajorUpdates",
":prHourlyLimitNone",
":renovatePrefix"
],
"rangeStrategy": "update-lockfile",
"customEnvVariables": {
"SSH_AUTH_SOCK": "process.env.SSH_AUTH_SOCK"
}
}.gitlab-ci.yml
include:
- local: /renovate.yml
stages:
- renovate
variables:
DOCKER_DRIVER: overlay2
COMPOSER_CACHE_DIR: "$CI_PROJECT_DIR/.cache/composer"
before_script:
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client git -y )'
- mkdir -p /tmp/renovate/cache
- eval $(ssh-agent -s -a /tmp/renovate/cache/ssh.sock)
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- ssh-keyscan example.com >> ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
Renovate:
extends: .renovate
stage: renovate
variables:
RENOVATE_LOG_LEVEL: "debug"renovate.yml
.renovate:
variables:
RENOVATE_GIT_AUTHOR: "${GITLAB_USER_NAME} <${GITLAB_USER_EMAIL}>"
RENOVATE_DRY_RUN: "false"
RENOVATE_LOG_LEVEL: "info"
LOG_LEVEL: "${RENOVATE_LOG_LEVEL}"
RENOVATE_CONFIG: '{"customEnvVariables":{"SSH_AUTH_SOCK":"process.env.SSH_AUTH_SOCK"}}'
image:
name: renovate/renovate:26.13-slim
script:
- >
renovate
--platform "gitlab"
--endpoint "${CI_API_V4_URL}"
--git-author "${RENOVATE_GIT_AUTHOR}"
"${CI_PROJECT_PATH}"
rules:
- if: '$CI_PIPELINE_SOURCE == "schedule"'
- when: neverI set GITHUB_COM_TOKEN, RENOVATE_TOKEN and SSH_PRIVATE_KEY as gitlab env variables and replaced my internal domain with example.com
viceice
commented
3 years ago COMPOSER_CACHE_DIR as it's not used with renovateentrypoint: [""]eval $(ssh-agent -s -a /tmp/renovate/cache/ssh.sock)customEnvVariables is a global option, so needs to be in runner config.js and not in repo renovate.jsonYou need to tell the ssh agent to use your custom socket path
Checkout https://gitlab.com/renovate-bot/renovate-runner for best practices / pipeline templates for running renovate in gitlab pipelines
viceice
commented
3 years ago Remove --dry-run "${RENOVATE_DRY_RUN}" arg, as RENOVATE_DRY_RUN will do the same
viceice
commented
3 years ago mkdir /tmp/renovate/cache before running agent? 😏
viceice
commented
3 years ago @simonschaufi Please open a discussion, as this is off-topic
viceice
commented
3 years ago @simonschaufi Please lets continue this in the discussion, so we only copy teh final working smple here.
viceice
commented
3 years ago Here are the working samples for gitlab pipeline
Slim image with DinD and SSH agent. Requires GitLab DinD configuration for self-hosted gitlab runner.
.gitlab-ci.yml
include:
- project: 'renovate-bot/renovate-runner'
file: '/templates/renovate-dind.gitlab-ci.yml'
ref: v4.4.2 # check latest at https://gitlab.com/renovate-bot/renovate-runner/-/releases
variables:
SSH_AUTH_SOCK: $RENOVATE_BASE_DIR/cache/ssh.sock
RENOVATE_CUSTOM_ENV_VARIABLES: '{"SSH_AUTH_SOCK":"$RENOVATE_BASE_DIR/cache/ssh.sock"}'
before_script:
- mkdir -p $RENOVATE_BASE_DIR/cache
- eval $(ssh-agent -s -a $SSH_AUTH_SOCK)
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- ssh-keyscan example.com >> ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hostsFull renovate image with SSH agent
.gitlab-ci.yml
include:
- project: 'renovate-bot/renovate-runner'
file: '/templates/renovate.gitlab-ci.yml'
ref: v4.4.2 # check latest at https://gitlab.com/renovate-bot/renovate-runner/-/releases
variables:
SSH_AUTH_SOCK: $RENOVATE_BASE_DIR/cache/ssh.sock
RENOVATE_CUSTOM_ENV_VARIABLES: '{"SSH_AUTH_SOCK":"$RENOVATE_BASE_DIR/cache/ssh.sock"}'
before_script:
- mkdir -p $RENOVATE_BASE_DIR/cache
- eval $(ssh-agent -s -a $SSH_AUTH_SOCK)
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- ssh-keyscan example.com >> ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts@viceice Is there any way to mount an external volume into the Renovate Docker container(s)? We do not set up the SSH agent manually on the host system but using a separate Docker container instead. This one exposes a volume which can be mounted in any other container and contains the SSH agent socket.
viceice
commented
3 years ago @mbrodala No, renovate can't mount arbitrary volumes to sidecars.
You need to use DinD then, like it's done in gitlab. Then you can mount your volume in the base container and share it between renovate and dind. Renovate needs a shared path between main and sidecar container.
Can you share your basic setup in a new discussion? maybe we can find a solution for you too.
mbrodala
commented
3 years ago @viceice We do use DinD all the time. But I have no idea how I could mount a volume into the base container here.
If I would manually launch Renovate using Docker or Docker Compose, everything would be clear. But I don't know if and how GitLab supports volume mounts for the Docker executor.
viceice
commented
3 years ago @mbrodala Let's discuss it in a new discussion
mbrodala
commented
3 years ago
Which Renovate are you using?
WhiteSource Renovate App
Which platform are you using?
GitHub.com
Have you checked the logs? Don't forget to include them if relevant
Yes. They contain more or less the same info as below.
What would you like to do?
We configured Renovate for a PHP/Composer project which contains packages from a private GitHub repository.
Renovate can't run composer to update the lockfile, as composer can't clone the private repository.
This is the full comment on the Renovate PR:
Our
composer.jsoncontains this fragment:Note: added the
nameas suggested by https://github.com/renovatebot/renovate/issues/4924, which didn't seem to change the error.Note: I tried pinning the dependency in the
requireobject, also without luck.Renovate has been granted access (through the GitHub app) to the private dependency referenced in the repositories field.
The
renovate.jsoncontains:However, adding the
ignoreDepsdidn't seem to change the error either.Is this use-case supported, and if so, how can I configure Renovate so composer can read private repositories?