Closed caphrim007 closed 11 months ago
You must have write or admin access.
is strange. Perhaps it's an incorrect log message and only read is necessary.
We will need an enhancement to our go artifacts code here: https://github.com/renovatebot/renovate/blob/6ec460b735e45f5a6d65662a2c5ffd51f8e3a5d6/lib/manager/gomod/artifacts.ts#L13-L29
@rarkins in much the same spirit as was done with python hash support, I'm happy to test anything that may be proposed. I will try duplicating the mentioned code and adjusting for Bitbucket to see if that results in any progress. Will provide feedback here for you.
One question, are host-rules capable of being specified in the Renovate configuration that runs as a cronjob? or must it be specified in the renovate.json within the repository under management.
hostRules can be defined in the bot config too
Hey, I'm having a similar issue.
I'm using renovate cli from docker image on bitbucket pipelines like:
renovate-bot:
- step:
image: renovate/renovate:23.89
name: Renovate Bot
clone:
enabled: false
script:
- export GOPRIVATE=bitbucket.org/my_org/*
- export BB_USER=renovateuser
- git config --global url."https://${BB_USER}:${RENOVATE_PASSWORD}@bitbucket.org".insteadOf "https://bitbucket.org"
- renovate --username=${BB_USER} --log-level debug --platform=bitbucket $BITBUCKET_REPO_FULL_NAME
however, getting the following error:
DEBUG: Failed to look up dependency bitbucket.org/my_org/go-lib
Question:
git config --global
I provided before, seems to be ignored. Is it because renovatebot
overwrites it during run?hostRules
or it will not work until #3496 is implemented? If adding hostRules
can help, what is the correct format for the CLI parameter? Is it --host-rules hostName:bitbucket.org,username:${BB_USER},password:${RENOVATE_PASSWORD}
?the parameter value need to be an stringified json array. so better add a config.js
and use that to configure hostrules with passwords from process.env.XXX_PASSWORD
.
Renovate doesn't use your gitconfig for lookup updates, it needs a hostRule for auth.
@viceice thanks for your reply!
Can you share the example of how to use process.env.XXX_PASSWORD
in config? I was trying to find something in the docs, but no luck for now.
Would it be something like:
"hostRules": [
{
"domainName": "bitbucket.org",
"username": process.env.BB_USER,
"password": process.env.RENOVATE_PASSWORD
}
]
and that is it?
yes:
module.exports = {
"hostRules": [
{
"domainName": "bitbucket.org",
"username": process.env.BB_USER,
"password": process.env.RENOVATE_PASSWORD
}
]
}
I had no luck with using this, it still fails.
From what I see, hostRules
are applied for the renovate
app itself, but not the external tools it calls (like go get).
Code here has hardcoded github, so no matter what is set in the hostRules
it is not being applied:
https://github.com/renovatebot/renovate/blob/a1cddeae2c0a9d5b942413456f46833560725782/lib/manager/gomod/artifacts.ts#L12-L25
So looks like there is no way to use it until https://github.com/renovatebot/renovate/issues/3496 is implemented.
Interesting that setting git config --global url."https://${BB_USER}:${RENOVATE_PASSWORD}@bitbucket.org".insteadOf "https://bitbucket.org"
makes go get
work for me, but when the renovatebot runs it doesn't work :(
Where/how do you run the bot? e.g. are you using our official image, or installing the CLI via npm into your own container, etc?
@rarkins right now I'm using the official image locally:
$ docker run -it --rm renovate/renovate -v
23.86.0
Also,
I'm using renovate cli from docker image on bitbucket pipelines like:
renovate-bot: - step: image: renovate/renovate:23.89 name: Renovate Bot clone: enabled: false script: - export GOPRIVATE=bitbucket.org/my_org/* - export BB_USER=renovateuser - git config --global url."https://${BB_USER}:${RENOVATE_PASSWORD}@bitbucket.org".insteadOf "https://bitbucket.org" - renovate --username=${BB_USER} --log-level debug --platform=bitbucket $BITBUCKET_REPO_FULL_NAME
Also, interesting that I do not see any specific error explaining what happened:
DEBUG: Failed to look up dependency bitbucket.org/my_org/go-dep1 (repository=my_org/my_service, packageFile=go.mod, dependency=bitbucket.org/my_org/go-dep1)
DEBUG: Failed to look up dependency bitbucket.org/my_org/go-dep2 (repository=my_org/my_service, packageFile=go.mod, dependency=bitbucket.org/my_org/go-dep2)
DEBUG: Failed to look up dependency bitbucket.org/my_org/go-dep3 (repository=my_org/my_service, packageFile=go.mod, dependency=bitbucket.org/my_org/go-dep3)
DEBUG: Failed to look up dependency bitbucket.org/my_org/go-dep4 (repository=my_org/my_service, packageFile=go.mod, dependency=bitbucket.org/my_org/go-dep4)
DEBUG: Failed to look up dependency bitbucket.org/my_org/go-dep5 (repository=my_org/my_service, packageFile=go.mod, dependency=bitbucket.org/my_org/go-dep5)
no details at all
You would need a host rule for Renovate, and the git config rule for go
. Did you have both? That lookup failure would most likely be a host rule missing.
I do have git config rule for go:
git config --global url."https://${BB_USER}:${RENOVATE_PASSWORD}@bitbucket.org".insteadOf "https://bitbucket.org"
And it works for me when I run go get
.
About the host rule, I tried in many different ways and none of them worked for me, can you share an example of how should it look like?
In some cases adding host config, the rule didn't change anything, in some I got permission denied, and also I got Too many redirects error
.
Also, in the trace logs, I was able to spot:
{"name":"renovate","hostname":"e19b786e7ef3","pid":1682,"level":10,"logContext":"bHvOPOxOl","repository":"my_org/my_service","url":"https://bitbucket.org/my_org/go-dep1?go-get=1","options":{"method":"get","hostType":"go","hooks":{"beforeRedirect":[null]},"headers":{"accept-encoding":"gzip, deflate","user-agent":"https://github.com/renovatebot/renovate"},"timeout":60000},"msg":"got request","time":"2020-12-04T12:44:34.068Z","v":0}
And when I tried to fetch the URL https://bitbucket.org/my_org/go-dep1?go-get=1
from Postman I also get Too Many Redirects URL
. Valid API URL would be the https://api.bitbucket.org/2.0/repositories/my_org/go-dep1?go-get=1
yes:
module.exports = { "hostRules": [ { "domainName": "bitbucket.org", "username": process.env.BB_USER, "password": process.env.RENOVATE_PASSWORD } ] }
Didn't this work in combination with your global git config?
Nope, let me try again, so I can share what was the outcome.
yes, some more debug log lines would be helpful
@Trane9991 Can you create a small public github repo with a go dependency pointing to a public bitbucket repo. If this works, we know it's and auth problem, otherwise it's a general renovate go datasource issue
ok, let me try that
maybe we need to add some special handling for bitbucket.org here: https://github.com/renovatebot/renovate/blob/2932e8859870972937b8478907e3f8a55f549a9d/lib%2Fdatasource%2Fgo%2Findex.ts#L31
so like we replace github.com
with api.github.com
Here, it failed with the public dependency as well: https://bitbucket.org/Trane9991/renovatebot-issue/addon/pipelines/home#!/results/1
DEBUG: Failed to look up dependency bitbucket.org/AiSee/common-go-lib (repository=Trane9991/renovatebot-issue, packageFile=go.mod, dependency=bitbucket.org/AiSee/common-go-lib)
You probably right about this:
maybe we need to add some special handling for bitbucket.org here: https://github.com/renovatebot/renovate/blob/2932e8859870972937b8478907e3f8a55f549a9d/lib%2Fdatasource%2Fgo%2Findex.ts#L31
so like we replace
github.com
withapi.github.com
For example, this handling of bitbucket.org
in the go code:
and finally here is the query against api.bitbucket.org: https://github.com/golang/go/blob/ac38af2f3db7b16067bd8983d1f5278c5c9ef706/src/cmd/go/internal/vcs/vcs.go#L1275_L1280
I'll check the code, maybe i can send a fast feature to support this
nope, needs a new bitbucket-tags
datasource
@viceice @rarkins I made an attempt to resolve this in PR #7892. Since I'm not very good with TS and not familiar with the codebase, would be glad to receive early feedback.
I want to try to run it on the repo I prepared before to make sure that it works, just need to figure out how to run it locally :)
Ok, was able to run it locally. Getting tags seems to work fine, but that repo has no Tags so it fails.
DEBUG: Found no results from datasource that look like a version (repository=trane9991/renovatebot-issue, dependency=bitbucket.org/AiSee/common-go-lib)
"result": {"sourceUrl": "https://api.bitbucket.org/AiSee/common-go-lib", "releases": []}
DEBUG: Could not getDigest (repository=trane9991/renovatebot-issue)
"newValue": "v0.0.0-20191112142628-3dc422aaa32a"
Looks like I need to implement getDigest
as well.
module.exports = {
"hostRules": [
{
"domainName": "bitbucket.org",
"username": process.env.BB_USER,
"password": process.env.RENOVATE_PASSWORD
}
]
}
Is the above config , the correct way to get the private go packages if they are in the bitbucket server. @viceice
It should probably work 🤷♂️ I do not use go nor bitbucket.org, so you need to validate yourself.
Please postback any issues here. If it works we can close this issue.
I hate to piggy back on another issue but I appear to be running into this exact issue trying to run renovate on a kubernetes cluster.
the logs show:
"artifactErrors": [
{
"lockFile": "go.sum",
"stderr": "Command failed: go get -d -t ./...\ngo: bitbucket.org/<org>/<repo>@<tag>: reading https://api.bitbucket.org/2.0/repositories/<org>/<repo>?fields=scm: 403 Forbidden\n\tserver response: Access denied. You must have write or admin access.\n"
}
]
for all of our private repos.
I can confirm from other experience that this error message is generic and shows up in places where only read access is needed.
Just like the OP my pull requests are opened but a note is left with an error for each failed go.sum update.
my abridged (removed the repos list) config is
{
"platform": "bitbucket",
"username": "<user>",
"password": "RENOVATE_APP_PASSWORD",
"hostRules": [
{
"domainName": "bitbucket.org",
"username": "<user>",
"password": "RENOVATE_APP_PASSWORD"
}
],
"postUpdateOptions": [
"gomodTidy"
]
}
My most recent run from today was using renovate/renovate:31.66.3
but I can happily test a dev build if someone has one.
So I couldn't really leave this alone lol...
I took another look at the renovate config docs as I knew I had seen a mention of GOPRIVATE
support (as this is what we normally do in our Go environments to be able to access these repos, normally we also set a URL in our gitconfig to force using git over ssh vs HTTPS (I don't think such a thing is needed for a bitbucket bot user token?)) but I'm unclear reading the documentation how this value is meant to be set from within the config.
So I took a look at the code and found https://github.com/renovatebot/renovate/blob/main/lib/datasource/go/releases-goproxy.ts#L175 so I was hoping that simply setting the GOPRIVATE
value within my kubernetes environment variables would work for this (I editted my yaml on my k8s workload directly to test this) but it seems that I'm still getting the same error in my previous post.
Happy to try to debug this (even by taking a stab at it myself, though my TS is blah at best) but I think I need at least some direction to attempt that.
Hi there,
Get your issue fixed faster by creating a minimal reproduction. This means a repository dedicated to reproducing this issue with the minimal dependencies and config possible.
Before we start working on your issue we need to know exactly what's causing the current behavior. A minimal reproduction helps us with this.
To get started, please read our guide on creating a minimal reproduction.
We may close the issue if you, or someone else, haven't created a minimal reproduction within two weeks. If you need more time, or are stuck, please ask for help or more time in a comment.
Good luck,
The Renovate team
Which Renovate are you using?
Renovate Open Source CLI
Which platform are you using?
Bitbucket Cloud
Have you checked the logs? Don't forget to include them if relevant
I'm not sure if the logs are relevant or not. If they are, please advise.
What would you like to do?
I have a BitBucket cloud repo with a go.mod file in it which resembles the following,
The relevant part of this question concerns the bitbucket.org
require
line. This is a private Bitbucket-cloud repository.Renovate successfully finds and creates an issue with an update to the github.com requirements. I have a
excludePackagePatterns
package rule defined for the bitbucket.org due to missing support for such things in Renovate.After Renovate creates the PR, it attempts to update the go.sum file. It fails though and adds a comment to the PR with the following in it
For the life of me, I can't see to figure out what I need to configure to give it the necessary access. The user that renovate runs as has an app token and is a member of a group in Bitbucket which has write access to all repos.
I was hoping there is an obvious config or environment variable I can configure on the renovate side (not the repo side) to give it the necessary info (like a host rule or something).
I see mention of Github specific config concerning preCommands here
https://github.com/renovatebot/renovate/blob/master/lib/manager/gomod/artifacts.ts#L13
though its unclear to me if applying the same for Bitbucket would make the above
go
command work as this appears to be specifically raised bygo
.Any help is appreciated.