search

renovatebot / renovate

Home of the Renovate CLI: Cross-platform Dependency Automation by Mend.io
https://mend.io/renovate
GNU Affero General Public License v3.0
17.14k stars 2.23k forks source link

Indirect dependencies in dependent go modules need to be updated #12999

Open eriksw opened 2 years ago

eriksw commented 2 years ago

What would you like Renovate to be able to do?

Because go modules include indirect dependencies in the go.mod and go.sum, when Renovate updates a go module, it also needs to update dependent modules.

Scenario:

monorepo/a:
    has a dependency that renovate is updating
monorepo/b:
    has a dependency on monorepo/a
    has a replace rule for monorepo/a => ../a
    does not directly import the library
    has "indirect" go.mod and go.sum entries for the dependencies of a because go adds/requires them now

When renovate updates monorepo/a/... it needs to also update monorepo/go/b/go.* at the same time, in the same commit.

Because renovate currently does not do this, the PR to update module a fails because we have a test that ensures our module files are always tidy. That fails because when the test runs go mod tidy in b, it is stale and updated/rewritten by the go mod tidy invocation in b, which updates b's indirect dependencies in go.mod and go.sum.

Reproduction repo/PR: https://github.com/eriksw/renovate-peer-go-mod-indirect/pull/1

Log:

DEBUG: No dangling containers to remove INFO: Repository started { "renovateVersion": "29.32.5" } DEBUG: Using localDir: /mnt/renovate/gh/eriksw/renovate-peer-go-mod-indirect DEBUG: Repository cache not found { "cacheFileName": "/tmp/renovate-cache/renovate/repository/github/eriksw/renovate-peer-go-mod-indirect.json" } DEBUG: initRepo("eriksw/renovate-peer-go-mod-indirect") DEBUG: Overriding default GitHub endpoint { "endpoint": "https://api.github.com/" } DEBUG: eriksw/renovate-peer-go-mod-indirect default branch = main DEBUG: Using app token for git init DEBUG: resetMemCache() DEBUG: Resetting npmrc DEBUG: detectSemanticCommits() DEBUG: Initializing git repository into /mnt/renovate/gh/eriksw/renovate-peer-go-mod-indirect DEBUG: Performing blobless clone DEBUG: git clone completed { "durationMs": 1234 } DEBUG: latest repository commit { "latestCommit": { "hash": "3d7e94578947ca98bffbd997a282b134ebbd1429", "date": "2021-12-07T12:53:59-08:00", "message": "Initial commit", "refs": "HEAD -> main, origin/main, origin/HEAD", "body": "", "author_name": "Erik Swanson", "author_email": "erik@retailnext.net" } } DEBUG: getCommitMessages DEBUG: Semantic commits detection: unknown DEBUG: No semantic commits detected DEBUG: checkOnboarding() DEBUG: isOnboarded() DEBUG: findFile(renovate.json) DEBUG: Config file exists { "fileName": "renovate.json" } DEBUG: Retrieving issueList DEBUG: Retrieved 0 issues DEBUG: Repo is onboarded DEBUG: Found renovate.json config file DEBUG: Repository config { "fileName": "renovate.json", "config": { "extends": [ "config:base" ], "postUpdateOptions": [ "gomodTidy", "gomodUpdateImportPaths" ], "prConcurrentLimit": 5, "rebaseWhen": "conflicted" } } DEBUG: migrateAndValidate() DEBUG: No config migration necessary DEBUG: massaged config { "config": { "extends": [ "github>whitesource/merge-confidence:beta", "config:base" ], "postUpdateOptions": [ "gomodTidy", "gomodUpdateImportPaths" ], "prConcurrentLimit": 5, "rebaseWhen": "conflicted" } } DEBUG: migrated config { "config": { "extends": [ "github>whitesource/merge-confidence:beta", "config:base" ], "postUpdateOptions": [ "gomodTidy", "gomodUpdateImportPaths" ], "prConcurrentLimit": 5, "rebaseWhen": "conflicted" } } DEBUG: Setting hostRules from config DEBUG: Found repo ignorePaths { "ignorePaths": [ "**/node_modules/**", "**/bower_components/**", "**/vendor/**", "**/examples/**", "**/__tests__/**", "**/test/**", "**/tests/**", "**/__fixtures__/**" ] } DEBUG: No vulnerability alerts found DEBUG: No vulnerability alerts found DEBUG: findIssue(Dependency Dashboard) DEBUG: No baseBranches DEBUG: extract() DEBUG: Setting current branch to main DEBUG: latest commit { "branchName": "main", "latestCommitDate": "2021-12-07T12:53:59-08:00" } DEBUG: Using file match: (^|/)tasks/[^/]+\.ya?ml$ for manager ansible DEBUG: Using file match: (^|/)requirements\.ya?ml$ for manager ansible-galaxy DEBUG: Using file match: (^|/)galaxy\.ya?ml$ for manager ansible-galaxy DEBUG: Using file match: azure.*pipelines?.*\.ya?ml$ for manager azure-pipelines DEBUG: Using file match: (^|/)batect(-bundle)?\.yml$ for manager batect DEBUG: Using file match: (^|/)batect$ for manager batect-wrapper DEBUG: Using file match: (^|/)WORKSPACE(|\.bazel)$ for manager bazel DEBUG: Using file match: \.bzl$ for manager bazel DEBUG: Using file match: (^|/)\.?bitbucket-pipelines\.ya?ml$ for manager bitbucket-pipelines DEBUG: Using file match: buildkite\.ya?ml for manager buildkite DEBUG: Using file match: \.buildkite/.+\.ya?ml$ for manager buildkite DEBUG: Using file match: (^|/)Gemfile$ for manager bundler DEBUG: Using file match: \.cake$ for manager cake DEBUG: Using file match: (^|/)Cargo.toml$ for manager cargo DEBUG: Using file match: (^|/).circleci/config.yml$ for manager circleci DEBUG: Using file match: (^|/)cloudbuild.ya?ml for manager cloudbuild DEBUG: Using file match: (^|/)Podfile$ for manager cocoapods DEBUG: Using file match: (^|/)([\w-]*)composer.json$ for manager composer DEBUG: Using file match: (^|/)deps\.edn$ for manager deps-edn DEBUG: Using file match: (^|/)docker-compose[^/]*\.ya?ml$ for manager docker-compose DEBUG: Using file match: (^|/|\.)Dockerfile$ for manager dockerfile DEBUG: Using file match: (^|/)Dockerfile\.[^/]*$ for manager dockerfile DEBUG: Using file match: (^|/).drone.yml$ for manager droneci DEBUG: Using file match: (^|/).gitmodules$ for manager git-submodules DEBUG: Using file match: ^(workflow-templates|\.github\/workflows)\/[^/]+\.ya?ml$ for manager github-actions DEBUG: Using file match: (^|\/)action\.ya?ml$ for manager github-actions DEBUG: Using file match: \.gitlab-ci\.yml$ for manager gitlabci DEBUG: Using file match: \.gitlab-ci\.yml$ for manager gitlabci-include DEBUG: Using file match: (^|/)go.mod$ for manager gomod DEBUG: Using file match: \.gradle(\.kts)?$ for manager gradle DEBUG: Using file match: (^|\/)gradle\.properties$ for manager gradle DEBUG: Using file match: (^|\/)gradle\/.+\.toml$ for manager gradle DEBUG: Using file match: \.versions\.toml$ for manager gradle DEBUG: Using file match: (^|/)gradle/wrapper/gradle-wrapper.properties$ for manager gradle-wrapper DEBUG: Using file match: (^|/)requirements\.yaml$ for manager helm-requirements DEBUG: Using file match: (^|/)values.yaml$ for manager helm-values DEBUG: Using file match: (^|/)helmfile.yaml$ for manager helmfile DEBUG: Using file match: (^|/)Chart.yaml$ for manager helmv3 DEBUG: Using file match: ^Formula/[^/]+[.]rb$ for manager homebrew DEBUG: Using file match: \.html?$ for manager html DEBUG: Using file match: (^|/)plugins\.(txt|ya?ml)$ for manager jenkins DEBUG: Using file match: (^|/)jsonnetfile.json$ for manager jsonnet-bundler DEBUG: Using file match: (^|/)kustomization\.yaml for manager kustomize DEBUG: Using file match: (^|/)project\.clj$ for manager leiningen DEBUG: Using file match: \.pom\.xml$ for manager maven DEBUG: Using file match: (^|/)pom\.xml$ for manager maven DEBUG: Using file match: (^|/)package.js$ for manager meteor DEBUG: Using file match: (^|/)mix\.exs$ for manager mix DEBUG: Using file match: (^|/).node-version$ for manager nodenv DEBUG: Using file match: (^|/)package.json$ for manager npm DEBUG: Using file match: \.(?:cs|fs|vb)proj$ for manager nuget DEBUG: Using file match: \.(?:props|targets)$ for manager nuget DEBUG: Using file match: (^|\/)dotnet-tools\.json$ for manager nuget DEBUG: Using file match: (^|\/)global\.json$ for manager nuget DEBUG: Using file match: (^|/)\.nvmrc$ for manager nvm DEBUG: Using file match: (^|/)([\w-]*)requirements\.(txt|pip)$ for manager pip_requirements DEBUG: Using file match: (^|/)setup.py$ for manager pip_setup DEBUG: Using file match: (^|/)Pipfile$ for manager pipenv DEBUG: Using file match: (^|/)pyproject\.toml$ for manager poetry DEBUG: Using file match: (^|/)\.pre-commit-config\.yaml$ for manager pre-commit DEBUG: Using file match: (^|/)pubspec\.ya?ml$ for manager pub DEBUG: Using file match: (^|/).python-version$ for manager pyenv DEBUG: Using file match: (^|/)\.ruby-version$ for manager ruby-version DEBUG: Using file match: \.sbt$ for manager sbt DEBUG: Using file match: project/[^/]*.scala$ for manager sbt DEBUG: Using file match: (^|/)setup\.cfg$ for manager setup-cfg DEBUG: Using file match: (^|/)Package\.swift for manager swift DEBUG: Using file match: \.tf$ for manager terraform DEBUG: Using file match: (^|/)\.terraform-version$ for manager terraform-version DEBUG: Using file match: (^|/)terragrunt\.hcl$ for manager terragrunt DEBUG: Using file match: (^|/)\.terragrunt-version$ for manager terragrunt-version DEBUG: Using file match: ^.travis.yml$ for manager travis DEBUG: Matched 1 file(s) for manager github-actions: .github/workflows/go.yml DEBUG: Matched 2 file(s) for manager gomod: a/go.mod, b/go.mod DEBUG: No multi-line match: go.uber.org/atomic v1.6.0 // indirect DEBUG: No multi-line match: go.uber.org/multierr v1.5.0 // indirect DEBUG: No multi-line match: go.uber.org/atomic v1.6.0 // indirect DEBUG: No multi-line match: go.uber.org/multierr v1.5.0 // indirect DEBUG: No multi-line match: go.uber.org/zap v1.16.0 // indirect DEBUG: Found github-actions package files DEBUG: Found gomod package files DEBUG: Found 3 package file(s) INFO: Dependency extraction complete { "baseBranch": "main", "stats": { "managers": { "github-actions": { "fileCount": 1, "depCount": 2 }, "gomod": { "fileCount": 2, "depCount": 2 } }, "total": { "fileCount": 3, "depCount": 4 } } } DEBUG: Goproxy error: trying next URL provided with GOPROXY { "err": { "name": "HTTPError", "code": "ERR_NON_2XX_3XX_RESPONSE", "timings": { "start": 1638910494479, "socket": 1638910494512, "lookup": 1638910494512, "connect": 1638910494523, "secureConnect": 1638910494542, "upload": 1638910494542, "response": 1638910494566, "end": 1638910494567, "phases": { "wait": 33, "dns": 0, "tcp": 11, "tls": 19, "request": 0, "firstByte": 24, "download": 1, "total": 88 } }, "message": "Response code 410 (Gone)", "stack": "HTTPError: Response code 410 (Gone)\n at Request. (/home/ubuntu/renovateapp/node_modules/got/dist/source/as-promise/index.js:117:42)\n at runMicrotasks ()\n at processTicksAndRejections (internal/process/task_queues.js:95:5)", "options": { "headers": { "user-agent": "Renovate Bot (GitHub App 2740)", "accept-encoding": "gzip, deflate, br" }, "url": "https://proxy.golang.org/github.com/eriksw/renovate-peer-go-mod-indirect/a/@v/list", "hostType": "go", "username": "", "password": "", "method": "GET", "http2": false }, "response": { "statusCode": 410, "statusMessage": "Gone", "body": "not found: module github.com/eriksw/renovate-peer-go-mod-indirect/a: git ls-remote -q origin in /tmp/gopath/pkg/mod/cache/vcs/c91760e9dcba81fb66c7001ea31c860bab6182581792a0d1aa9ffd8b350085b5: exit status 128:\n\tfatal: could not read Username for 'https://github.com': terminal prompts disabled\nConfirm the import path was entered correctly.\nIf this is a private repository, see https://golang.org/doc/faq#git_https for additional information.", "headers": { "access-control-allow-origin": "*", "cache-control": "public, max-age=60", "content-type": "text/plain; charset=UTF-8", "date": "Tue, 07 Dec 2021 20:54:54 GMT", "expires": "Tue, 07 Dec 2021 20:55:54 GMT", "x-content-type-options": "nosniff", "x-frame-options": "SAMEORIGIN", "x-xss-protection": "0", "content-length": "441", "alt-svc": "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"", "connection": "close" }, "httpVersion": "1.1" } } } DEBUG: Found no results from datasource that look like a version (github.com/eriksw/renovate-peer-go-mod-indirect/a)(dependency="github.com/eriksw/renovate-peer-go-mod-indirect/a") { "result": { "sourceUrl": "https://github.com/eriksw/renovate-peer-go-mod-indirect", "releases": [] } } DEBUG: GitHub 404 { "url": "https://api.github.com/repos/eriksw/renovate-peer-go-mod-indirect/git/refs/tags/v0.0.0-00010101000000-000000000000" } DEBUG: Error getting tag commit from GitHub repo { "githubRepo": "eriksw/renovate-peer-go-mod-indirect", "err": { "name": "HTTPError", "code": "ERR_NON_2XX_3XX_RESPONSE", "timings": { "start": 1638910495000, "socket": 1638910495000, "lookup": 1638910495000, "connect": 1638910495010, "secureConnect": 1638910495021, "upload": 1638910495021, "response": 1638910495176, "end": 1638910495177, "phases": { "wait": 0, "dns": 0, "tcp": 10, "tls": 11, "request": 0, "firstByte": 155, "download": 1, "total": 177 } }, "message": "Response code 404 (Not Found)", "stack": "HTTPError: Response code 404 (Not Found)\n at Request. (/home/ubuntu/renovateapp/node_modules/got/dist/source/as-promise/index.js:117:42)\n at runMicrotasks ()\n at processTicksAndRejections (internal/process/task_queues.js:95:5)", "options": { "headers": { "user-agent": "Renovate Bot (GitHub App 2740)", "accept": "application/vnd.github.machine-man-preview+json", "authorization": "***********", "accept-encoding": "gzip, deflate, br" }, "url": "https://api.github.com/repos/eriksw/renovate-peer-go-mod-indirect/git/refs/tags/v0.0.0-00010101000000-000000000000", "hostType": "github-tags", "username": "", "password": "", "method": "GET", "http2": false }, "response": { "statusCode": 404, "statusMessage": "Not Found", "body": { "message": "Not Found", "documentation_url": "https://docs.github.com/rest" }, "headers": { "server": "GitHub.com", "date": "Tue, 07 Dec 2021 20:54:55 GMT", "content-type": "application/json; charset=utf-8", "transfer-encoding": "chunked", "x-poll-interval": "300", "x-github-media-type": "github.v3; param=machine-man-preview; format=json", "x-ratelimit-limit": "5000", "x-ratelimit-remaining": "4995", "x-ratelimit-reset": "1638910672", "x-ratelimit-used": "5", "x-ratelimit-resource": "core", "access-control-expose-headers": "ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset", "access-control-allow-origin": "*", "strict-transport-security": "max-age=31536000; includeSubdomains; preload", "x-frame-options": "deny", "x-content-type-options": "nosniff", "x-xss-protection": "0", "referrer-policy": "origin-when-cross-origin, strict-origin-when-cross-origin", "content-security-policy": "default-src 'none'", "vary": "Accept-Encoding, Accept, X-Requested-With", "content-encoding": "gzip", "x-github-request-id": "9608:73D6:1192E85:122D322:61AFCA1F", "connection": "close" }, "httpVersion": "1.1" } } } DEBUG: Package releases lookups complete { "baseBranch": "main" } DEBUG: branchifyUpgrades DEBUG: 1 flattened updates found: go.uber.org/zap DEBUG: Returning 1 branch(es) DEBUG: Fetching changelog: https://github.com/uber-go/zap (v1.16.0 -> v1.19.1) DEBUG: config.repoIsOnboarded=true DEBUG: packageFiles with updates { "config": { "github-actions": [ { "packageFile": ".github/workflows/go.yml", "deps": [ { "depName": "actions/checkout", "commitMessageTopic": "{{{depName}}} action", "datasource": "github-tags", "versioning": "docker", "depType": "action", "replaceString": "actions/checkout@v2", "autoReplaceStringTemplate": "{{depName}}@{{#if newDigest}}{{newDigest}}{{#if newValue}} # renovate: tag={{newValue}}{{/if}}{{/if}}{{#unless newDigest}}{{newValue}}{{/unless}}", "currentValue": "v2", "depIndex": 0, "updates": [], "warnings": [], "sourceUrl": "https://github.com/actions/checkout", "currentVersion": "v2", "fixedVersion": "v2" }, { "depName": "actions/setup-go", "commitMessageTopic": "{{{depName}}} action", "datasource": "github-tags", "versioning": "docker", "depType": "action", "replaceString": "actions/setup-go@v2", "autoReplaceStringTemplate": "{{depName}}@{{#if newDigest}}{{newDigest}}{{#if newValue}} # renovate: tag={{newValue}}{{/if}}{{/if}}{{#unless newDigest}}{{newValue}}{{/unless}}", "currentValue": "v2", "depIndex": 1, "updates": [], "warnings": [], "sourceUrl": "https://github.com/actions/setup-go", "currentVersion": "v2", "fixedVersion": "v2" } ] } ], "gomod": [ { "packageFile": "a/go.mod", "constraints": { "go": "^1.17" }, "deps": [ { "managerData": { "lineNumber": 4 }, "depName": "go.uber.org/zap", "depType": "require", "currentValue": "v1.16.0", "datasource": "go", "depIndex": 0, "updates": [ { "bucket": "non-major", "newVersion": "v1.19.1", "newValue": "v1.19.1", "releaseTimestamp": "2021-09-08T21:32:28.000Z", "newMajor": 1, "newMinor": 19, "updateType": "minor", "branchName": "renovate/go.uber.org-zap-1.x" } ], "warnings": [], "versioning": "semver", "sourceUrl": "https://github.com/uber-go/zap", "currentVersion": "v1.16.0", "isSingleVersion": true, "fixedVersion": "v1.16.0" } ] }, { "packageFile": "b/go.mod", "constraints": { "go": "^1.17" }, "deps": [ { "managerData": { "lineNumber": 4 }, "depName": "github.com/eriksw/renovate-peer-go-mod-indirect/a", "depType": "require", "currentValue": "v0.0.0-00010101000000-000000000000", "datasource": "go", "currentDigest": "000000000000", "digestOneAndOnly": true, "depIndex": 0, "updates": [], "warnings": [], "versioning": "semver", "sourceUrl": "https://github.com/eriksw/renovate-peer-go-mod-indirect", "currentVersion": "v0.0.0-00010101000000-000000000000", "fixedVersion": "v0.0.0-00010101000000-000000000000" } ] } ] } } DEBUG: processRepo() DEBUG: Processing 1 branch: renovate/go.uber.org-zap-1.x DEBUG: Calculating hourly PRs remaining DEBUG: Retrieving PR list DEBUG: Retrieved 0 Pull Requests DEBUG: currentHourStart=2021-12-07T20:00:00.000+00:00 DEBUG: PR hourly limit remaining: 2 DEBUG: Calculating prConcurrentLimit (5) DEBUG: getBranchPr(renovate/go.uber.org-zap-1.x) DEBUG: findPr(renovate/go.uber.org-zap-1.x, undefined, open) DEBUG: findPr(renovate/go.uber.org-zap-1.x, undefined, closed) DEBUG: 0 PRs are currently open DEBUG: PR concurrent limit remaining: 5 DEBUG: Calculated maximum PRs remaining this run { "prsRemaining": 2 } DEBUG: PullRequests limit = 2 DEBUG: Calculating hourly PRs remaining DEBUG: currentHourStart=2021-12-07T20:00:00.000+00:00 DEBUG: PR hourly limit remaining: 2 DEBUG: Calculating branchConcurrentLimit (5) DEBUG: 0 already existing branches found: DEBUG: Branch concurrent limit remaining: 5 DEBUG: Calculated maximum branches remaining this run { "branchesRemaining": 2 } DEBUG: Branches limit = 2 DEBUG: Setting current branch to main(branch="renovate/go.uber.org-zap-1.x") DEBUG: latest commit(branch="renovate/go.uber.org-zap-1.x") { "branchName": "main", "latestCommitDate": "2021-12-07T12:53:59-08:00" } DEBUG: getBranchPr(renovate/go.uber.org-zap-1.x)(branch="renovate/go.uber.org-zap-1.x") DEBUG: findPr(renovate/go.uber.org-zap-1.x, undefined, open)(branch="renovate/go.uber.org-zap-1.x") DEBUG: findPr(renovate/go.uber.org-zap-1.x, undefined, closed)(branch="renovate/go.uber.org-zap-1.x") DEBUG: branchExists=false(branch="renovate/go.uber.org-zap-1.x") DEBUG: dependencyDashboardCheck=undefined(branch="renovate/go.uber.org-zap-1.x") DEBUG: recreateClosed is false(branch="renovate/go.uber.org-zap-1.x") DEBUG: findPr(renovate/go.uber.org-zap-1.x, Update module go.uber.org/zap to v1.19.1, !open)(branch="renovate/go.uber.org-zap-1.x") DEBUG: prAlreadyExisted=false(branch="renovate/go.uber.org-zap-1.x") DEBUG: Checking schedule(at any time, null)(branch="renovate/go.uber.org-zap-1.x") DEBUG: No schedule defined(branch="renovate/go.uber.org-zap-1.x") DEBUG: Branch needs creating(branch="renovate/go.uber.org-zap-1.x") DEBUG: Using reuseExistingBranch: false(branch="renovate/go.uber.org-zap-1.x") DEBUG: manager.getUpdatedPackageFiles() reuseExistinbranch=false(branch="renovate/go.uber.org-zap-1.x") DEBUG: gomod.updateDependency: v1.19.1(branch="renovate/go.uber.org-zap-1.x") DEBUG: Updating go.uber.org/zap in a/go.mod(branch="renovate/go.uber.org-zap-1.x") DEBUG: gomod.updateArtifacts(a/go.mod)(branch="renovate/go.uber.org-zap-1.x") DEBUG: Adding Git authentication for Go Module retrieval for https://api.github.com/ using token auth.(branch="renovate/go.uber.org-zap-1.x") DEBUG: go get command included(branch="renovate/go.uber.org-zap-1.x") { "cmd": "go", "args": "get -d ./..." } DEBUG: go mod tidy command included(branch="renovate/go.uber.org-zap-1.x") { "cmd": "go", "args": "mod tidy" } DEBUG: additional go mod tidy command included(branch="renovate/go.uber.org-zap-1.x") { "cmd": "go", "args": "mod tidy" } DEBUG: Using docker to execute(branch="renovate/go.uber.org-zap-1.x") DEBUG: Found version constraint - checking for a compatible image to use(branch="renovate/go.uber.org-zap-1.x") { "depName": "docker.io/renovate/go", "scheme": "npm", "constraint": "^1.17" } DEBUG: getLabels(https://index.docker.io, renovate/go, latest)(branch="renovate/go.uber.org-zap-1.x") DEBUG: Found compatible image version(branch="renovate/go.uber.org-zap-1.x") { "depName": "docker.io/renovate/go", "scheme": "npm", "constraint": "^1.17", "version": "1.17.4" } DEBUG: Resolved tag constraint(branch="renovate/go.uber.org-zap-1.x") { "image": "docker.io/renovate/go", "tagConstraint": "^1.17", "tagVersioning": "npm", "tag": "1.17.4" } DEBUG: Fetching Docker image: docker.io/renovate/go:1.17.4(branch="renovate/go.uber.org-zap-1.x") DEBUG: Finished fetching Docker image(branch="renovate/go.uber.org-zap-1.x") DEBUG: Executing command(branch="renovate/go.uber.org-zap-1.x") { "command": [ "docker run --rm --name=renovate_go --label=renovate_child -v \"/mnt/renovate/gh/eriksw/renovate-peer-go-mod-indirect\":\"/mnt/renovate/gh/eriksw/renovate-peer-go-mod-indirect\" -v \"/tmp/renovate-cache\":\"/tmp/renovate-cache\" -e GOPATH -e GOPROXY -e GOFLAGS -e CGO_ENABLED -e GIT_CONFIG_KEY_0 -e GIT_CONFIG_VALUE_0 -e GIT_CONFIG_COUNT -e GIT_CONFIG_KEY_1 -e GIT_CONFIG_VALUE_1 -w \"/mnt/renovate/gh/eriksw/renovate-peer-go-mod-indirect/a\" docker.io/renovate/go:1.17.4 bash -l -c \"go get -d ./... && go mod tidy && go mod tidy\"" ] } DEBUG: exec completed(branch="renovate/go.uber.org-zap-1.x") { "cmd": "docker run --rm --name=renovate_go --label=renovate_child -v \"/mnt/renovate/gh/eriksw/renovate-peer-go-mod-indirect\":\"/mnt/renovate/gh/eriksw/renovate-peer-go-mod-indirect\" -v \"/tmp/renovate-cache\":\"/tmp/renovate-cache\" -e GOPATH -e GOPROXY -e GOFLAGS -e CGO_ENABLED -e GIT_CONFIG_KEY_0 -e GIT_CONFIG_VALUE_0 -e GIT_CONFIG_COUNT -e GIT_CONFIG_KEY_1 -e GIT_CONFIG_VALUE_1 -w \"/mnt/renovate/gh/eriksw/renovate-peer-go-mod-indirect/a\" docker.io/renovate/go:1.17.4 bash -l -c \"go get -d ./... && go mod tidy && go mod tidy\"", "durationMs": 7327, "stdout": "", "stderr": "go: downloading go.uber.org/zap v1.19.1\ngo: downloading go.uber.org/atomic v1.7.0\ngo: downloading go.uber.org/multierr v1.6.0\ngo get: upgraded go.uber.org/atomic v1.6.0 => v1.7.0\ngo get: upgraded go.uber.org/multierr v1.5.0 => v1.6.0\ngo: downloading github.com/pkg/errors v0.8.1\ngo: downloading github.com/stretchr/testify v1.7.0\ngo: downloading go.uber.org/goleak v1.1.11-0.20210813005559-691160354723\ngo: downloading github.com/benbjohnson/clock v1.1.0\ngo: downloading gopkg.in/yaml.v2 v2.2.8\ngo: downloading github.com/davecgh/go-spew v1.1.1\ngo: downloading github.com/pmezard/go-difflib v1.0.0\ngo: downloading gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b\n" } DEBUG: Returning updated go.sum(branch="renovate/go.uber.org-zap-1.x") DEBUG: Found updated go.mod after go.sum update(branch="renovate/go.uber.org-zap-1.x") DEBUG: Updated 1 package files(branch="renovate/go.uber.org-zap-1.x") DEBUG: Updated 2 lock files(branch="renovate/go.uber.org-zap-1.x") { "updatedArtifacts": [ "a/go.sum", "a/go.mod" ] } DEBUG: 2 file(s) to commit(branch="renovate/go.uber.org-zap-1.x") DEBUG: Committing files to branch renovate/go.uber.org-zap-1.x(branch="renovate/go.uber.org-zap-1.x") DEBUG: Setting git author name(branch="renovate/go.uber.org-zap-1.x") { "gitAuthorName": "Renovate Bot" } DEBUG: Setting git author email(branch="renovate/go.uber.org-zap-1.x") { "gitAuthorEmail": "bot@renovateapp.com" } DEBUG: git commit(branch="renovate/go.uber.org-zap-1.x") { "deletedFiles": [], "ignoredFiles": [], "result": { "author": null, "branch": "renovate/go.uber.org-zap-1.x", "commit": "8654953", "root": false, "summary": { "changes": 2, "insertions": 35, "deletions": 0 } } } DEBUG: git push(branch="renovate/go.uber.org-zap-1.x") { "result": { "pushed": [ { "deleted": false, "tag": false, "branch": true, "new": true, "alreadyUpdated": false, "local": "refs/heads/renovate/go.uber.org-zap-1.x", "remote": "refs/heads/renovate/go.uber.org-zap-1.x" } ], "branch": { "local": "renovate/go.uber.org-zap-1.x", "remote": "renovate/go.uber.org-zap-1.x", "remoteName": "origin" }, "ref": { "local": "refs/remotes/origin/renovate/go.uber.org-zap-1.x" }, "remoteMessages": { "all": [ "Create a pull request for 'renovate/go.uber.org-zap-1.x' on GitHub by visiting:", "https://github.com/eriksw/renovate-peer-go-mod-indirect/pull/new/renovate/go.uber.org-zap-1.x" ], "pullRequestUrl": "https://github.com/eriksw/renovate-peer-go-mod-indirect/pull/new/renovate/go.uber.org-zap-1.x" } } } INFO: Branch created(branch="renovate/go.uber.org-zap-1.x") { "commitSha": "8654953" } DEBUG: Ensuring PR(branch="renovate/go.uber.org-zap-1.x") DEBUG: There are 0 errors and 0 warnings(branch="renovate/go.uber.org-zap-1.x") DEBUG: getBranchPr(renovate/go.uber.org-zap-1.x)(branch="renovate/go.uber.org-zap-1.x") DEBUG: findPr(renovate/go.uber.org-zap-1.x, undefined, open)(branch="renovate/go.uber.org-zap-1.x") DEBUG: findPr(renovate/go.uber.org-zap-1.x, undefined, closed)(branch="renovate/go.uber.org-zap-1.x") DEBUG: Creating PR(branch="renovate/go.uber.org-zap-1.x") { "prTitle": "Update module go.uber.org/zap to v1.19.1" } DEBUG: Creating PR(branch="renovate/go.uber.org-zap-1.x") { "title": "Update module go.uber.org/zap to v1.19.1", "head": "eriksw:renovate/go.uber.org-zap-1.x", "base": "main", "draft": false } DEBUG: PR created(branch="renovate/go.uber.org-zap-1.x") { "pr": 1, "draft": false } DEBUG: Adding labels '' to #1(branch="renovate/go.uber.org-zap-1.x") INFO: PR created(branch="renovate/go.uber.org-zap-1.x") { "pr": 1, "prTitle": "Update module go.uber.org/zap to v1.19.1" } DEBUG: addAssigneesReviewers(pr=1)(branch="renovate/go.uber.org-zap-1.x") DEBUG: Created Pull Request #1(branch="renovate/go.uber.org-zap-1.x") DEBUG: PR is not configured for automerge(branch="renovate/go.uber.org-zap-1.x") DEBUG: getBranchPr(renovate/go.uber.org-zap-1.x) DEBUG: findPr(renovate/go.uber.org-zap-1.x, undefined, open) DEBUG: Found PR #1 DEBUG: Returning from graphql open PR list DEBUG: Ensuring Dependency Dashboard DEBUG: ensureIssue(Dependency Dashboard) INFO: Issue created DEBUG: Removing any stale branches DEBUG: config.repoIsOnboarded=true DEBUG: Branch lists { "branchList": [ "renovate/go.uber.org-zap-1.x" ], "renovateBranches": [ "renovate/go.uber.org-zap-1.x" ] } DEBUG: remainingBranches= DEBUG: No branches to clean up DEBUG: Retrieving issueList DEBUG: Retrieved 1 issues DEBUG: Repository timing splits (milliseconds) { "splits": { "init": 3531, "extract": 918, "lookup": 902, "update": 12877 }, "total": 19493 } DEBUG: http statistics { "urls": { "https://api.github.com/graphql (POST)": 5, "https://api.github.com/repos/eriksw/renovate-peer-go-mod-indirect/issues (POST)": 1, "https://api.github.com/repos/eriksw/renovate-peer-go-mod-indirect/pulls (GET)": 1, "https://api.github.com/repos/eriksw/renovate-peer-go-mod-indirect/pulls (POST)": 1, "https://api.github.com/repos/eriksw/renovate-peer-go-mod-indirect/releases (GET)": 1, "https://api.github.com/repos/eriksw/renovate-peer-go-mod-indirect/tags (GET)": 1, "https://api.github.com/repos/whitesource/merge-confidence/contents/beta.json (GET)": 1 }, "hostStats": { "api.github.com": { "requestCount": 11, "requestAvgMs": 343, "queueAvgMs": 0 } }, "totalRequests": 11 } INFO: Repository finished { "durationMs": 19493 }

If you have any ideas on how this should be implemented, please tell us here.

Renovate needs to be aware of relative path replace directives amongst go modules and ensure update/tidy actions propagate following those relationships.

Is this a feature you are interested in implementing yourself?

No

mowies commented 2 years ago

We have the same issue in https://github.com/keptn/keptn :( Renovate PR: https://github.com/keptn/keptn/pull/6377 Update PR to fix the dependencies: https://github.com/keptn/keptn/pull/6381

oschwald commented 2 years ago

We would also like to see this feature. It looks like Renovate used to update indirect dependencies by default, but that was removed in https://github.com/renovatebot/renovate/pull/4650. I don't fully understand the reasoning for why this was done, but it seems like updating indirect dependencies should be an option at the very least.

woohgit commented 2 years ago

With golang 1.17 and above the indirect dependencies should be updated too, otherwise it'll fail with:

12:51:56  === Errors
12:51:56  go: updates to go.mod needed; to update it:
12:51:56    go mod tidy

I hope someone will pick this up and make the necessary changes to make this happen.

rarkins commented 2 years ago

@woohgit are you able to create a minimal reproduction for that type of failure?

woohgit commented 2 years ago

@rarkins

Yes probably I'll need to create a module (shared) which uses aws-go-sdk for example. Then create 2 more modules that uses the shared module and for go 1.17+ you have to have the indirect deps for the shared for all other modules too, even if they don't directly use the aws-sdk-go.

If you wan't I can create a minimal app with the proper setup to reproduce the issue.

rarkins commented 2 years ago

@woohgit thanks, that would be great. I would really like to get this issue solved, and this might be the root cause

woohgit commented 2 years ago

@rarkins On it. I'm waiting for the first dependency PR to be created by renovatebot and I can show it to you then.

woohgit commented 2 years ago

@rarkins And it happened.

There you go: https://github.com/woohgit/renovate-12999/pull/3/files

This repo consists of a shared module (which depends on aws-sdk-go) and 2 other consumer modules which depend on the shared. For go 1.17+ the go.sum and go.mod has to contain indirect entries for the shared module but that's not updated with the PR. So if you want to build the consumer1 or consumer2 using this PR you'll get the error:

❯ go build .
go: updates to go.mod needed; to update it:
    go mod tidy
❯ pwd
/home/wooh/repos/renovate-12999/consumer1

It think at least it should be a config option to update it everywhere (and run go mod tidy in all relevant directories) and maybe set it to true when the group:monorepo is enabled and the project is golang.

woohgit commented 2 years ago

@rarkins Any comment / update / more help needed etc?

rarkins commented 2 years ago

@woohgit thanks for your updates to the issue. Previously I think people had been reporting "I need to run go mod tidy in the same directory one more time". In your case though it sounds like you'd need Renovate to build up a kind of "dependency tree" of each module to know which depends on which and then run "go mod tidy" on each directory which depends on the one which just got updated?

woohgit commented 2 years ago

Actually everyone is reporting the same as me:

Look at the original description of this issue: https://github.com/eriksw/renovate-peer-go-mod-indirect/pull/1

This is exactly the same setup as I presented to you.

It's not that we need to run go mod tidy multiple times in the same directory. It's that the //indirect dependencies are not updated, but they should be because from go 1.17+ you need to keep them sync too.

rarkins commented 2 years ago

Actually we have a few Go tidying issues which seem to be bugs, some pre-dating 1.17. I thought we may have found the root cause but it seems not.

As discussed above, this will require Renovate to build a dependency tree of related go.mod files, which wasn't necessary before.

Classification as a feature request instead of bug was correct in this case.

woohgit commented 2 years ago

@rarkins It's fine! I know it's getting a bit more complex with 1.17. I hope you guys can figure it out quickly how to do it :)

rarkins commented 2 years ago

Whoever implements this will need to decide whether the go.mod dependency tree should be built during the extract phase (extractAllPackageFiles) or instead can be determined dynamically during this artifacts updating stage.

viceice commented 2 years ago

there is currently ongoing work for nuget, where we need to detect dependent projects. i think we can refactor and reuse some of the code from that PR so solve this issue afterwards.

malt3 commented 1 year ago

I believe the nuget feature was implemented here: https://github.com/renovatebot/renovate/pull/15698 maybe this unblocks the feature request in this PR?

katexochen commented 1 year ago

A simple solution for this issue might be to provide an option that just runs go mod tidy for all modules in the repository, instead of evaluating the replace statements and creating a dependency tree. Currently, the postUpdateOption gomodTidy seem to only update modules that were changed by renovate.

viceice commented 1 year ago

@malt3 nuget manager has nothing to do with golang manager

gnuletik commented 1 year ago

I'm open to work on a fix for this!

The easiest way I see to implement this is to run go mod tidy inside each go modules.

Does that would save everyone use case?

I'm currently running the following command locally to patch renovate PRs:

for i in $(find . -name go.mod); do cd $(dirname "$i") && go mod tidy && cd -; done && git commit -am "chore: go mod tidy in every go modules" && git push

However, it is a pain to do that inside every PR.

I was thinking about adding an option to renovate to implement a similar behavior, for example a postUpdateOptions: ["goModTidyAll"]. But I can't find where I could implement it.

go mod tidy is currently runs individually in each module here:

https://github.com/renovatebot/renovate/blob/9669ba320ffa426fbf6ce0965653eb71f132f505/lib/modules/manager/gomod/artifacts.ts#L307-L337

Is there a package where we could implement "repository wide" operations?

rarkins commented 1 year ago

goModTidyAll would be a good way to opt into it.

Check out https://github.com/renovatebot/renovate/blob/main/lib/modules/manager/nuget/package-tree.ts#L16 for an example of how nuget does it

eriksw commented 1 year ago

I would love to see a goModTidyAll option. That would indeed solve my use case!

cevich commented 1 year ago

sigh this seems to be impacting golang vulnerability related updates, if they happen to be against indirect dependencies (disabled by default). Dependabot is catching these, but renovate is not. Perhaps a workaround could be (untested, no idea if this would work):

"vulnerabilityAlerts": {
  "packageRules": [
    {
      "matchManagers": ["gomod"],
      "matchDepTypes": ["indirect"],
      "enabled": true
    }
  ]
}

Obviously that wouldn't do anything WRT a goModTidyAll feature, but at least I might have a PR open to do that manually?

cevich commented 1 year ago

Clarification: The intent of the above is - It's better to get a broken proposed update PR than no update PR at all (for a vulnerability) - assuming Dependabot is turned off.

Edit: Opened https://github.com/renovatebot/renovate/discussions/22599

rarkins commented 1 year ago

@cevich your requirements don't appear directly related to this topic, so please start a separate Discussion with your own reproduction repo

nyetwurk commented 1 year ago

Almost all of the indirect major updates from renovate cause problems, mostly due to broken semver repos.

https://github.com/asaskevich/govalidator/issues/491

github.com/asaskevich/govalidator@v11.0.1: invalid version: module contains a go.mod file, so module path must match major version ("github.com/asaskevich/govalidator/v11"

https://github.com/twmb/franz-go/issues/494

require github.com/twmb/franz-go/pkg/kmsg: version "v2.0.2" invalid: should be v0 or v1, not v2

I don't think renovate should be updating any indirects at all; if go get -u ./... and go mod tidy miss them, that's a go mod bug.

Please, restore the original behavior of not touching // indirect dependencies, or at least not trying to do major updates on them. There is absolutely NO reason why the major version of an indirect should get bumped, period. That is up to the actual mod that directly has that dependency.

rarkins commented 1 year ago

@nyetwurk please start a new discussion and provide a minimal reproduction repo with such a failing case so that we can fix it

Also, comments like "The whole thing is hopelessly broken" are unwelcome here - please refrain from such negativity in this repo

nyetwurk commented 1 year ago

@nyetwurk please start a new discussion and provide a minimal reproduction repo with such a failing case so that we can fix it

Also, comments like "The whole thing is hopelessly broken" are unwelcome here - please refrain from such negativity in this repo

I don't understand the rationale of bumping a major version of the indirect. If the included module is not compatible with the major version (for example, because the paths are all wrong), there is no amount of work in the local repo that can fix it. The major version must be updated in the actual mod that includes it directly, if only because of the path changes.

nyetwurk commented 1 year ago

Note that https://github.com/asaskevich/govalidator is abandoned, so nobody is going to fix it, and in https://github.com/twmb/franz-go/issues/494 it is clear that repo will also never be fixed.

The former is being imported due to github.com/go-ozzo/ozzo-validation/v4/is, perhaps I can build a minimal repo around that that demonstrates the issue.

https://github.com/renovatebot/renovate/issues/4586 seems to be taking the correct approach; I'm not sure why after that was fixed, renovate is trying to update indirect dependencies - or even worse, updating to different major versions.

rarkins commented 1 year ago

To clarify: I'm asking you to provide a minimal reproduction repo showing Renovate (incorrectly) updating an indirect dependency with a new major version. If that's not the case, I don't understand what it is you're objecting to. Neither of the repos you link to use Renovate so I'm also not sure why you're referring to them either.

rarkins commented 1 year ago

I've also asked you to start a new discussion, please do that and discontinue in this issue.

eiffel-fl commented 3 months ago

Hi!

We are interested in using renovate as we thought it would handle this case. I tested it on my fork and sadly it does not handle running go mod tidy in examples/: https://github.com/eiffel-fl/inspektor-gadget/actions/runs/9516190108/job/26231972803#step:6:308

For the reference, here is a schematic view of my repository:

$ find . -name go.mod                                                        main % u=
./tools/eks-cleanup/go.mod
./tools/testjson2md/go.mod
./tools/dnstester/go.mod
./go.mod
./examples/go.mod
$ cat examples/go.mod
module examples

go 1.22.2

require (
        github.com/cilium/ebpf v0.15.0
        github.com/inspektor-gadget/inspektor-gadget v0.28.1
...
// use current code of inspektor-gadget
replace github.com/inspektor-gadget/inspektor-gadget => ../

From my understanding, the goModTidyAll was only a proposition and was not implemented so far. I am not fluent in typescript, so can you please share more information on a potential implementation of such an option? Note that, I prefer to be 100% honest on this, I cannot guarantee you I would work on this feature, as I rather lean toward having only one "main" go.mod in my repository.

Best regards.

rarkins commented 3 months ago

@eiffel-fl please start a separate discussion, and create a reproduction repo to demonstrate