search

renovatebot / renovate

Home of the Renovate CLI: Cross-platform Dependency Automation by Mend.io
https://mend.io/renovate
GNU Affero General Public License v3.0
17.62k stars 2.32k forks source link

GitHub: External host error causing abort blocking all updates #16343

Closed DanySK closed 2 years ago

DanySK commented 2 years ago

How are you running Renovate?

Mend Renovate hosted app on github.com

If you're self-hosting Renovate, tell us what version of Renovate you run.

No response

Please select which platform you are using if self-hosting.

No response

If you're self-hosting Renovate, tell us what version of the platform you run.

No response

Was this something which used to work for you, and then stopped?

It used to work, and then stopped

Describe the bug

For some reason that I do not understand, renovate began failing when fetching a repo update to AlchemistSimulator/Alchemist. Apparently, GitHub is replying with a Bad Gateway, and this error blocks every update, not just the ones from the simulator.

I think that:

  1. We should investigate why there is this 502
  2. Renovate should continue with the other updates, and warn clearly on the dashboard.

The issue is happening at this repository: https://github.com/DanySK/publish-on-central

Relevant debug logs

Logs ``` DEBUG: GitHub failure: 5xx { "err": { "name": "HTTPError", "code": "ERR_NON_2XX_3XX_RESPONSE", "timings": { "start": 1656601652604, "socket": 1656601653125, "lookup": 1656601653607, "connect": 1656601653914, "secureConnect": 1656601654103, "upload": 1656601654119, "response": 1656601664627, "end": 1656601664627, "phases": { "wait": 521, "dns": 482, "tcp": 307, "tls": 189, "request": 16, "firstByte": 10508, "download": 0, "total": 12023 } }, "message": "Response code 502 (Bad Gateway)", "stack": "HTTPError: Response code 502 (Bad Gateway)\n at Request. (/home/ubuntu/renovateapp/node_modules/got/dist/source/as-promise/index.js:118:42)\n at runMicrotasks ()\n at processTicksAndRejections (internal/process/task_queues.js:95:5)", "options": { "headers": { "user-agent": "Renovate Bot (GitHub App 2740)", "accept": "application/vnd.github.machine-man-preview+json", "authorization": "***********", "content-type": "application/json", "content-length": "875", "accept-encoding": "gzip, deflate, br" }, "url": "https://api.github.com/graphql", "hostType": "github-tags", "username": "", "password": "", "method": "POST", "http2": false }, "response": { "statusCode": 502, "statusMessage": "Bad Gateway", "body": { "data": null, "errors": [ { "message": "Something went wrong while executing your query. This may be the result of a timeout, or it could be a GitHub bug. Please include `D3AA:3140:11252F0:12073D0:62BDBC35` when reporting this issue." } ] }, "headers": { "server": "GitHub.com", "date": "Thu, 30 Jun 2022 15:07:44 GMT", "content-type": "application/json", "transfer-encoding": "chunked", "access-control-expose-headers": "ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset", "access-control-allow-origin": "*", "vary": "Accept-Encoding, Accept, X-Requested-With", "x-github-request-id": "D3AA:3140:11252F0:12073D0:62BDBC35", "connection": "close" }, "httpVersion": "1.1" } } } WARN: Host error { "hostType": "github-tags", "packageName": "AlchemistSimulator/Alchemist", "err": { "name": "HTTPError", "code": "ERR_NON_2XX_3XX_RESPONSE", "timings": { "start": 1656601652604, "socket": 1656601653125, "lookup": 1656601653607, "connect": 1656601653914, "secureConnect": 1656601654103, "upload": 1656601654119, "response": 1656601664627, "end": 1656601664627, "phases": { "wait": 521, "dns": 482, "tcp": 307, "tls": 189, "request": 16, "firstByte": 10508, "download": 0, "total": 12023 } }, "message": "Response code 502 (Bad Gateway)", "stack": "HTTPError: Response code 502 (Bad Gateway)\n at Request. (/home/ubuntu/renovateapp/node_modules/got/dist/source/as-promise/index.js:118:42)\n at runMicrotasks ()\n at processTicksAndRejections (internal/process/task_queues.js:95:5)", "options": { "headers": { "user-agent": "Renovate Bot (GitHub App 2740)", "accept": "application/vnd.github.machine-man-preview+json", "authorization": "***********", "content-type": "application/json", "content-length": "875", "accept-encoding": "gzip, deflate, br" }, "url": "https://api.github.com/graphql", "hostType": "github-tags", "username": "", "password": "", "method": "POST", "http2": false }, "response": { "statusCode": 502, "statusMessage": "Bad Gateway", "body": { "data": null, "errors": [ { "message": "Something went wrong while executing your query. This may be the result of a timeout, or it could be a GitHub bug. Please include `D3AA:3140:11252F0:12073D0:62BDBC35` when reporting this issue." } ] }, "headers": { "server": "GitHub.com", "date": "Thu, 30 Jun 2022 15:07:44 GMT", "content-type": "application/json", "transfer-encoding": "chunked", "access-control-expose-headers": "ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset", "access-control-allow-origin": "*", "vary": "Accept-Encoding, Accept, X-Requested-With", "x-github-request-id": "D3AA:3140:11252F0:12073D0:62BDBC35", "connection": "close" }, "httpVersion": "1.1" } } } INFO: External host error causing abort - skipping ```

Have you created a minimal reproduction repository?

No reproduction, but I have linked to a public repo where it occurs

rarkins commented 2 years ago

Disagree. Renovate should not just continue when it gets 502s from GitHub. Doing so would mean noisy flapping of PRs any time GitHub has problems.

DanySK commented 2 years ago

Uhmm... why should the PRs flap? In case of 502, ignoring the update (not closing existing nor opening new PRs) should work, shouldn't it?

Do you have clue on the reason why that project is getting a 502? Is this something common? It appears to be reproducible (at least, I retried several times with a few hours between the attempts).

rarkins commented 2 years ago

If there are external host errors then we'll have the wrong state so should not update or close any PRs during that run. Therefore we choose to abort.

rarkins commented 2 years ago

Run 1: works, creates PR Run 2: 502, autoclose a PR Run 3: works again, recreates or reopens PR

DanySK commented 2 years ago

I see. My question is: why autoclose on 502? Can't renovate just leave that open? (maybe adding a comment that it errored)

rarkins commented 2 years ago

Because there is not a simple 1:1 relationship between lookups and PRs (due to grouping etc). It's really more complicated than you think

DanySK commented 2 years ago

I still do not get why all other updates (from other managers) should get discarded, is it for the grouping, too? I'm in this strange situation by which updates on Actions fail with 502 (and I cannot understand why, but it reproducibly fails), and I get my (completely unrelated) gradle updates blocked as well.

Can dependencies get grouped across managers? And if not, why not let non-failing managers complete?

Also, I had to fetch the logs manually: if renovate fails, it might alert the user somehow. At this moment, there is a silent failure, all updates stop, and no notification is provided. If this is intended behaviour, I believe it should get at least documented, so that users can set up some reminder to go check the dashboard every some time to ensure the bot is running.

Moreover, I still cannot understand why I keep getting 502s for updates on that repository only. Can this behaviour get reported to GitHub, in your opinion?

Failing log ``` DEBUG: No dangling containers to remove INFO: Repository started { "renovateVersion": "32.100.3" } DEBUG: Using localDir: /mnt/renovate/gh/DanySK/publish-on-central DEBUG: PackageFiles.clear() - Package files deleted { "baseBranches": [] } DEBUG: resetMemCache() DEBUG: initRepo("DanySK/publish-on-central") DEBUG: DanySK/publish-on-central default branch = master DEBUG: Using app token for git init DEBUG: Repository cache is valid DEBUG: Resetting npmrc DEBUG: detectSemanticCommits() DEBUG: checkOnboarding() DEBUG: isOnboarded() DEBUG: Checking cached config file name DEBUG: Existing config file confirmed DEBUG: Repository config { "fileName": "renovate.json", "config": { "$schema": "https://docs.renovatebot.com/renovate-schema.json", "extends": [ "github>DanySK/renovate-config:gradle-plugin" ], "packageRules": [ { "description": "Trigger a minor update on some dependencies", "matchPackageNames": [ "io.github.gradle-nexus.publish-plugin", "io.github.gradle-nexus.publish-plugin" ], "semanticCommitScope": "api-deps" } ] } } DEBUG: Repo is onboarded DEBUG: migrateAndValidate() DEBUG: No config migration necessary DEBUG: massaged config { "config": { "$schema": "https://docs.renovatebot.com/renovate-schema.json", "extends": [ "github>whitesource/merge-confidence:beta", "github>renovatebot/spring-remediations", "github>DanySK/renovate-config:gradle-plugin" ], "packageRules": [ { "description": [ "Trigger a minor update on some dependencies" ], "matchPackageNames": [ "io.github.gradle-nexus.publish-plugin", "io.github.gradle-nexus.publish-plugin" ], "semanticCommitScope": "api-deps" } ] } } DEBUG: migrated config { "config": { "$schema": "https://docs.renovatebot.com/renovate-schema.json", "extends": [ "github>whitesource/merge-confidence:beta", "github>renovatebot/spring-remediations", "github>DanySK/renovate-config:gradle-plugin" ], "packageRules": [ { "description": [ "Trigger a minor update on some dependencies" ], "matchPackageNames": [ "io.github.gradle-nexus.publish-plugin", "io.github.gradle-nexus.publish-plugin" ], "semanticCommitScope": "api-deps" } ] } } DEBUG: Setting hostRules from config DEBUG: Found repo ignorePaths { "ignorePaths": [ "**/node_modules/**", "**/bower_components/**", "**/vendor/**", "**/examples/**", "**/__tests__/**", "**/test/**", "**/tests/**", "**/__fixtures__/**" ] } DEBUG: No vulnerability alerts found DEBUG: No vulnerability alerts found DEBUG: findIssue(Dependency Dashboard) DEBUG: Retrieving issueList DEBUG: Retrieved 1 issues DEBUG: Found issue 142 DEBUG: No baseBranches DEBUG: extract() DEBUG: Setting current branch to master DEBUG: Initializing git repository into /mnt/renovate/gh/DanySK/publish-on-central DEBUG: Performing blobless clone DEBUG: git clone completed { "durationMs": 1036 } DEBUG: latest repository commit { "latestCommit": { "hash": "dac9153b0cf689a8f137205cc796cc1569df2aa1", "date": "2022-07-01T09:30:52+02:00", "message": "ci: use lowercase references for actions", "refs": "HEAD -> master, origin/master, origin/HEAD", "body": "", "author_name": "Danilo Pianini", "author_email": "danilo.pianini@unibo.it" } } DEBUG: latest commit { "branchName": "master", "latestCommitDate": "2022-07-01T09:30:52+02:00" } DEBUG: Using file match: (^|/)tasks/[^/]+\.ya?ml$ for manager ansible DEBUG: Using file match: (^|/)requirements\.ya?ml$ for manager ansible-galaxy DEBUG: Using file match: (^|/)galaxy\.ya?ml$ for manager ansible-galaxy DEBUG: Using file match: azure.*pipelines?.*\.ya?ml$ for manager azure-pipelines DEBUG: Using file match: (^|/)batect(-bundle)?\.yml$ for manager batect DEBUG: Using file match: (^|/)batect$ for manager batect-wrapper DEBUG: Using file match: (^|/)WORKSPACE(|\.bazel)$ for manager bazel DEBUG: Using file match: \.bzl$ for manager bazel DEBUG: Using file match: (^|\/)\.bazelversion$ for manager bazelisk DEBUG: Using file match: (^|/)\.?bitbucket-pipelines\.ya?ml$ for manager bitbucket-pipelines DEBUG: Using file match: buildkite\.ya?ml for manager buildkite DEBUG: Using file match: \.buildkite/.+\.ya?ml$ for manager buildkite DEBUG: Using file match: (^|/)Gemfile$ for manager bundler DEBUG: Using file match: \.cake$ for manager cake DEBUG: Using file match: (^|/)Cargo.toml$ for manager cargo DEBUG: Using file match: (^|/).circleci/config.yml$ for manager circleci DEBUG: Using file match: (^|/)cloudbuild.ya?ml for manager cloudbuild DEBUG: Using file match: (^|/)Podfile$ for manager cocoapods DEBUG: Using file match: (^|/)([\w-]*)composer.json$ for manager composer DEBUG: Using file match: (^|/)conanfile\.(txt|py)$ for manager conan DEBUG: Using file match: (^|/)(?:deps|bb)\.edn$ for manager deps-edn DEBUG: Using file match: (^|/)(?:docker-)?compose[^/]*\.ya?ml$ for manager docker-compose DEBUG: Using file match: (^|/|\.)Dockerfile$ for manager dockerfile DEBUG: Using file match: (^|/)Dockerfile[^/]*$ for manager dockerfile DEBUG: Using file match: (^|/).drone.yml$ for manager droneci DEBUG: Using file match: (^|/)flux-system/gotk-components\.yaml$ for manager flux DEBUG: Using file match: (^|\/)\.fvm\/fvm_config\.json$ for manager fvm DEBUG: Using file match: (^|/).gitmodules$ for manager git-submodules DEBUG: Using file match: ^(workflow-templates|\.github\/workflows)\/[^/]+\.ya?ml$ for manager github-actions DEBUG: Using file match: (^|\/)action\.ya?ml$ for manager github-actions DEBUG: Using file match: \.gitlab-ci\.yml$ for manager gitlabci DEBUG: Using file match: \.gitlab-ci\.yml$ for manager gitlabci-include DEBUG: Using file match: (^|/)go.mod$ for manager gomod DEBUG: Using file match: \.gradle(\.kts)?$ for manager gradle DEBUG: Using file match: (^|\/)gradle\.properties$ for manager gradle DEBUG: Using file match: (^|\/)gradle\/.+\.toml$ for manager gradle DEBUG: Using file match: \.versions\.toml$ for manager gradle DEBUG: Using file match: (^|/)gradle/wrapper/gradle-wrapper.properties$ for manager gradle-wrapper DEBUG: Using file match: (^|/)requirements\.yaml$ for manager helm-requirements DEBUG: Using file match: (^|/)values.yaml$ for manager helm-values DEBUG: Using file match: (^|/)helmfile.yaml$ for manager helmfile DEBUG: Using file match: (^|/)Chart.yaml$ for manager helmv3 DEBUG: Using file match: ^Formula/[^/]+[.]rb$ for manager homebrew DEBUG: Using file match: \.html?$ for manager html DEBUG: Using file match: (^|/)plugins\.(txt|ya?ml)$ for manager jenkins DEBUG: Using file match: (^|/)jsonnetfile.json$ for manager jsonnet-bundler DEBUG: Using file match: (^|/)kustomization\.ya?ml$ for manager kustomize DEBUG: Using file match: (^|/)project\.clj$ for manager leiningen DEBUG: Using file match: (^|/|\.)pom\.xml$ for manager maven DEBUG: Using file match: ^(((\.mvn)|(\.m2))/)?settings\.xml$ for manager maven DEBUG: Using file match: (^|/)package.js$ for manager meteor DEBUG: Using file match: (^|/)mix\.exs$ for manager mix DEBUG: Using file match: (^|/).node-version$ for manager nodenv DEBUG: Using file match: (^|/)package.json$ for manager npm DEBUG: Using file match: \.(?:cs|fs|vb)proj$ for manager nuget DEBUG: Using file match: \.(?:props|targets)$ for manager nuget DEBUG: Using file match: (^|\/)dotnet-tools\.json$ for manager nuget DEBUG: Using file match: (^|\/)global\.json$ for manager nuget DEBUG: Using file match: (^|/)\.nvmrc$ for manager nvm DEBUG: Using file match: (^|/)([\w-]*)requirements\.(txt|pip)$ for manager pip_requirements DEBUG: Using file match: (^|/)setup.py$ for manager pip_setup DEBUG: Using file match: (^|/)Pipfile$ for manager pipenv DEBUG: Using file match: (^|/)pyproject\.toml$ for manager poetry DEBUG: Using file match: (^|/)\.pre-commit-config\.yaml$ for manager pre-commit DEBUG: Using file match: (^|/)pubspec\.ya?ml$ for manager pub DEBUG: Using file match: (^|/).python-version$ for manager pyenv DEBUG: Using file match: (^|/)\.ruby-version$ for manager ruby-version DEBUG: Using file match: \.sbt$ for manager sbt DEBUG: Using file match: project/[^/]*.scala$ for manager sbt DEBUG: Using file match: (^|/)setup\.cfg$ for manager setup-cfg DEBUG: Using file match: (^|/)Package\.swift for manager swift DEBUG: Using file match: \.tf$ for manager terraform DEBUG: Using file match: (^|/)\.terraform-version$ for manager terraform-version DEBUG: Using file match: (^|/)terragrunt\.hcl$ for manager terragrunt DEBUG: Using file match: (^|/)\.terragrunt-version$ for manager terragrunt-version DEBUG: Using file match: ^.travis.yml$ for manager travis DEBUG: Using file match: (^|/).vela.ya?ml$ for manager velaci DEBUG: Matched 1 file(s) for manager git-submodules: .gitmodules DEBUG: Matched 2 file(s) for manager github-actions: .github/workflows/build-and-deploy.yml, .github/workflows/dispatcher.yml DEBUG: Matched 4 file(s) for manager gradle: build.gradle.kts, gradle.properties, gradle/libs.versions.toml, settings.gradle.kts DEBUG: Matched 1 file(s) for manager gradle-wrapper: gradle/wrapper/gradle-wrapper.properties DEBUG: Matched 1 file(s) for manager npm: package.json DEBUG: packageFile has no content(packageFile=".gitmodules") DEBUG: npm file package.json has name undefined DEBUG: Detecting pnpm Workspaces DEBUG: Detecting Lerna and Yarn Workspaces DEBUG: Finding locked versions DEBUG: Found package-lock.json for package.json DEBUG: Found github-actions package files DEBUG: Found gradle package files DEBUG: Found gradle-wrapper package files DEBUG: Found npm package files DEBUG: Found 7 package file(s) INFO: Dependency extraction complete { "baseBranch": "master", "stats": { "managers": { "github-actions": { "fileCount": 1, "depCount": 12 }, "gradle": { "fileCount": 4, "depCount": 15 }, "gradle-wrapper": { "fileCount": 1, "depCount": 1 }, "npm": { "fileCount": 1, "depCount": 2 } }, "total": { "fileCount": 7, "depCount": 30 } } } DEBUG: Looking up com.gradle.enterprise:com.gradle.enterprise.gradle.plugin in repository https://repo.maven.apache.org/maven2/ DEBUG: Looking up org.danilopianini.gradle-pre-commit-git-hooks:org.danilopianini.gradle-pre-commit-git-hooks.gradle.plugin in repository https://repo.maven.apache.org/maven2/ DEBUG: Looking up com.uchuhimo:konf-yaml in repository https://repo.maven.apache.org/maven2/ DEBUG: Looking up io.kotest:kotest-runner-junit5-jvm in repository https://repo.maven.apache.org/maven2/ DEBUG: Looking up io.kotest:kotest-assertions-core-jvm in repository https://repo.maven.apache.org/maven2/ DEBUG: Looking up io.github.gradle-nexus:publish-plugin in repository https://repo.maven.apache.org/maven2/ DEBUG: Looking up org.jetbrains.dokka:org.jetbrains.dokka.gradle.plugin in repository https://repo.maven.apache.org/maven2/ DEBUG: Found 15 new releases for com.uchuhimo:konf-yaml in repository https://repo.maven.apache.org/maven2/ DEBUG: Found 62 new releases for io.kotest:kotest-runner-junit5-jvm in repository https://repo.maven.apache.org/maven2/ DEBUG: Found 61 new releases for io.kotest:kotest-assertions-core-jvm in repository https://repo.maven.apache.org/maven2/ DEBUG: Found 9 new releases for org.jetbrains.dokka:org.jetbrains.dokka.gradle.plugin in repository https://repo.maven.apache.org/maven2/ DEBUG: Content is not found for Maven url { "url": "https://repo.maven.apache.org/maven2/com/gradle/enterprise/com.gradle.enterprise.gradle.plugin/maven-metadata.xml" } DEBUG: Looking up com.gradle.enterprise:com.gradle.enterprise.gradle.plugin in repository https://plugins.gradle.org/m2/ DEBUG: Content is not found for Maven url { "url": "https://repo.maven.apache.org/maven2/org/danilopianini/gradle-pre-commit-git-hooks/org.danilopianini.gradle-pre-commit-git-hooks.gradle.plugin/maven-metadata.xml" } DEBUG: Found 29 new releases for com.gradle.enterprise:com.gradle.enterprise.gradle.plugin in repository https://plugins.gradle.org/m2/ DEBUG: Content is not found for Maven url { "url": "https://repo.maven.apache.org/maven2/io/github/gradle-nexus/publish-plugin/maven-metadata.xml" } DEBUG: Looking up org.danilopianini.gradle-pre-commit-git-hooks:org.danilopianini.gradle-pre-commit-git-hooks.gradle.plugin in repository https://plugins.gradle.org/m2/ DEBUG: Looking up org.jetbrains.dokka:org.jetbrains.dokka.gradle.plugin in repository https://plugins.gradle.org/m2/ DEBUG: Looking up com.uchuhimo:konf-yaml in repository https://plugins.gradle.org/m2/ DEBUG: Looking up io.kotest:kotest-assertions-core-jvm in repository https://plugins.gradle.org/m2/ DEBUG: Looking up io.github.gradle-nexus:publish-plugin in repository https://plugins.gradle.org/m2/ DEBUG: Looking up io.kotest:kotest-runner-junit5-jvm in repository https://plugins.gradle.org/m2/ DEBUG: Found 13 new releases for org.danilopianini.gradle-pre-commit-git-hooks:org.danilopianini.gradle-pre-commit-git-hooks.gradle.plugin in repository https://plugins.gradle.org/m2/ DEBUG: Found 32 new releases for org.jetbrains.dokka:org.jetbrains.dokka.gradle.plugin in repository https://plugins.gradle.org/m2/ DEBUG: Found 15 new releases for com.uchuhimo:konf-yaml in repository https://plugins.gradle.org/m2/ DEBUG: Found 61 new releases for io.kotest:kotest-assertions-core-jvm in repository https://plugins.gradle.org/m2/ DEBUG: Found 2 new releases for io.github.gradle-nexus:publish-plugin in repository https://plugins.gradle.org/m2/ DEBUG: Found 62 new releases for io.kotest:kotest-runner-junit5-jvm in repository https://plugins.gradle.org/m2/ DEBUG: Looking up org.danilopianini.git-sensitive-semantic-versioning:org.danilopianini.git-sensitive-semantic-versioning.gradle.plugin in repository https://repo.maven.apache.org/maven2/ DEBUG: Content is not found for Maven url { "url": "https://repo.maven.apache.org/maven2/org/danilopianini/git-sensitive-semantic-versioning/org.danilopianini.git-sensitive-semantic-versioning.gradle.plugin/maven-metadata.xml" } DEBUG: Looking up org.danilopianini.git-sensitive-semantic-versioning:org.danilopianini.git-sensitive-semantic-versioning.gradle.plugin in repository https://plugins.gradle.org/m2/ DEBUG: Found 145 new releases for org.danilopianini.git-sensitive-semantic-versioning:org.danilopianini.git-sensitive-semantic-versioning.gradle.plugin in repository https://plugins.gradle.org/m2/ DEBUG: Looking up com.gradle.plugin-publish:com.gradle.plugin-publish.gradle.plugin in repository https://repo.maven.apache.org/maven2/ DEBUG: Content is not found for Maven url { "url": "https://repo.maven.apache.org/maven2/com/gradle/plugin-publish/com.gradle.plugin-publish.gradle.plugin/maven-metadata.xml" } DEBUG: Looking up com.gradle.plugin-publish:com.gradle.plugin-publish.gradle.plugin in repository https://plugins.gradle.org/m2/ DEBUG: Found 28 new releases for com.gradle.plugin-publish:com.gradle.plugin-publish.gradle.plugin in repository https://plugins.gradle.org/m2/ DEBUG: Looking up org.jetbrains.kotlin.jvm:org.jetbrains.kotlin.jvm.gradle.plugin in repository https://repo.maven.apache.org/maven2/ DEBUG: Looking up org.danilopianini.gradle-kotlin-qa:org.danilopianini.gradle-kotlin-qa.gradle.plugin in repository https://repo.maven.apache.org/maven2/ DEBUG: Found 4 new releases for org.jetbrains.kotlin.jvm:org.jetbrains.kotlin.jvm.gradle.plugin in repository https://repo.maven.apache.org/maven2/ DEBUG: Looking up org.jetbrains.kotlin.jvm:org.jetbrains.kotlin.jvm.gradle.plugin in repository https://plugins.gradle.org/m2/ DEBUG: Content is not found for Maven url { "url": "https://repo.maven.apache.org/maven2/org/danilopianini/gradle-kotlin-qa/org.danilopianini.gradle-kotlin-qa.gradle.plugin/maven-metadata.xml" } DEBUG: Looking up org.danilopianini.gradle-kotlin-qa:org.danilopianini.gradle-kotlin-qa.gradle.plugin in repository https://plugins.gradle.org/m2/ DEBUG: Found 92 new releases for org.jetbrains.kotlin.jvm:org.jetbrains.kotlin.jvm.gradle.plugin in repository https://plugins.gradle.org/m2/ DEBUG: Found 87 new releases for org.danilopianini.gradle-kotlin-qa:org.danilopianini.gradle-kotlin-qa.gradle.plugin in repository https://plugins.gradle.org/m2/ DEBUG: Looking up org.danilopianini.multi-jvm-test-plugin:org.danilopianini.multi-jvm-test-plugin.gradle.plugin in repository https://repo.maven.apache.org/maven2/ DEBUG: Looking up io.github.gradle-nexus.publish-plugin:io.github.gradle-nexus.publish-plugin.gradle.plugin in repository https://repo.maven.apache.org/maven2/ DEBUG: Looking up org.danilopianini.publish-on-central:org.danilopianini.publish-on-central.gradle.plugin in repository https://repo.maven.apache.org/maven2/ DEBUG: Looking up com.dorongold.task-tree:com.dorongold.task-tree.gradle.plugin in repository https://repo.maven.apache.org/maven2/ DEBUG: Found 3 new releases for org.danilopianini.publish-on-central:org.danilopianini.publish-on-central.gradle.plugin in repository https://repo.maven.apache.org/maven2/ DEBUG: Content is not found for Maven url { "url": "https://repo.maven.apache.org/maven2/io/github/gradle-nexus/publish-plugin/io.github.gradle-nexus.publish-plugin.gradle.plugin/maven-metadata.xml" } DEBUG: Content is not found for Maven url { "url": "https://repo.maven.apache.org/maven2/org/danilopianini/multi-jvm-test-plugin/org.danilopianini.multi-jvm-test-plugin.gradle.plugin/maven-metadata.xml" } DEBUG: Looking up io.github.gradle-nexus.publish-plugin:io.github.gradle-nexus.publish-plugin.gradle.plugin in repository https://plugins.gradle.org/m2/ DEBUG: Looking up org.danilopianini.multi-jvm-test-plugin:org.danilopianini.multi-jvm-test-plugin.gradle.plugin in repository https://plugins.gradle.org/m2/ DEBUG: Found 2 new releases for io.github.gradle-nexus.publish-plugin:io.github.gradle-nexus.publish-plugin.gradle.plugin in repository https://plugins.gradle.org/m2/ DEBUG: Found 58 new releases for org.danilopianini.multi-jvm-test-plugin:org.danilopianini.multi-jvm-test-plugin.gradle.plugin in repository https://plugins.gradle.org/m2/ DEBUG: Looking up org.danilopianini.publish-on-central:org.danilopianini.publish-on-central.gradle.plugin in repository https://plugins.gradle.org/m2/ DEBUG: Content is not found for Maven url { "url": "https://repo.maven.apache.org/maven2/com/dorongold/task-tree/com.dorongold.task-tree.gradle.plugin/maven-metadata.xml" } DEBUG: Looking up com.dorongold.task-tree:com.dorongold.task-tree.gradle.plugin in repository https://plugins.gradle.org/m2/ DEBUG: Found 343 new releases for org.danilopianini.publish-on-central:org.danilopianini.publish-on-central.gradle.plugin in repository https://plugins.gradle.org/m2/ DEBUG: Found 13 new releases for com.dorongold.task-tree:com.dorongold.task-tree.gradle.plugin in repository https://plugins.gradle.org/m2/ DEBUG: GitHub failure: 5xx { "err": { "name": "HTTPError", "code": "ERR_NON_2XX_3XX_RESPONSE", "timings": { "start": 1656661149266, "socket": 1656661149731, "lookup": 1656661150136, "connect": 1656661150372, "secureConnect": 1656661150476, "upload": 1656661150485, "response": 1656661161024, "end": 1656661161024, "phases": { "wait": 465, "dns": 405, "tcp": 236, "tls": 104, "request": 9, "firstByte": 10539, "download": 0, "total": 11758 } }, "message": "Response code 502 (Bad Gateway)", "stack": "HTTPError: Response code 502 (Bad Gateway)\n at Request. (/home/ubuntu/renovateapp/node_modules/got/dist/source/as-promise/index.js:118:42)\n at runMicrotasks ()\n at processTicksAndRejections (internal/process/task_queues.js:95:5)", "options": { "headers": { "user-agent": "Renovate Bot (GitHub App 2740)", "accept": "application/vnd.github.machine-man-preview+json", "authorization": "***********", "content-type": "application/json", "content-length": "875", "accept-encoding": "gzip, deflate, br" }, "url": "https://api.github.com/graphql", "hostType": "github-tags", "username": "", "password": "", "method": "POST", "http2": false }, "response": { "statusCode": 502, "statusMessage": "Bad Gateway", "body": { "data": null, "errors": [ { "message": "Something went wrong while executing your query. This may be the result of a timeout, or it could be a GitHub bug. Please include `AD8C:7E98:AC1D0:C341C:62BEA49E` when reporting this issue." } ] }, "headers": { "server": "GitHub.com", "date": "Fri, 01 Jul 2022 07:39:20 GMT", "content-type": "application/json", "transfer-encoding": "chunked", "access-control-expose-headers": "ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset", "access-control-allow-origin": "*", "vary": "Accept-Encoding, Accept, X-Requested-With", "x-github-request-id": "AD8C:7E98:AC1D0:C341C:62BEA49E", "connection": "close" }, "httpVersion": "1.1" } } } WARN: Host error { "hostType": "github-tags", "packageName": "alchemistsimulator/alchemist", "err": { "name": "HTTPError", "code": "ERR_NON_2XX_3XX_RESPONSE", "timings": { "start": 1656661149266, "socket": 1656661149731, "lookup": 1656661150136, "connect": 1656661150372, "secureConnect": 1656661150476, "upload": 1656661150485, "response": 1656661161024, "end": 1656661161024, "phases": { "wait": 465, "dns": 405, "tcp": 236, "tls": 104, "request": 9, "firstByte": 10539, "download": 0, "total": 11758 } }, "message": "Response code 502 (Bad Gateway)", "stack": "HTTPError: Response code 502 (Bad Gateway)\n at Request. (/home/ubuntu/renovateapp/node_modules/got/dist/source/as-promise/index.js:118:42)\n at runMicrotasks ()\n at processTicksAndRejections (internal/process/task_queues.js:95:5)", "options": { "headers": { "user-agent": "Renovate Bot (GitHub App 2740)", "accept": "application/vnd.github.machine-man-preview+json", "authorization": "***********", "content-type": "application/json", "content-length": "875", "accept-encoding": "gzip, deflate, br" }, "url": "https://api.github.com/graphql", "hostType": "github-tags", "username": "", "password": "", "method": "POST", "http2": false }, "response": { "statusCode": 502, "statusMessage": "Bad Gateway", "body": { "data": null, "errors": [ { "message": "Something went wrong while executing your query. This may be the result of a timeout, or it could be a GitHub bug. Please include `AD8C:7E98:AC1D0:C341C:62BEA49E` when reporting this issue." } ] }, "headers": { "server": "GitHub.com", "date": "Fri, 01 Jul 2022 07:39:20 GMT", "content-type": "application/json", "transfer-encoding": "chunked", "access-control-expose-headers": "ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset", "access-control-allow-origin": "*", "vary": "Accept-Encoding, Accept, X-Requested-With", "x-github-request-id": "AD8C:7E98:AC1D0:C341C:62BEA49E", "connection": "close" }, "httpVersion": "1.1" } } } INFO: External host error causing abort - skipping DEBUG: Repository result: external-host-error, status: onboarded, enabled: true, onboarded: true DEBUG: Repository timing splits (milliseconds) { "splits": { "init": 3055, "extract": 3131 }, "total": 18291 } DEBUG: http statistics { "urls": { "https://api.github.com/graphql (POST,200)": 2, "https://api.github.com/graphql (POST,502)": 1, "https://api.github.com/repos/DanySK/publish-on-central/contents/renovate.json (GET,200)": 1, "https://api.github.com/repos/DanySK/publish-on-central/issues/142 (GET,200)": 1, "https://api.github.com/repos/DanySK/renovate-config/contents/default.json (GET,200)": 1, "https://api.github.com/repos/DanySK/renovate-config/contents/gradle-plugin.json (GET,200)": 1, "https://api.github.com/repos/renovatebot/spring-remediations/contents/base.json (GET,200)": 1, "https://api.github.com/repos/renovatebot/spring-remediations/contents/default.json (GET,200)": 1, "https://api.github.com/repos/whitesource/merge-confidence/contents/beta.json (GET,200)": 1, "https://plugins.gradle.org/m2/com/dorongold/task-tree/com.dorongold.task-tree.gradle.plugin/2.1.0/com.dorongold.task-tree.gradle.plugin-2.1.0.pom (GET,200)": 1, "https://plugins.gradle.org/m2/com/gradle/enterprise/com.gradle.enterprise.gradle.plugin/3.10.2/com.gradle.enterprise.gradle.plugin-3.10.2.pom (GET,200)": 1, "https://plugins.gradle.org/m2/com/gradle/plugin-publish/com.gradle.plugin-publish.gradle.plugin/1.0.0/com.gradle.plugin-publish.gradle.plugin-1.0.0.pom (GET,200)": 1, "https://plugins.gradle.org/m2/com/uchuhimo/konf-yaml/1.1.2/konf-yaml-1.1.2.pom (GET,200)": 1, "https://plugins.gradle.org/m2/io/github/gradle-nexus/publish-plugin/1.1.0/publish-plugin-1.1.0.pom (GET,200)": 1, "https://plugins.gradle.org/m2/io/github/gradle-nexus/publish-plugin/io.github.gradle-nexus.publish-plugin.gradle.plugin/1.1.0/io.github.gradle-nexus.publish-plugin.gradle.plugin-1.1.0.pom (GET,200)": 1, "https://plugins.gradle.org/m2/io/kotest/kotest-assertions-core-jvm/5.3.2/kotest-assertions-core-jvm-5.3.2.pom (GET,200)": 1, "https://plugins.gradle.org/m2/io/kotest/kotest-runner-junit5-jvm/5.3.2/kotest-runner-junit5-jvm-5.3.2.pom (GET,200)": 1, "https://plugins.gradle.org/m2/org/danilopianini/git-sensitive-semantic-versioning/org.danilopianini.git-sensitive-semantic-versioning.gradle.plugin/0.3.0-dev03-1aa1285/org.danilopianini.git-sensitive-semantic-versioning.gradle.plugin-0.3.0-dev03-1aa1285.pom (GET,200)": 1, "https://plugins.gradle.org/m2/org/danilopianini/gradle-kotlin-qa/org.danilopianini.gradle-kotlin-qa.gradle.plugin/0.19.1/org.danilopianini.gradle-kotlin-qa.gradle.plugin-0.19.1.pom (GET,200)": 1, "https://plugins.gradle.org/m2/org/danilopianini/gradle-pre-commit-git-hooks/org.danilopianini.gradle-pre-commit-git-hooks.gradle.plugin/1.0.12/org.danilopianini.gradle-pre-commit-git-hooks.gradle.plugin-1.0.12.pom (GET,200)": 1, "https://plugins.gradle.org/m2/org/danilopianini/multi-jvm-test-plugin/org.danilopianini.multi-jvm-test-plugin.gradle.plugin/0.4.3/org.danilopianini.multi-jvm-test-plugin.gradle.plugin-0.4.3.pom (GET,200)": 1, "https://plugins.gradle.org/m2/org/danilopianini/publish-on-central/org.danilopianini.publish-on-central.gradle.plugin/2.0.2/org.danilopianini.publish-on-central.gradle.plugin-2.0.2.pom (GET,200)": 1, "https://plugins.gradle.org/m2/org/jetbrains/dokka/org.jetbrains.dokka.gradle.plugin/1.7.0/org.jetbrains.dokka.gradle.plugin-1.7.0.pom (GET,200)": 1, "https://plugins.gradle.org/m2/org/jetbrains/kotlin/jvm/org.jetbrains.kotlin.jvm.gradle.plugin/1.7.0/org.jetbrains.kotlin.jvm.gradle.plugin-1.7.0.pom (GET,200)": 1, "https://repo.maven.apache.org/maven2/com/dorongold/task-tree/com.dorongold.task-tree.gradle.plugin/maven-metadata.xml (GET,404)": 1, "https://repo.maven.apache.org/maven2/com/gradle/enterprise/com.gradle.enterprise.gradle.plugin/maven-metadata.xml (GET,404)": 1, "https://repo.maven.apache.org/maven2/com/gradle/plugin-publish/com.gradle.plugin-publish.gradle.plugin/maven-metadata.xml (GET,404)": 1, "https://repo.maven.apache.org/maven2/com/uchuhimo/konf-yaml/1.1.2/konf-yaml-1.1.2.pom (GET,200)": 1, "https://repo.maven.apache.org/maven2/io/github/gradle-nexus/publish-plugin/io.github.gradle-nexus.publish-plugin.gradle.plugin/maven-metadata.xml (GET,404)": 1, "https://repo.maven.apache.org/maven2/io/github/gradle-nexus/publish-plugin/maven-metadata.xml (GET,404)": 1, "https://repo.maven.apache.org/maven2/io/kotest/kotest-assertions-core-jvm/5.3.2/kotest-assertions-core-jvm-5.3.2.pom (GET,200)": 1, "https://repo.maven.apache.org/maven2/io/kotest/kotest-runner-junit5-jvm/5.3.2/kotest-runner-junit5-jvm-5.3.2.pom (GET,200)": 1, "https://repo.maven.apache.org/maven2/org/danilopianini/git-sensitive-semantic-versioning/org.danilopianini.git-sensitive-semantic-versioning.gradle.plugin/maven-metadata.xml (GET,404)": 1, "https://repo.maven.apache.org/maven2/org/danilopianini/gradle-kotlin-qa/org.danilopianini.gradle-kotlin-qa.gradle.plugin/maven-metadata.xml (GET,404)": 1, "https://repo.maven.apache.org/maven2/org/danilopianini/gradle-pre-commit-git-hooks/org.danilopianini.gradle-pre-commit-git-hooks.gradle.plugin/maven-metadata.xml (GET,404)": 1, "https://repo.maven.apache.org/maven2/org/danilopianini/multi-jvm-test-plugin/org.danilopianini.multi-jvm-test-plugin.gradle.plugin/maven-metadata.xml (GET,404)": 1, "https://repo.maven.apache.org/maven2/org/danilopianini/publish-on-central/org.danilopianini.publish-on-central.gradle.plugin/2.0.2/org.danilopianini.publish-on-central.gradle.plugin-2.0.2.pom (GET,200)": 1, "https://repo.maven.apache.org/maven2/org/jetbrains/dokka/org.jetbrains.dokka.gradle.plugin/1.7.0/org.jetbrains.dokka.gradle.plugin-1.7.0.pom (GET,200)": 1, "https://repo.maven.apache.org/maven2/org/jetbrains/kotlin/jvm/org.jetbrains.kotlin.jvm.gradle.plugin/1.7.0/org.jetbrains.kotlin.jvm.gradle.plugin-1.7.0.pom (GET,200)": 1 }, "hostStats": { "api.github.com": { "requestCount": 10, "requestAvgMs": 1348, "queueAvgMs": 0 }, "plugins.gradle.org": { "requestCount": 15, "requestAvgMs": 398, "queueAvgMs": 0 }, "repo.maven.apache.org": { "requestCount": 15, "requestAvgMs": 346, "queueAvgMs": 0 } }, "totalRequests": 40 } INFO: Repository finished { "durationMs": 18291 } ```
rarkins commented 2 years ago

A lookup results in one or more updated versions. Updated versions are placed into buckets, and can produce one or more updates. Updates are later grouped together according to possibly very complex rules into branches/PRs. Therefore if a lookup fails, we can't know which of those branches/PRs would have been "touched", and cannot safely update or autoclose any of them.

Essentially, if Renovate had less features and was less configurable, there'd be a simple link guaranteed between lookups and PRs, and what you suggest might be possible. But with Renovate's configurability, it's not practical.

Can dependencies get grouped across managers? And if not, why not let non-failing managers complete?

Yes, and it's common

Also, I had to fetch the logs manually: if renovate fails, it might alert the user somehow. At this moment, there is a silent failure, all updates stop, and no notification is provided. If this is intended behaviour, I believe it should get at least documented, so that users can set up some reminder to go check the dashboard every some time to ensure the bot is running.

I would be happy if you can work out a way to flag this into the Dependency Dashboard issue. It would still cause a little "noise", but at least not so much as if we open or close PRs repeatedly during error conditions.

Moreover, I still cannot understand why I keep getting 502s for updates on that repository only. Can this behaviour get reported to GitHub, in your opinion?

Can you also reproduce it in a minimal reproduction repo? Can you reproduce it using e.g. curl?

It does sound like a GitHub bug, but also like something they may not be interested fixing. Essentially it's a known problem with their GraphQL that if a query takes too long to run, you can get 5xx errors.

Right away I'd suggest ignoring whatever dependency causes it. Then once you can reproduce it in a minimal repo, @zharinov can look into whether we can catch/retry this with a lower page count to hopefully avoid the 5xx.

DanySK commented 2 years ago

Thank you so much for the explanation, it is clear now. I would gladly try to reproduce it, I already opened a ticket GitHub-side, Is there any way I can understand which query is renovate performing from the logs?

DanySK commented 2 years ago

I got a response from GitHub:

I took a look at the logs using the request-id in your report and as the message indicated -- you hit a timeout running the query. All API requests, both for the REST API and GraphQL API, have a 10-second limit on execution time. If that limit is reached for a request, the request is terminated and you get back that error. This normally happens when the query involves too much data, so the way to avoid timeouts is to write smaller queries. If the query touches a lot of data -- split it into several smaller queries and execute them separately.

Now... is there any way I can configure Renovate to run "several smaller queries"? Or any workaround I might apply?

rarkins commented 2 years ago

Seeing as this seems fairly isolated, I think we need to reduce the page size and retry when it happens. A reproduction would be really helpful for this, then @zharinov can advise as he wrote the same for other GraphQL queries.

It's likely the query here which throws: https://github.com/renovatebot/renovate/blob/main/lib/modules/datasource/github-tags/cache.ts

The logic is mostly from here, including fetching 100 at a time: https://github.com/renovatebot/renovate/blob/main/lib/modules/datasource/github-releases/cache/cache-base.ts

github-actions[bot] commented 2 years ago

Hi there,

Get your issue fixed faster by creating a minimal reproduction. This means a repository dedicated to reproducing this issue with the minimal dependencies and config possible.

Before we start working on your issue we need to know exactly what's causing the current behavior. A minimal reproduction helps us with this.

To get started, please read our guide on creating a minimal reproduction.

We may close the issue if you, or someone else, haven't created a minimal reproduction within two weeks. If you need more time, or are stuck, please ask for help or more time in a comment.

Good luck,

The Renovate team

DanySK commented 2 years ago

@rarkins @zharinov minimal reproduction, here it is https://github.com/DanySK/minimal-reproduction-renovate-16343

rarkins commented 2 years ago

Forked to https://github.com/renovate-reproductions/16343

rarkins commented 2 years ago

The fix we need is:

I changed the default from 100 to 50 locally and re-ran, and the lookup succeeds, so one catch/retry should be enough.

renovate-release commented 2 years ago

:tada: This issue has been resolved in version 32.103.1 :tada:

The release is available on:

Your semantic-release bot :package::rocket: