Dependabot is timing out

[MikeAlhayek]

The Dependabot action runs as schedule. It times out after running for 60 mins.

I believe that this issue happened after CentralPackageTransitivePinningEnabled was enabled. https://github.com/OrchardCMS/OrchardCore/pull/16566

Last successful run was August 13th.

[Piedone]

The last couple of runs failed with various different errors, as visible in the output of the workflow run. One error that appeared thrice in slightly different variations, including the most recent run is this:

2024-09-16T19:42:14.1794297Z Dependabot encountered '6' error(s) during execution, please check the logs for more details.
2024-09-16T19:42:14.1795324Z +-----------------------------------------------------------------+
2024-09-16T19:42:14.1796052Z |                  Dependencies failed to update                  |
2024-09-16T19:42:14.1796863Z +-------------------------------------------------+---------------+
2024-09-16T19:42:14.1797843Z | GraphQL                                         | unknown_error |
2024-09-16T19:42:14.1798654Z | GraphQL.MicrosoftDI                             | unknown_error |
2024-09-16T19:42:14.1799521Z | GraphQL.SystemTextJson                          | unknown_error |
2024-09-16T19:42:14.1800459Z | Microsoft.IdentityModel.Protocols.OpenIdConnect | unknown_error |
2024-09-16T19:42:14.1801442Z | OpenIddict.Validation.SystemNetHttp             | unknown_error |
2024-09-16T19:42:14.1802348Z | MessagePack                                     | unknown_error |
2024-09-16T19:42:14.1803160Z +-------------------------------------------------+---------------+
2024-09-16T19:42:14.6885640Z Failure running container 50880fd4a02906f4326c9766bee5e9f11d2b7da4ddbe9dae0e91bd24fd28bd7d
2024-09-16T19:42:16.4978038Z Cleaned up container 50880fd4a02906f4326c9766bee5e9f11d2b7da4ddbe9dae0e91bd24fd28bd7d
2024-09-16T19:42:16.5307419Z   proxy | 2024/09/16 19:42:16 264/2078 calls cached (12%)
2024-09-16T19:42:16.5308361Z 2024/09/16 19:42:16 Posting metrics to remote API endpoint
2024-09-16T19:42:16.5598325Z   proxy | 2024/09/16 19:42:16 Successfully posted metrics data via api client
2024-09-16T19:42:17.6763512Z ##[error]Dependabot encountered an error performing the update

Error: The updater encountered one or more errors.

For more information see: https://github.com/OrchardCMS/OrchardCore/network/updates/885578226 (write access to the repository is required to view the log)

The more info link is a circular reference that just eventually brings you back to the workflow run.

If you look for e.g. MessagePack in the log, this is what you get:

2024-09-16T19:41:29.7691127Z updater | 2024/09/16 19:41:29 INFO <job_885578226> Checking if MessagePack 2.2.60 needs updating
2024-09-16T19:41:29.8655727Z   proxy | 2024/09/16 19:41:29 [500] GET https://api.nuget.org:443/v3/registration5-gz-semver2/messagepack/index.json
2024-09-16T19:41:29.8913078Z   proxy | 2024/09/16 19:41:29 [500] 200 https://api.nuget.org:443/v3/registration5-gz-semver2/messagepack/index.json
2024-09-16T19:41:29.9046322Z updater | 2024/09/16 19:41:29 INFO <job_885578226> Filtered out 29 pre-release versions
2024-09-16T19:41:30.0017308Z   proxy | 2024/09/16 19:41:30 [502] GET https://api.nuget.org:443/v3-flatcontainer/messagepack/2.2.60/messagepack.nuspec
2024-09-16T19:41:30.0038706Z   proxy | 2024/09/16 19:41:30 [502] 200 https://api.nuget.org:443/v3-flatcontainer/messagepack/2.2.60/messagepack.nuspec
2024-09-16T19:41:30.0088939Z updater | 2024/09/16 19:41:30 INFO <job_885578226> Latest version is 2.5.172
2024-09-16T19:41:30.0092307Z updater | 2024/09/16 19:41:30 INFO <job_885578226> Requirements to unlock all
2024-09-16T19:41:30.0093019Z 2024/09/16 19:41:30 INFO <job_885578226> Requirements update strategy 
2024-09-16T19:41:30.0093592Z updater | Finding updated dependencies for MessagePack.
2024-09-16T19:41:30.1055766Z   proxy | 2024/09/16 19:41:30 [504] GET https://api.nuget.org:443/v3-flatcontainer/messagepack/2.5.172/messagepack.nuspec
2024-09-16T19:41:30.1075099Z   proxy | 2024/09/16 19:41:30 [504] 200 https://api.nuget.org:443/v3-flatcontainer/messagepack/2.5.172/messagepack.nuspec
2024-09-16T19:41:30.2096633Z   proxy | 2024/09/16 19:41:30 [506] GET https://api.nuget.org:443/v3-flatcontainer/messagepack.annotations/2.5.172/messagepack.annotations.nuspec
2024-09-16T19:41:30.2120579Z   proxy | 2024/09/16 19:41:30 [506] 200 https://api.nuget.org:443/v3-flatcontainer/messagepack.annotations/2.5.172/messagepack.annotations.nuspec
2024-09-16T19:41:30.3137128Z   proxy | 2024/09/16 19:41:30 [508] GET https://api.nuget.org:443/v3-flatcontainer/microsoft.net.stringtools/17.6.3/microsoft.net.stringtools.nuspec
2024-09-16T19:41:30.3155672Z   proxy | 2024/09/16 19:41:30 [508] 200 https://api.nuget.org:443/v3-flatcontainer/microsoft.net.stringtools/17.6.3/microsoft.net.stringtools.nuspec
2024-09-16T19:41:30.4217707Z   proxy | 2024/09/16 19:41:30 [510] GET https://api.nuget.org:443/v3-flatcontainer/system.collections.immutable/6.0.0/system.collections.immutable.nuspec
2024-09-16T19:41:30.4261155Z   proxy | 2024/09/16 19:41:30 [510] 200 https://api.nuget.org:443/v3-flatcontainer/system.collections.immutable/6.0.0/system.collections.immutable.nuspec
2024-09-16T19:41:30.5297824Z   proxy | 2024/09/16 19:41:30 [512] GET https://api.nuget.org:443/v3-flatcontainer/system.reflection.emit.lightweight/4.7.0/system.reflection.emit.lightweight.nuspec
2024-09-16T19:41:30.5471419Z   proxy | 2024/09/16 19:41:30 [512] 200 https://api.nuget.org:443/v3-flatcontainer/system.reflection.emit.lightweight/4.7.0/system.reflection.emit.lightweight.nuspec
2024-09-16T19:41:30.5565570Z updater | 2024/09/16 19:41:30 INFO <job_885578226> Updating MessagePack from 2.2.60 to 2.5.172
2024-09-16T19:41:31.0060621Z   proxy | 2024/09/16 19:41:31 [514] POST /update_jobs/885578226/record_update_job_unknown_error
2024-09-16T19:41:31.0641532Z   proxy | 2024/09/16 19:41:31 [514] 204 /update_jobs/885578226/record_update_job_unknown_error
2024-09-16T19:41:31.1095191Z   proxy | 2024/09/16 19:41:31 [516] POST /update_jobs/885578226/record_update_job_error
2024-09-16T19:41:31.1096652Z   proxy | 2024/09/16 19:41:31 [516] 204 /update_jobs/885578226/record_update_job_error
2024-09-16T19:41:31.1577255Z   proxy | 2024/09/16 19:41:31 [518] POST /update_jobs/885578226/increment_metric
2024-09-16T19:41:31.1578694Z 2024/09/16 19:41:31 [518] 204 /update_jobs/885578226/increment_metric
2024-09-16T19:41:31.1629892Z   proxy | 2024/09/16 19:41:31 [520] POST /update_jobs/885578226/record_update_job_unknown_error
2024-09-16T19:41:31.1631266Z   proxy | 2024/09/16 19:41:31 [520] 204 /update_jobs/885578226/record_update_job_unknown_error
2024-09-16T19:41:31.1641654Z updater | 2024/09/16 19:41:31 ERROR <job_885578226> Error processing MessagePack (Dependabot::DependabotError)

So, not too helpful with all these unknowns, and the listed packages don't seem to be related to the changes under https://github.com/OrchardCMS/OrchardCore/pull/16566.

MessagePack is NOT a package we directly reference, BTW.

The last PR that Dependabot opened was https://github.com/OrchardCMS/OrchardCore/pull/16549 on 12 August, which interestingly originates from a failing run (its successful rerun didn't actually do anything since "Dependabot workflows cannot be re-run. Retrigger this update via Dependabot instead.").

My guess is that Dependabot is simply choking on the update due to us having a huge solution with an extreme amount of packages being referenced, and we occasionally have to do manual updates. I've updated a GitHub support request about this.

[MikeAlhayek]

this is one fixed.

[Piedone]

GitHub support only pointed to the docs for now BTW, what I've previously read but didn't help.

[Piedone]

Unfortunately, this isn't fixed. Dependabot has a limitation of 150 manifests per repo: https://docs.github.com/en/enterprise-cloud@latest/code-security/supply-chain-security/understanding-your-software-supply-chain/troubleshooting-the-dependency-graph#are-there-limits-which-affect-the-dependency-graph-data According to GitHub support, we have 190 (we have 217 projects in the solution but apparently, not all of those have NuGet dependencies).

We could perhaps batch updates by having multiple updates, scheduled hours apart, in dependabot.yml, with different directory patterns to match a roughly even number of projects (or at least less than 150) each. Or a similar strategy but with dependency name patterns.

[sebastienros]

We can probably create a GH action for our own needs. With the latest SDKs there is a dotnet tool that does the same thing. I could run dotnet list package --outdated to list all the potential updates, you can ask for minor versions only. I am almost sure there is another command in the recent SDK version to update the files too. Worst case we could create a script out of the json result of the command I shared.

@MikeAlhayek suggested we only update the vulnerable versions. My preference is to update all the possible version once a week, to prevent getting in the same state as before where we would get manual updates of all packages several times a week.

[github-actions[bot]]

We triaged this issue and set the milestone according to the priority we think is appropriate (see the docs on how we triage and prioritize issues).

This indicates when the core team may start working on it. However, if you'd like to contribute, we'd warmly welcome you to do that anytime. See our guide on contributions here.

[sebastienros]

not all of those have NuGet dependencies

Maybe these are JS dependencies we could remove from dependabot?

[Piedone]

I think that's already the case. In Dependabot terminology, everything that may reference a package is a "manifest", I think. I.e. package.json for NPM, and csproj for NuGet. Since we have more csprojs than manifests, this checks out.

However, I think we can have a sort of batching. See: https://github.com/OrchardCMS/OrchardCore/pull/16820

[Piedone]

Not yet fixed, see: https://github.com/OrchardCMS/OrchardCore/pull/16820#issuecomment-2392452285

[Piedone]

https://github.com/OrchardCMS/OrchardCore/pull/16827 should be a good fix now.

[Piedone]

Argh: https://github.com/OrchardCMS/OrchardCore/actions/runs/11178578942 update_not_possible, for libphonenumber-csharp, no details. But why? This should be a trivial update. I'm updating this manually once, and then we'll see if Dependabot will work, but I'm afraid that it won't.

Also opened https://github.com/dependabot/dependabot-core/issues/10728 about having descriptive error messages in such cases.

[Piedone]

Still not working: https://github.com/OrchardCMS/OrchardCore/actions/runs/11203765523/job/31141396767 I'm out of ideas why. Waiting for the reply of GitHub support.

[Piedone]

Sebastien started to work on a custom .NET CLI implementation instead of Dependabot here: https://github.com/OrchardCMS/OrchardCore/tree/sebros/outdated In case we can't use Dependabot, what appears the be the case currently (but I'm still waiting for GitHub support), we'll go with that.

[Piedone]

The ball is still in GitHub support's court. They're investigating.

[Piedone]

We could also use https://github.com/renovatebot/renovate instead.