cyberblast
commented
9 months ago hmm not sure why it is using such large fonts in the end. sorry it's unintentional...
Closed cyberblast closed 4 months ago
cyberblast
commented
9 months ago hmm not sure why it is using such large fonts in the end. sorry it's unintentional...
JensSchadron
commented
9 months ago Just for reference, this is related to #919
cyberblast
commented
9 months ago It seems to only happen with dotnet/nuget projects. Our frontend/npm repos run fine.
I temporarily set dependabot task to run 1.24 image for now to have everything fully functional again.
dockerImageTag: '1.24'
Also, regarding that comment:
I also recognized, that it is suddenly fetching > 1000 nuget dependency files. Where as before it were only like ~90.
I recognized that I only counted log lines, so real number of requests would be half of it I guess... Not sure if it matters a lot.
davesmits
commented
9 months ago Not sure if related but we get tihs error
/usr/local/lib/ruby/3.1.0/openssl/buffering.rb:214:in `sysread_nonblock': SSL_read: unexpected eof while reading (OpenSSL::SSL::SSLError)
from /usr/local/lib/ruby/3.1.0/openssl/buffering.rb:214:in `read_nonblock'
from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/excon-0.104.0/lib/excon/socket.rb:209:in `read_nonblock'
from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/excon-0.104.0/lib/excon/socket.rb:79:in `block in readline'
from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/excon-0.104.0/lib/excon/socket.rb:70:in `loop'
from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/excon-0.104.0/lib/excon/socket.rb:70:in `readline'
from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/excon-0.104.0/lib/excon/response.rb:73:in `block in parse'
from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/excon-0.104.0/lib/excon/response.rb:72:in `loop'
from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/excon-0.104.0/lib/excon/response.rb:72:in `parse'
from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/excon-0.104.0/lib/excon/middlewares/response_parser.rb:7:in `response_call'
from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/excon-0.104.0/lib/excon/connection.rb:460:in `response'
from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/excon-0.104.0/lib/excon/connection.rb:291:in `request'
from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/update_checker/nupkg_fetcher.rb:98:in `block in fetch_stream'
from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/update_checker/nupkg_fetcher.rb:90:in `loop'
from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/update_checker/nupkg_fetcher.rb:90:in `fetch_stream'
from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/update_checker/nupkg_fetcher.rb:45:in `fetch_nupkg_buffer_from_repository'
from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/update_checker/nuspec_fetcher.rb:31:in `fetch_nuspec_from_repository'
from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/update_checker/nuspec_fetcher.rb:17:in `block in fetch_nuspec'
from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/update_checker/nuspec_fetcher.rb:16:in `each'
from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/update_checker/nuspec_fetcher.rb:16:in `reduce'
from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/update_checker/nuspec_fetcher.rb:16:in `fetch_nuspec'
from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/update_checker/dependency_finder.rb:172:in `fetch_dependencies'
from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/update_checker/dependency_finder.rb:139:in `fetch_transitive_dependencies_impl'
from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/update_checker/dependency_finder.rb:134:in `fetch_transitive_dependencies'
from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/update_checker/dependency_finder.rb:39:in `transitive_dependencies'
from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/file_parser/project_file_parser.rb:183:in `block in transitive_dependencies_from_packages'
from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/file_parser/project_file_parser.rb:178:in `each'
from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/file_parser/project_file_parser.rb:178:in `transitive_dependencies_from_packages'
from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/file_parser/project_file_parser.rb:172:in `add_transitive_dependencies_from_packages'
from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/file_parser/project_file_parser.rb:131:in `add_transitive_dependencies'
from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/file_parser/project_file_parser.rb:104:in `parse_dependencies'
from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/file_parser/project_file_parser.rb:58:in `dependency_set'
from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/file_parser.rb:39:in `block in project_file_dependencies'
from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/file_parser.rb:37:in `each'
from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/file_parser.rb:37:in `project_file_dependencies'
from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-nuget-0.239.0/lib/dependabot/nuget/file_parser.rb:25:in `parse'
from bin/update_script.rb:521:in `<main>'
mburumaxwell
commented
9 months ago As @JensSchadron has stated, this is indeed an issue resulting from the merge of #919 but it was something on hold for a while. See comment. The issue is only affecting NuGet when private registries/feeds are in use. All other scenarios and ecosystems seem to continue working as usual.
The Microsoft team decided to overhaul the way NuGet works in dependabot so that it supports the new GitHub Advanced Security for Azure DevOps in PR #8179.
How come it does not affects the GitHub-hosted version, you asked? Well, there are two things I can think of:
First some context:
Hopefully, this gets fixed in the core repo. The solution is probably similar to that for PNPM and it's open for anyone to give a try.
Secretly, I am hoping that this break would result in Azure DevOps getting native support and this extension being killed. It's better for everyone (or so I think).
Before a solution is found, you can do something.
If you are using the server, set the dockerImageTag parameter to 1.24.
If you are using the extension:
- task: dependabot@1
inputs:
dockerImageTag: '1.24'
// your other inputs here ...This will exclude some new changes such as HTML in the PR body and the fix for #730
I hope this gives the much needed clarity.
davesmits
commented
9 months ago thanks @mburumaxwell ; using 1.24 for now worked.
mburumaxwell
commented
9 months ago Tracking https://github.com/dependabot/dependabot-core/issues/8597 which will possibly be fixed by https://github.com/dependabot/dependabot-core/pull/8679
mburumaxwell
commented
9 months ago I am tracking fixes in the core repo, such as https://github.com/dependabot/dependabot-core/pull/8748, then adding them to #927 for testing purposes. If anyone is interested in helping me test you can check out https://github.com/tinglesoftware/dependabot-azure-devops/pull/927#issuecomment-1888560706 and subscribe to the PR for updates.
cyberblast
commented
9 months ago Not sure, but according to this MS article, the problem could be that for v3 nuget protocol at Azure DevOps Artifact Feed it is not sufficient to assemble a nuget.config containing the PAT (as done here by dependabot-core), but Azure Artifacts Credential Provider needs to be used...
mburumaxwell
commented
9 months ago Not sure, but according to this MS article, the problem could be that for v3 nuget protocol at Azure DevOps Artifact Feed it is not sufficient to assemble a nuget.config containing the PAT (as done here by dependabot-core), but Azure Artifacts Credential Provider needs to be used...
This is an interesting find. Thank you. Seeing that Microsoft (Azure DevOps) guys are the ones that changed the implementation, I hope they can either rollback or fix it.
cyberblast
commented
9 months ago Or we add the nuget plugin to the image and provide it the PAT via env variable as described here.
mburumaxwell
commented
9 months ago Or we add the nuget plugin to the image and provide it the PAT via env variable as described here.
@cyberblast can you contribute a PR for that?
cyberblast
commented
9 months ago Not sure if I can manage to do that.
If anybody can support with that, it would be highly appreciated. In case I'm wrong and I can manage to find some time to set something up which seems to work, I'll definitely let you know...
cyberblast
commented
8 months ago I found some piece of dockerfile code which our developers are usually using to create images for our own apps. Maybe this could help somebody when looking into the issue...
ARG PAT
ENV NUGET_CREDENTIALPROVIDER_SESSIONTOKENCACHE_ENABLED true
ENV VSS_NUGET_EXTERNAL_FEED_ENDPOINTS '{"endpointCredentials":[{"endpoint":"https://pkgs.dev.azure.com/<org>/_packaging/<feed>/nuget/v3/index.json","username":"docker","password":"'${PAT}'"}]}'
RUN curl --proto "=https" --tlsv1.2 -sSf -L https://raw.githubusercontent.com/Microsoft/artifacts-credprovider/master/helpers/installcredprovider.sh | sh
COPY ./nuget.config ./I guess the missing piece would be to assemble the correct endpointCredentials json format from dependabot config and set as VSS_NUGET_EXTERNAL_FEED_ENDPOINTS before resolving dependencies .. or something like that
JensSchadron
commented
8 months ago I found some piece of dockerfile code which our developers are usually using to create images for our own apps. Maybe this could help somebody when looking into the issue...
ARG PAT ENV NUGET_CREDENTIALPROVIDER_SESSIONTOKENCACHE_ENABLED true ENV VSS_NUGET_EXTERNAL_FEED_ENDPOINTS '{"endpointCredentials":[{"endpoint":"https://pkgs.dev.azure.com/<org>/_packaging/<feed>/nuget/v3/index.json","username":"docker","password":"'${PAT}'"}]}' RUN curl --proto "=https" --tlsv1.2 -sSf -L https://raw.githubusercontent.com/Microsoft/artifacts-credprovider/master/helpers/installcredprovider.sh | sh COPY ./nuget.config ./I guess the missing piece would be to assemble the correct endpointCredentials json format from dependabot config and set as VSS_NUGET_EXTERNAL_FEED_ENDPOINTS before resolving dependencies .. or something like that
Just linking these here as I think they're related: https://github.com/dependabot/dependabot-core/pull/8927 https://github.com/dependabot/dependabot-core/pull/9004
mburumaxwell
commented
8 months ago May be fixed by https://github.com/dependabot/dependabot-core/pull/8927
Patrick-3000
commented
8 months ago This seems to have fixed the issue for me.
evgenyvalavin
commented
7 months ago Was the fix released? Ho to get it working with the latest dependabot version?
Patrick-3000
commented
7 months ago Not really sure anymore what is happening: the day before yesterday it worked. Yesterday our dependabot pipeline failed with
dotnet build in GetAllPackageDependenciesAsync failed. STDOUT: MSBuild version 17.8.3+195e7f5a3 for .NET`Today it worked again.
And today I see this error in a different dependabot pipeline of ours, which worked yesterday.
Now I am in the situation that I can`t use dependabot for nuget packages due to this issue, and neither for npm packages due to #729 :worried:
Blackunknown
commented
7 months ago I just updated the image version from 1.24 to 1.26.4 because suddenly we were no longer getting work items linked to the PRs created. However 1.26.4 seems to give an error accessing the private feed we have like so:
/usr/local/dotnet/current/sdk/8.0.100/NuGet.targets(156,5): error : Unable to load the service index for source https://*.pkgs.visualstudio.com/_packaging/*.*/nuget/v3/index.json. [/tmp/package-dependency-resolution_DPiBKt/Project.csproj] /usr/local/dotnet/current/sdk/8.0.100/NuGet.targets(156,5): error : Response status code does not indicate success: 401 (Unauthorized). [/tmp/package-dependency-resolution_DPiBKt/Project.csproj] 0 Warning(s) 371 Error(s)
If anyone has any idea how we can get the private feed to work that would be great. Seems this issue is still actively preventing people from going beyond 1.24.
Sieberkev
commented
7 months ago Downgrading from 1.26.671 -> 1.24 "fixed" our pipeline (Telerik nuget feed gave authentication errors), hoping on a permanent fix :)
Thulasi225
commented
6 months ago I just updated the image version from 1.24 to 1.26.4 because suddenly we were no longer getting work items linked to the PRs created. However 1.26.4 seems to give an error accessing the private feed we have like so:
/usr/local/dotnet/current/sdk/8.0.100/NuGet.targets(156,5): error : Unable to load the service index for source https://*.pkgs.visualstudio.com/_packaging/*.*/nuget/v3/index.json. [/tmp/package-dependency-resolution_DPiBKt/Project.csproj] /usr/local/dotnet/current/sdk/8.0.100/NuGet.targets(156,5): error : Response status code does not indicate success: 401 (Unauthorized). [/tmp/package-dependency-resolution_DPiBKt/Project.csproj] 0 Warning(s) 371 Error(s)
If anyone has any idea how we can get the private feed to work that would be great. Seems this issue is still actively preventing people from going beyond 1.24.
Rutix
commented
5 months ago @mburumaxwell it's a bit hard to see which issue I should monitor. Is this still broken?
Blackunknown
commented
5 months ago We are actually deleting dependabot because of this issue. Because we can't update so linking work items works again. It was a nice idea to save us some work but in the end leads to more work as it is now
Rutix
commented
5 months ago We are actually deleting dependabot because of this issue. Because we can't update so linking work items works again. It was a nice idea to save us some work but in the end leads to more work as it is now
What updated that broke linking work items? It's still works on my side with 1.24 as imageTag
mburumaxwell
commented
5 months ago @mburumaxwell it's a bit hard to see which issue I should monitor. Is this still broken?
I am not sure if it still is broken. Haven't used this in a while. The challenge with this is how much the NuGet space was changing. It is still not back to what it was before Microsoft overhauled it. On GitHub there are still issues like the whole update failing because of multiple target frameworks in Directory.Build.props or timing out for big repository (I have one with 50 projects in it). Overall, the changes in typings are quite a number so I kept off till it stabilizes to focus on where my time is better spent than chasing my own rail.
We are actually deleting dependabot because of this issue. Because we can't update so linking work items works again. It was a nice idea to save us some work but in the end leads to more work as it is now.
Deleting the use of Dependabot in your repository is one way to go about it. There are better options like contributing a fix, pressurize Microsoft to bring official support, migrate to GitHub, or keep using 1.24.
rhyskoedijk
commented
5 months ago 1.24 works fine for me too, including work item linking.
@Blackunknown it might be worth double checking that the user account Dependabot is running as has read access to the project and can read work items as originally I had your same issue but it started working after giving the user account project read access.
Despite the regression in dependabot-core, this project is still huge time saver for me and I really appreciate all your work @mburumaxwell.
rgrace-puck
commented
5 months ago Are there any considerations when pinning the 1.24 version? E.g. is the image going to be retained for the foreseeable future?
ryangribble
commented
4 months ago We just set this up for the first time yesterday (Azure DevOps repos, private Azure DevOps nuget feed) and naturally ran into various of these problems. Pinning back to 1.24 did get things to work at least - thanks for the issues and discussions here for providing that information.
However we then wanted to implement "grouping" of dependabot updates, and the settings didn't seem to be working no matter how we configured them in the dependabot.yml file.
We then discovered that grouped updates for nuget was also broken and only "fixed" recently in upstream dependabot-core here - AFTER the above extension version 1.2.4 (roughly around 1.27 of this extension I believe)... however due to the private feed nuget auth issue it seems there is no workable solution if you want grouping of updates?
So I guess I would say one consideration when pinning to 1.24 is that you wont get "grouping"
It's a bit hard to follow all the open issues etc so Im not sure where to look to understand if this is a known issue and there is a PR with proposed fixes we could help test on our setups. Do we just keep trying latest in our dockerImageTag setting every now and then? Will that automatically inlcude the latest from dependabot-core or is it only the latest of the extension and the version of dependabot is still set somewhere else?
DaleMckeown
commented
4 months ago Today my dependabot pipelines are failing even on 1.24. Can anyone confirm whether theirs are working please?
SeMuell
commented
4 months ago @DaleMckeown worked 1h ago for me with a private nuget repository.
Rutix
commented
4 months ago Today my dependabot pipelines are failing even on
1.24. Can anyone confirm whether theirs are working please?
We run this periodically and the last time was 5 AM and all our repositories worked :)
DaleMckeown
commented
4 months ago Cheers both, I'll check over my config. I might have broken something else.
DaleMckeown
commented
4 months ago @SeMuell @Rutix Would you be able to tell me how you're setting the PAT token? We currently have it set in a template file (which used to work) but now the auth docs specify that it doesn't work from template variables.
Are you creating the PAT as a variable group in DevOps -> Pipelines -> Libraries, as an environment variable (not sure how that is done), or adding them to each pipeline directly?
mburumaxwell
commented
4 months ago Closing this, everything should be working fine now, especially with version 1.29.0. Maybe some private feeds may not be working if they need https://github.com/dependabot/dependabot-core/pull/8927 to be resolved. Let us use that issue to track that particular behaviour/problem.
LockTar
commented
4 months ago For me updating to 1.29 worked with a private Azure DevOps artifact feed.
rhyskoedijk
commented
4 months ago Same, 1.29 works much better now.
Maybe some private feeds may not be working if they need dependabot/dependabot-core#8927 to be resolved. Let us use that issue to track that particular behaviour/problem.
If you are having the issue where the NuGet/MSBuild/dotnet tools are failing with "error NU1301: Unable to load the service index for source" then the below workaround resolved it for me.
EDIT: This workaround is not required if using v1.30.1 or higher. See: https://github.com/tinglesoftware/dependabot-azure-devops/pull/1233.
This workaround hijacks the fix for #834 to install the NuGet credential provider, then injects the creds via environment variables. This ensures that dotnet CLI tools are able to auth with the NuGet feed. It's really hacky, but it works.
- task: dependabot@1
inputs:
dockerImageTag: '1.29'
...snip...
extraEnvironmentVariables: 'WORKAROUND_CMD=sh -c "$(curl -fsSL https://aka.ms/install-artifacts-credprovider.sh)";NUGET_CREDENTIALPROVIDER_SESSIONTOKENCACHE_ENABLED=true;VSS_NUGET_EXTERNAL_FEED_ENDPOINTS={"endpointCredentials":[{"endpoint":"https://pkgs.dev.azure.com/...your_feed_slug.../nuget/v3/index.json","username":"unused","password":"$(System.AccessToken)"}]}'
@mburumaxwell I'm not sure if this is a dependabot-core issue or not, but from my limited understanding auth should be passed from dependabot down to the dotnet CLI tools right? it doesn't appear to do this correctly for me and it seems like https://github.com/dependabot/dependabot-core/pull/8927 would remove the need to do this workaround right?
mburumaxwell
commented
4 months ago @rhyskoedijk this is a perfect workaround. Hadn't envisioned the PR by @BobSilent being of use this way. Guess it will stick around indefinitely.
What you have done in the workaround is exactly what https://github.com/dependabot/dependabot-core/pull/8927 intends to do. From what I understand, this only affects NuGet feeds hosted by Azure DevOps. The hosted version of dependabot handles authentication quite differently and this might be why they haven't seen the issue and needed to fix it. IIRC, they do not pass credentials to the job but instead inject them on outgoing requests that match the feed which is why it works on GitHub but not on other self-hosted places. This is must be what is being explained in https://github.com/dependabot/dependabot-core/pull/8927#issuecomment-2161233519 This authentication injection functionality is not public for anyone to probe or copy.
One way would have been to override settings when building docker images but that can't be generalized for everyone given the feed URL needs to be known before hand. (Forgetting the extras scoped just to the NuGet ecosystem).
I may be a bit wrong with my approach/understanding but before moving to doing updates with NuGet/dotnet CLI, credentials worked just fine. They still do for other ecosystems unless in cases where one is fighting with the Azure DevOps permissions mess.
My take is keep running the workaround and simultaneously apply pressure on the PR to be merged.
cyberblast
commented
4 months ago Marvelous. Works well with the workaround as provided by @rhyskoedijk. Thanks a lot for that one! 😃
Unfortunately, now I have errors in all npm based repos as described in #1046 (uninitialized constant RequirementsUpdateStrategy). But seems thats a different issue again...
Anyway, thanks for solving it that far, first time it runs without pinning to 1.24 😄
A bit sad that it's so slow now. With 1.24 it took about 65 - 75 minutes for dependabot to crawl over our repos (same job iterating dependabot tasks over ~100 repos). Now with 1.29, current run is already at 100 minutes and only half of repos covered 😞 I have to bump job timeout ...
cyberblast
commented
4 months ago Pinning it back to 1.24 as sometimes it keeps hanging on a task until the job times out (after 300 minutes), keeping subsequent repos unprocessed.
Plus npm repos not working with latest image due to other issues.
Describe the bug
Hi, we are having sudden issues with Dependabot in Azure DevOps since 1.25. We did not change anything on the setup but it is simply failing.
Here's some excerpt from the Pipe log:
I also recognized, that it is suddenly fetching > 1000 nuget dependency files. Where as before it were only like ~90.
Eventually the pipe has been cancelled by DevOps due to long run execution.
I also doublechecked permission of build service on the private Artifact stream (due to the 401) and it is unchanged and should work as before. But as seen in the log excerpt, it is also failing while checking for a public package and we are not maintaining upstream in our private artifact feed but fetching public packages directly from nuget instead.
To Reproduce
We are running Dependabot nightly like this:
With a git repo in Azure DevOps and this configuration:
Expected behavior We did not change anything. It should simply run as before.
Screenshots -
Extension (please complete the following information):
Server (please complete the following information): -
Additional context