Error on converting ASP.NET MVC 5 to ASP.NET CORE MVC projet, due to NuGet proxy error

IgalOre commented 2 years ago

Error occurs during conversion of ASP.NET MVC 5 .Net 4.8 project to ASP.NET Core with upgrade-assistant, related to project NuGet packages restore, since not allowed access to NuGet.Org directly but throw proxy.

[ upgrade-assistant.zip ] Include the log file upgrade-assistant.clef that is produced in the working directory

Describe the bug

During conversion of .Net ASP.NET MVC 5 project an exception occuring mentioning [10:59:23 ERR] Unexpected error NuGet.Protocol.Core.Types.FatalProtocolException: Unable to load the service index for source https://api.nuget.org/v3/index.json. ---> System.Net.Http.HttpRequestException: The proxy tunnel request to proxy 'http://proxy.services.XXXX:3128/' failed with status code '407'." More detailed exception stack-trace are bellow.

During migration of same solution, at different projects next warnings had been appearing :

[09:45:47 INF] Initializing upgrade step Upgrade assistant reference analyzer [09:45:47 WRN] Failed to get package versions from source https://api.nuget.org/v3/index.json due to a NuGet protocol error [09:45:47 INF] If NuGet packages are coming from an authenticated source, Upgrade Assistant requires a .NET Core-compatible v2 credential provider be installed. To authenticate with an Azure DevOps NuGet source, for example, see https://github.com/microsoft/artifacts-credprovider#setup [09:45:47 WRN] .NET Upgrade Assistant analyzer NuGet package reference cannot be added because the package cannot be found

[09:45:47 INF] Initializing upgrade step Windows Compatibility Pack Analyzer [09:45:48 WRN] Failed to get package versions from source https://api.nuget.org/v3/index.json due to a NuGet protocol error [09:45:48 INF] If NuGet packages are coming from an authenticated source, Upgrade Assistant requires a .NET Core-compatible v2 credential provider be installed. To authenticate with an Azure DevOps NuGet source, for example, see https://github.com/microsoft/artifacts-credprovider#setup [09:45:48 WRN] Could not find null

microsoft/artifacts-credprovider are installed and configured as explained at artifacts-credprovider project

Exceptions (if any)

[10:59:23 ERR] Unexpected error NuGet.Protocol.Core.Types.FatalProtocolException: Unable to load the service index for source https://api.nuget.org/v3/index.json. ---> System.Net.Http.HttpRequestException: The proxy tunnel request to proxy 'http://proxy.services.rq:3128/' failed with status code '407'." at System.Net.Http.HttpConnectionPool.EstablishProxyTunnelAsync(Boolean async, HttpRequestHeaders headers, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request) at System.Threading.Tasks.TaskCompletionSourceWithCancellation1.WaitWithCancellationAsync(CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.AuthenticationHelper.SendWithAuthAsync(HttpRequestMessage request, Uri authUri, Boolean async, ICredentials credentials, Boolean preAuthenticate, Boolean isProxyAuth, Boolean doRequestAuth, HttpConnectionPool pool, CancellationToken cancellationToken) at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at NuGet.Protocol.ServerWarningLogHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at NuGet.Protocol.HttpSourceAuthenticationHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) at NuGet.Protocol.HttpRetryHandler.<>c__DisplayClass5_1.<<SendAsync>b__0>d.MoveNext() --- End of stack trace from previous location --- at NuGet.Protocol.TimeoutUtility.StartWithTimeout[T](Func2 getTask, TimeSpan timeout, String timeoutMessage, CancellationToken token) at NuGet.Protocol.HttpRetryHandler.SendAsync(HttpRetryHandlerRequest request, String source, ILogger log, CancellationToken cancellationToken) at NuGet.Protocol.HttpSource.GetThrottledResponse(Func1 requestFactory, TimeSpan requestTimeout, TimeSpan downloadTimeout, Int32 maxTries, Boolean isRetry, Boolean isLastAttempt, Guid sessionId, ILogger log, CancellationToken cancellationToken) at NuGet.Protocol.HttpSource.<>c__DisplayClass15_01.<b0>d.MoveNext() --- End of stack trace from previous location --- at NuGet.Common.ConcurrencyUtilities.ExecuteWithFileLockedAsync[T](String filePath, Func2 action, CancellationToken token) at NuGet.Common.ConcurrencyUtilities.ExecuteWithFileLockedAsync[T](String filePath, Func2 action, CancellationToken token) at NuGet.Protocol.HttpSource.GetAsync[T](HttpSourceCachedRequest request, Func`2 processAsync, ILogger log, CancellationToken token) at NuGet.Protocol.ServiceIndexResourceV3Provider.GetServiceIndexResourceV3(SourceRepository source, DateTime utcNow, ILogger log, CancellationToken token) --- End of inner exception stack trace --- at NuGet.Protocol.ServiceIndexResourceV3Provider.GetServiceIndexResourceV3(SourceRepository source, DateTime utcNow, ILogger log, CancellationToken token) at NuGet.Protocol.ServiceIndexResourceV3Provider.TryCreate(SourceRepository source, CancellationToken token) at NuGet.Protocol.Core.Types.SourceRepository.GetResourceAsync[T](CancellationToken token) at NuGet.Protocol.Core.Types.SourceRepository.GetResourceAsync[T]() at NuGet.Protocol.HttpFileSystemBasedFindPackageByIdResourceProvider.TryCreate(SourceRepository sourceRepository, CancellationToken token) at NuGet.Protocol.Core.Types.SourceRepository.GetResourceAsync[T](CancellationToken token) at NuGet.Protocol.Core.Types.SourceRepository.GetResourceAsync[T]() at NuGet.Commands.SourceRepositoryDependencyProvider.EnsureResource() at NuGet.Commands.SourceRepositoryDependencyProvider.FindLibraryCoreAsync(LibraryRange libraryRange, SourceCacheContext cacheContext, ILogger logger, CancellationToken cancellationToken) at NuGet.Commands.SourceRepositoryDependencyProvider.<>cDisplayClass19_0.<b0>d.MoveNext() --- End of stack trace from previous location --- at NuGet.Commands.SourceRepositoryDependencyProvider.FindLibraryAsync(LibraryRange libraryRange, NuGetFramework targetFramework, SourceCacheContext cacheContext, ILogger logger, CancellationToken cancellationToken) at NuGet.DependencyResolver.ResolverUtility.<>cDisplayClass9_1.<b0>d.MoveNext() --- End of stack trace from previous location --- at NuGet.DependencyResolver.ResolverUtility.FindLibraryFromSourcesAsync(LibraryRange libraryRange, IEnumerable1 providers, Func2 action) at NuGet.DependencyResolver.ResolverUtility.FindLibraryByVersionAsync(LibraryRange libraryRange, NuGetFramework framework, IEnumerable1 providers, SourceCacheContext cacheContext, ILogger logger, CancellationToken token) at NuGet.DependencyResolver.ResolverUtility.FindPackageLibraryMatchAsync(LibraryRange libraryRange, NuGetFramework framework, IEnumerable1 remoteProviders, IEnumerable1 localProviders, SourceCacheContext cacheContext, ILogger logger, CancellationToken cancellationToken) at NuGet.DependencyResolver.ResolverUtility.FindLibraryMatchAsync(LibraryRange libraryRange, NuGetFramework framework, String runtimeIdentifier, IEnumerable1 remoteProviders, IEnumerable1 localProviders, IEnumerable1 projectProviders, IDictionary2 lockFileLibraries, SourceCacheContext cacheContext, ILogger logger, CancellationToken cancellationToken) at NuGet.DependencyResolver.ResolverUtility.FindLibraryEntryAsync(LibraryRange libraryRange, NuGetFramework framework, String runtimeIdentifier, RemoteWalkContext context, CancellationToken cancellationToken) at NuGet.DependencyResolver.RemoteDependencyWalker.CreateGraphNode(LibraryRange libraryRange, NuGetFramework framework, String runtimeName, RuntimeGraph runtimeGraph, Func2 predicate, GraphEdge1 outerEdge, TransitiveCentralPackageVersions transitiveCentralPackageVersions) at NuGet.DependencyResolver.RemoteDependencyWalker.CreateGraphNode(LibraryRange libraryRange, NuGetFramework framework, String runtimeName, RuntimeGraph runtimeGraph, Func2 predicate, GraphEdge1 outerEdge, TransitiveCentralPackageVersions transitiveCentralPackageVersions) at NuGet.DependencyResolver.RemoteDependencyWalker.CreateGraphNode(LibraryRange libraryRange, NuGetFramework framework, String runtimeName, RuntimeGraph runtimeGraph, Func2 predicate, GraphEdge1 outerEdge, TransitiveCentralPackageVersions transitiveCentralPackageVersions) at NuGet.DependencyResolver.RemoteDependencyWalker.WalkAsync(LibraryRange library, NuGetFramework framework, String runtimeIdentifier, RuntimeGraph runtimeGraph, Boolean recursive) at NuGet.Commands.ProjectRestoreCommand.WalkDependenciesAsync(LibraryRange projectRange, NuGetFramework framework, String runtimeIdentifier, RuntimeGraph runtimeGraph, RemoteDependencyWalker walker, RemoteWalkContext context, CancellationToken token) at NuGet.Commands.ProjectRestoreCommand.TryRestoreAsync(LibraryRange projectRange, IEnumerable1 frameworkRuntimePairs, NuGetv3LocalRepository userPackageFolder, IReadOnlyList1 fallbackPackageFolders, RemoteDependencyWalker remoteWalker, RemoteWalkContext context, Boolean forceRuntimeGraphCreation, CancellationToken token, TelemetryActivity telemetryActivity, String telemetryPrefix) at NuGet.Commands.RestoreCommand.ExecuteRestoreAsync(NuGetv3LocalRepository userPackageFolder, IReadOnlyList1 fallbackPackageFolders, RemoteWalkContext context, CancellationToken token, TelemetryActivity telemetryActivity) at NuGet.Commands.RestoreCommand.ExecuteAsync(CancellationToken token) at NuGet.Commands.RestoreRunner.ExecuteAsync(RestoreSummaryRequest summaryRequest, CancellationToken token) at NuGet.Commands.RestoreRunner.CompleteTaskAsync(List1 restoreTasks) at NuGet.Commands.RestoreRunner.RunWithoutCommit(IEnumerable1 restoreRequests, RestoreArgs restoreContext) at Microsoft.DotNet.UpgradeAssistant.Extensions.NuGet.NuGetTransitiveDependencyIdentifier.RestoreProjectAsync(IEnumerable1 packages, IEnumerable1 tfms, CancellationToken token) at Microsoft.DotNet.UpgradeAssistant.Extensions.NuGet.NuGetTransitiveDependencyIdentifier.GetTransitiveDependenciesAsync(IEnumerable1 packages, IEnumerable1 tfms, CancellationToken token) at Microsoft.DotNet.UpgradeAssistant.Dependencies.TransitiveDependencyExtensions.IsTransitiveDependencyAsync(ITransitiveDependencyIdentifier identifier, String packageName, IEnumerable1 packages, IEnumerable1 tfms, CancellationToken token) in //src/common/Microsoft.DotNet.UpgradeAssistant.Abstractions/Dependencies/TransitiveDependencyExtensions.cs:line 110 at Microsoft.DotNet.UpgradeAssistant.Extensions.Windows.WindowsComponentIdentifier.IsWinRt(IProject project, CancellationToken token) at Microsoft.DotNet.UpgradeAssistant.Extensions.Windows.WindowsComponentIdentifier.GetComponentsAsync(IProject project, CancellationToken token) at Microsoft.DotNet.UpgradeAssistant.MSBuild.MSBuildProject.GetComponentsAsync(CancellationToken token) in //src/components/Microsoft.DotNet.UpgradeAssistant.MSBuild/MSBuildProject.cs:line 128 at Microsoft.DotNet.UpgradeAssistant.Extensions.Maui.MauiPlatformTargetFrameworkUpgradeStep.IsApplicableImplAsync(IUpgradeContext context, CancellationToken token) at Microsoft.DotNet.UpgradeAssistant.UpgradeStep.IsApplicableAsync(IUpgradeContext context, CancellationToken token) in /_/src/common/Microsoft.DotNet.UpgradeAssistant.Abstractions/UpgradeStep.cs:line 233 at Microsoft.DotNet.UpgradeAssistant.UpgraderManager.<>c__DisplayClass9_0.<b0>d.MoveNext() in //src/components/Microsoft.DotNet.UpgradeAssistant/UpgraderManager.cs:line 168 --- End of stack trace from previous location --- at System.Linq.AsyncEnumerable.WhereEnumerableAsyncIteratorWithTaskAndCancellation`1.MoveNextCore() in //Ix.NET/Source/System.Linq.Async/System/Linq/Operators/Where.cs:line 339 at System.Linq.AsyncIteratorBase1.MoveNextAsync() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 77 at System.Linq.AsyncIteratorBase1.MoveNextAsync() in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:line 77 at System.Linq.AsyncEnumerable.gCore|2070[TSource](IAsyncEnumerable1 source, Func2 predicate, CancellationToken cancellationToken) in //Ix.NET/Source/System.Linq.Async/System/Linq/Operators/All.cs:line 34 at System.Linq.AsyncEnumerable.gCore|2070[TSource](IAsyncEnumerable1 source, Func2 predicate, CancellationToken cancellationToken) in //Ix.NET/Source/System.Linq.Async/System/Linq/Operators/All.cs:line 34 at Microsoft.DotNet.UpgradeAssistant.UpgraderManager.GetNextStepAsync(IUpgradeContext context, CancellationToken token) in //src/components/Microsoft.DotNet.UpgradeAssistant/UpgraderManager.cs:line 99 at Microsoft.DotNet.UpgradeAssistant.Cli.ConsoleUpgrade.RunAsync(CancellationToken token) in //src/cli/Microsoft.DotNet.UpgradeAssistant.Cli/Commands/Upgrade/ConsoleUpgrade.cs:line 70 at Microsoft.DotNet.UpgradeAssistant.Cli.ConsoleUpgrade.RunAsync(CancellationToken token) in //src/cli/Microsoft.DotNet.UpgradeAssistant.Cli/Commands/Upgrade/ConsoleUpgrade.cs:line 78 at Microsoft.DotNet.UpgradeAssistant.Cli.ConsoleRunner.StartAsync(CancellationToken token) in //src/cli/Microsoft.DotNet.UpgradeAssistant.Cli/ConsoleRunner.cs:line 61

Further technical details

Windows version and bitness info (32-bit or 64-bit)
Windows 10 Entreprise, 20H2, 19042.1889 64-bit
Include the output of dotnet --list-sdks
PS C:\WINDOWS\system32> dotnet --list-sdks
2.1.521 [C:\Program Files\dotnet\sdk] 3.1.101 [C:\Program Files\dotnet\sdk] 6.0.107 [C:\Program Files\dotnet\sdk] 6.0.201 [C:\Program Files\dotnet\sdk] 6.0.202 [C:\Program Files\dotnet\sdk] 6.0.302 [C:\Program Files\dotnet\sdk]
Include the output of upgrade-assistant --version PS C:\WINDOWS\system32> upgrade-assistant --version 0.4.336902+3a2177acaa53d719e5da4ccde275f84ff18f0611

marijnz0r commented 1 year ago

Hi @IgalOre, I'm experiencing the same problem. Did you find a way to use the upgrade assistant with your NuGet proxy in the meantime? Kind regards

IgalOre commented 1 year ago

Unfortunately, no.

I got few ways to try, like : Recompiling upgrade-assistant with replacing all default NuGet recuperation URI to one that are allowed - this requires to have .Net Core 7 beta on machine to work out. To try. Another would be to try to redirect all calls towards NuGet URI to one that is needed throw Windows hosts file. I am uncertain if that could work, but one of my colleagues suggested trying it out. A bit skeptical about it.

Keep faith that someone else will find the solution or maintainers will suggest something. I had heard from someone working closely to Microsoft development team, that unfortunately upgrade-assitant do not receive enough attention due to geopolitical decisions. Not sure neither if It's true nor false.

Birgos commented 1 year ago

For me it looks like, that:

the upgrade-assistant seems to call the method GetPackageSources from class NuGetPackageSourceFactory with a NULL value for the parameter path, so packageSources.Count == 0 is true:

if (packageSources.Count == 0) { packageSources.Add(new PackageSource(DefaultPackageSource)); }

And the field DefaultPackageSource contains: https://api.nuget.org/v3/index.json

Machine-wide or user-wide settings seem to be unconsidered.

Therefore I tried the following quick and dirty. I changed the following line in the method GetPackageSources of the class NuGetPackageSourceFactory

   if (path != null)

to

   if (string.IsNullOrEmpty(path))

so the following paragraph takes action:

   if (string.IsNullOrEmpty(path))
   {
            var nugetSettings = Settings.LoadDefaultSettings(path);
            var sourceProvider = new PackageSourceProvider(nugetSettings);
            packageSources.AddRange(sourceProvider.LoadPackageSources().Where(e => e.IsEnabled));
   }

I've builded the solution locally and now the tool is able to determine the local nuget configs and add the proxy addresses to the list of packages sources. Because of the disabled nuget.org feed the tool uses only our proxies.

I guess path is null, because the method AddNuGet from the class NuGetExtensionBuilder calls the method GetPackageSources with a NULL value, because the property PackageSourcePath of NuGetDownloaderOptions isn't set. The following code line does the call:

services.AddSingleton(ctx => ctx.GetRequiredService<INuGetPackageSourceFactory>().GetPackageSources(ctx.GetRequiredService<IOptions<NuGetDownloaderOptions>>().Value.PackageSourcePath));

Perhaps someone can verify this and confirm or refute accordingly?

IgalOre commented 1 year ago

Thank you very much @Birgos for your suggestion. Hopefully, maintainers of the project will pick it up for testing. @brandonh-msft Any idea when this could occur?

IgalOre commented 1 year ago

Finally i got time to tests your modification @Birgos , and indeed it works as a charm locally, based on latest source as of 4 Novembre 2022. No more errors on modified locally upgrade-assistant related to NuGet repository discovery.

Will you create an PR request so the credit will go for you, or if you too busy can do it for you ?

Birgos commented 1 year ago

I have taken another closer look at the method GetPackageSources from the class NuGetPackageSourceFactory and it seems to me, that the if-condition from line 26 shall cover the logical fact that the parameter 'path' could be null and therefore has to be checked.

But parameter 'path' is just used once inside the condition to call Settings.LoadDefaultSettings(path);

This method at the end internals calls LoadUserSpecificSettings from the NuGet Settings class that checks the parameter 'root' for null and if so, set it to 'string.empty':

            if (root == null)
            {
                // Path.Combine is performed with root so it should not be null
                // However, it is legal for it be empty in this method
                root = String.Empty;
            } However, it is legal for it be empty in this method
                root = String.Empty;
            }

So I guess it would be easier and probably less irritating to remove the redundant if-condition from the 'GetPackagesSources' method instead of inverting the condition.

Thus if the parameter 'path' contains a value, loading of the NuGet configuration could nevertheless be done, starting from the specified 'path' - that namely would not happen if the condition will just be inverted - but exactly this might probably have been the original intention...

I would like to test this change before creating a pull request... ;-)

dotnet / upgrade-assistant