search

OctopusDeploy / Issues

| Public | Bug reports and known issues for Octopus Deploy and all related tools
https://octopus.com
162 stars 20 forks source link

Calamari fails to download packages on some Deployment Targets for NuGet feeds with GZip enabled on `index.json` #7267

Open paraicoceallaigh opened 2 years ago

paraicoceallaigh commented 2 years ago

Team

Severity

Blocking one customer - no real workaround

Version

2021.3.8275 Cloud

Latest Version

I could reproduce the problem in the latest build

What happened?

When attempting to download a package from a privately hosted NuGet feed, some Deployment Targets are running into an error where the index.json NuGet Catalogue file cannot be read.

Reproduction

  1. Host a private NuGet feed with a NuGet server version above 3.4 (GZip compression was added to the whole NuGet stack in 3.4)
  2. Configure Server with the NuGet feed in Library > External Feeds
  3. Install a Tentacle on a machine running .NET 4.0
  4. Create a project which deploys a package from the feed to the .NET 4.0 target

(.NET 4.0 vs dotnet core is our working hypothesis on the cause; not 100% - needs repro to prove)

Expected: Package should download correctly for any supported Tentacle/Calamari hosting combination Actual: NuGet downloader fails to read index.json and fails to download the package.

Error and Stacktrace

When running the download-package command:

C:\Octopus\Tools\Calamari.netfx\20.4.0\Calamari.exe download-package --packageId "delive
ryrecoveryservice" --packageVersion "2021.12.1.5422-ignore-inactive-account-suspension-notifications-6c322ee7" --feedUri
 "https://artifacts.postmark.ci.wildbit.com/nuget/builds/index.json" --feedUsername "redacted" --feedPassword "redacted" --feedId "Production"

Calamari fails with the following error:

Downloading NuGet package DeliveryRecoveryService vxxxxxxx from feed: 'https://xxxxxxxxx.xom/nuget/builds/index.json'
00:23:59   Verbose  |         Downloaded package will be stored in: 'C:\Octopus\Files\Feeds-1021'
00:23:59   Verbose  |         Downloading package (attempt 1 of 5)
00:24:00   Verbose  |         Attempt 1 of 5: Unexpected character encountered while parsing value: . Path '', line 0, position 0.
00:24:00   Verbose  |         Going to wait 10s before attempting the download from the external feed again.
00:24:10   Verbose  |         Downloading package (attempt 2 of 5)
00:24:10   Verbose  |         Attempt 2 of 5: Unexpected character encountered while parsing value: . Path '', line 0, position 0.
00:24:10   Verbose  |         Going to wait 20s before attempting the download from the external feed again.
00:24:31   Verbose  |         Downloading package (attempt 3 of 5)
00:24:31   Verbose  |         Attempt 3 of 5: Unexpected character encountered while parsing value: . Path '', line 0, position 0.
00:24:31   Verbose  |         Going to wait 30s before attempting the download from the external feed again.
00:25:01   Verbose  |         Downloading package (attempt 4 of 5)
00:25:01   Verbose  |         Attempt 4 of 5: Unexpected character encountered while parsing value: . Path '', line 0, position 0.
00:25:01   Verbose  |         Going to wait 40s before attempting the download from the external feed again.
00:25:42   Verbose  |         Downloading package (attempt 5 of 5)
00:25:42   Verbose  |         Attempt 5 of 5: Unexpected character encountered while parsing value: . Path '', line 0, position 0.
00:25:42   Error    |         Failed to download package xxxxxxxx from feed: 'https://xxxxxxx.com/nuget/builds/index.json'
00:25:42   Error    |         The package DeliveryRecoveryService version xxxxxxxxx could not be downloaded from the external feed 'https://xxxxxxxxx.com/nuget/builds/index.json' after making 5 attempts over a total of 101s. Make sure the package is pushed to the external feed and try the deployment again. For a detailed troubleshooting guide go to http://g.octopushq.com/TroubleshootMissingPackages
00:25:42   Error    |         Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: . Path '', line 0, position 0.
00:25:42   Error    |         at Newtonsoft.Json.JsonTextReader.ParseValue()
00:25:42   Error    |         at Newtonsoft.Json.Linq.JObject.Load(JsonReader reader, JsonLoadSettings settings)
00:25:42   Error    |         at Calamari.Integration.Packages.NuGet.NuGetV3Downloader.<>c__DisplayClass12_0.<GetJsonResponse>b__0(Stream stream)
00:25:42   Error    |         at Calamari.Integration.Packages.NuGet.NuGetV3Downloader.GetHttp(Uri uri, ICredentials credentials, TimeSpan httpTimeout, Action`1 processContent)
00:25:42   Error    |         at Calamari.Integration.Packages.NuGet.NuGetV3Downloader.GetJsonResponse(Uri feedUri, ICredentials feedCredentials, TimeSpan httpTimeout)
00:25:42   Error    |         at Calamari.Integration.Packages.NuGet.NuGetV3Downloader.GetDownloadUri(PackageIdentifier packageIdentifier, Uri feedUri, ICredentials feedCredentials, TimeSpan httpTimeout)
00:25:42   Error    |         at Calamari.Integration.Packages.NuGet.NuGetV3Downloader.DownloadPackage(String packageId, IVersion version, Uri feedUri, ICredentials feedCredentials, String targetFilePath, TimeSpan httpTimeout)
00:25:42   Error    |         at Calamari.Integration.Packages.NuGet.InternalNuGetPackageDownloader.DownloadPackage(String packageId, IVersion version, Uri feedUri, ICredentials feedCredentials, String targetFilePath, Int32 maxDownloadAttempts, TimeSpan downloadAttemptBackoff, Action`5 action)
00:25:42   Error    |         System.Exception
00:25:42   Error    |         at Calamari.Integration.Packages.NuGet.InternalNuGetPackageDownloader.DownloadPackage(String packageId, IVersion version, Uri feedUri, ICredentials feedCredentials, String targetFilePath, Int32 maxDownloadAttempts, TimeSpan downloadAttemptBackoff, Action`5 action)
00:25:42   Error    |         at Calamari.Integration.Packages.Download.NuGetPackageDownloader.DownloadPackage(String packageId, IVersion version, Uri feedUri, ICredentials feedCredentials, String cacheDirectory, Int32 maxDownloadAttempts, TimeSpan downloadAttemptBackoff)
00:25:42   Error    |         at Calamari.Integration.Packages.Download.NuGetPackageDownloader.DownloadPackage(String packageId, IVersion version, String feedId, Uri feedUri, String feedUsername, String feedPassword, Boolean forcePackageDownload, Int32 maxDownloadAttempts, TimeSpan downloadAttemptBackoff)
00:25:42   Error    |         at Calamari.Integration.Packages.Download.PackageDownloaderStrategy.DownloadPackage(String packageId, IVersion version, String feedId, Uri feedUri, FeedType feedType, String feedUsername, String feedPassword, Boolean forcePackageDownload, Int32 maxDownloadAttempts, TimeSpan downloadAttemptBackoff)
00:25:42   Error    |         at Calamari.Commands.DownloadPackageCommand.Execute(String[] commandLineArguments)
00:25:42   Error    |         --Inner Exception--
00:25:42   Error    |         Unexpected character encountered while parsing value: . Path '', line 0, position 0.
00:25:42   Error    |         Newtonsoft.Json.JsonReaderException
00:25:42   Error    |         at Newtonsoft.Json.JsonTextReader.ParseValue()
00:25:42   Error    |         at Newtonsoft.Json.Linq.JObject.Load(JsonReader reader, JsonLoadSettings settings)
00:25:42   Error    |         at Calamari.Integration.Packages.NuGet.NuGetV3Downloader.<>c__DisplayClass12_0.<GetJsonResponse>b__0(Stream stream)
00:25:42   Error    |         at Calamari.Integration.Packages.NuGet.NuGetV3Downloader.GetHttp(Uri uri, ICredentials credentials, TimeSpan httpTimeout, Action`1 processContent)
00:25:42   Error    |         at Calamari.Integration.Packages.NuGet.NuGetV3Downloader.GetJsonResponse(Uri feedUri, ICredentials feedCredentials, TimeSpan httpTimeout)
00:25:42   Error    |         at Calamari.Integration.Packages.NuGet.NuGetV3Downloader.GetDownloadUri(PackageIdentifier packageIdentifier, Uri feedUri, ICredentials feedCredentials, TimeSpan httpTimeout)
00:25:42   Error    |         at Calamari.Integration.Packages.NuGet.NuGetV3Downloader.DownloadPackage(String packageId, IVersion version, Uri feedUri, ICredentials feedCredentials, String targetFilePath, TimeSpan httpTimeout)
00:25:42   Error    |         at Calamari.Integration.Packages.NuGet.InternalNuGetPackageDownloader.DownloadPackage(String packageId, IVersion version, Uri feedUri, ICredentials feedCredentials, String targetFilePath, Int32 maxDownloadAttempts, TimeSpan downloadAttemptBackoff, Action`5 action)
00:25:42   Verbose  |         Process C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe in C:\Octopus\Work\20211202002358-136455-10 exited with code 100
00:25:42   Verbose  |         Exit code: 100

More Information

Suspect this is due to a difference between how the NuGet downloader works for .NET4.0 targets vs .NET4.5+ targets.

The code path for newer .NET Framework builds explicitly handles GZip decompression, but the .NET 4.0 code path doesn't appear to.

Links

🔒ZenDesk Ticket 80936 (Internal)

Workaround

None known

mjhilton commented 1 year ago

We need to investigate here whether the best path is to fix the .NET 4.0 code, or if we are upgrading Calamari to always use .NET 4.5+, then we may automatically get a fix from doing that.