Dotnet nuget push to GPR fails randomly

RehanSaeed commented 4 years ago

Details about Problem

dotnet.exe --version: 3.1.301 OS version: GitHub Actions windows-latest - Microsoft Windows Server 2019 10.0.17763

Detailed repro steps so we can see the same problem

I have two repos Schema.NET and FastestNuGet where I have the exact same build.yml file for GitHub Actions. However, dotnet nuget push fails on one and succeeds on the other.

The only difference I can understand is that one project has a . in the name and the other does not.

Failing CLI Output

Run dotnet nuget push .\windows-latest\*.nupkg --source GitHub --skip-duplicate
warn : No API Key was provided and no API Key could be found for 'https://nuget.pkg.github.com/RehanSaeed'. To save an API Key for a source use the 'setApiKey' command.
Pushing Schema.NET.7.1.1-preview.0.33.nupkg to 'https://nuget.pkg.github.com/RehanSaeed'...
  PUT https://nuget.pkg.github.com/RehanSaeed/
An error was encountered when fetching 'PUT https://nuget.pkg.github.com/RehanSaeed/'. The request will now be retried.
Error while copying content to a stream.
  Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
  An existing connection was forcibly closed by the remote host.
  PUT https://nuget.pkg.github.com/RehanSaeed/
An error was encountered when fetching 'PUT https://nuget.pkg.github.com/RehanSaeed/'. The request will now be retried.
Error while copying content to a stream.
  Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
  An existing connection was forcibly closed by the remote host.
  PUT https://nuget.pkg.github.com/RehanSaeed/
error: Error while copying content to a stream.
error:   Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
error:   An existing connection was forcibly closed by the remote host.

build.yml

name: Build

on:
  push:
  pull_request:
  release:
    types:
      - published

env:
  # Set the DOTNET_SKIP_FIRST_TIME_EXPERIENCE environment variable to stop wasting time caching packages
  DOTNET_SKIP_FIRST_TIME_EXPERIENCE: true
  # Disable sending usage data to Microsoft
  DOTNET_CLI_TELEMETRY_OPTOUT: true
  # Set the build number in MinVer
  MINVERBUILDMETADATA: build.${{github.run_number}}

jobs:
  build:
    name: Build-${{matrix.os}}
    runs-on: ${{matrix.os}}
    strategy:
      matrix:
        os: [ubuntu-latest, windows-latest, macOS-latest]
    steps:
    - name: 'Checkout'
      uses: actions/checkout@v2
      with:
        lfs: true
        fetch-depth: 0
    - name: 'Git Fetch Tags'
      run: git fetch --tags
      shell: pwsh
    - name: 'Install .NET Core SDK'
      uses: actions/setup-dotnet@v1
      with:
        dotnet-version: 3.1.301
    - name: 'Dotnet Tool Restore'
      run: dotnet tool restore
      shell: pwsh
    - name: 'Dotnet Cake Build'
      run: dotnet cake --target=Build
      shell: pwsh
    - name: 'Dotnet Cake Test'
      run: dotnet cake --target=Test
      shell: pwsh
    - name: 'Dotnet Cake Pack'
      run: dotnet cake --target=Pack
      shell: pwsh
    - name: 'Publish Artefacts'
      uses: actions/upload-artifact@v1.0.0
      with:
        name: ${{matrix.os}}
        path: './Artefacts'

  push-github-packages:
    name: 'Push GitHub Packages'
    needs: build
    if: github.ref == 'refs/heads/master' || github.event_name == 'release'
    runs-on: windows-latest
    steps:
      - name: 'Download Artefact'
        uses: actions/download-artifact@v1
        with:
          name: 'windows-latest'
      - name: 'Dotnet NuGet Add Source'
        run: dotnet nuget add source https://nuget.pkg.github.com/RehanSaeed/index.json --name GitHub --username RehanSaeed --password ${{secrets.GITHUB_TOKEN}}
        shell: pwsh
      - name: 'Dotnet NuGet Push'
        run: dotnet nuget push .\windows-latest\*.nupkg --source GitHub --skip-duplicate
        shell: pwsh

RehanSaeed commented 4 years ago

There seem to be multiple issues describing this bug but they all seem to have been closed with unsatisfactory workarounds. The core issue is that the dotnet nuget push or dotnet nuget add source commands (I'm not sure which) seem to be broken.

RehanSaeed commented 4 years ago

I've tried re-running the build and it fails randomly 4/5 times for the Schema.NET build but is rock solid for FastestNuGet.

RehanSaeed commented 4 years ago

I've resorted to retrying to push a NuGet package to GitHub Actions.

$count = 0;
do { 
    ++$count;
    dotnet nuget push .\*.nupkg --source GitHub --skip-duplicate;
} 
while ($count -lt 30 -and $LastExitCode -ne 0)

PeterKnealeBizCover commented 4 years ago

Schema.NET contains a '.' in the name, while 'FastestNuGet' doesn't. I can't find where I read it but I recall someone noticing unreliable nuget publishing when the package name contained a period. Would it be difficult to try removing it?

RehanSaeed commented 4 years ago

Removing it is not an option.

RehanSaeed commented 4 years ago

Someone has written an entire dotnet global tool to workaround this bug:

https://github.com/jcansdale/gpr

There seem to be a lot of others experiencing the same issue:

https://twitter.com/RehanSaeedUK/status/1281705870917881856?s=20

jcansdale commented 4 years ago

Try passing the DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER=0 env var to dotnet nuget push.

This seems to make it work most of the time.

Unfortunately I then see the odd error like this:

Pushing nuget-symbols.1.0.0-g6f7e8c13efc9a5ad6e2d28195a975d1b4b1a8d7c-33.nupkg to 'https://nuget.pkg.github.com/jcansdale-test'...
  PUT https://nuget.pkg.github.com/jcansdale-test/
warn : Missing nuget package in request
  BadRequest https://nuget.pkg.github.com/jcansdale-test/ 32ms
error: Response status code does not indicate success: 400 (Bad Request).

You can see what happens in the matrix run here: https://github.com/jcansdale-test/nuget-symbols/actions/runs/178328401

chadly commented 4 years ago

Someone has written an entire dotnet global tool to workaround this bug:

I second @jcansdale's gpr tool. It solved all my problems pushing to GH packages and I'm not worried about this issue anymore.

I mean...the issue still needs to be dealt with, but I can at least move on with my life now.

jcansdale commented 4 years ago

I'm dropping some notes here about the DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLE property, incase they might be useful. I grabbed them from an internal GitHub PR that repros this issue. It is still a mystery what is going on!

Description

Starting with .NET Core 2.1, the SocketsHttpHandler class provides the implementation used by higher-level HTTP networking classes such as HttpClient. The use of SocketsHttpHandler offers a number of advantages:

A significant performance improvement when compared with the previous implementation.
The elimination of platform dependencies, which simplifies deployment and servicing. For example, libcurl is no longer a dependency on .NET Core for macOS and .NET Core for Linux.
Consistent behavior across all .NET platforms.

By defining an environment variable named DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLE and setting it to false, we can force dotnet nuget push to use the legacy HttpClientHandler instead of SocketsHttpHandler.

When SocketsHttpHandler is used, publishing sometimes fails on the client side like this:

info : Pushing Chicken.1.0.0-gb945a02962fd3753b51f67efb3b98c631f2f5189-macos-latest-2020-07-24-080210.nupkg to 'https://nuget.pkg.github.com/sweethaven-village'...
info :   PUT https://nuget.pkg.github.com/sweethaven-village/
info : An error was encountered when fetching 'PUT https://nuget.pkg.github.com/sweethaven-village/'. The request will now be retried.
info : An error occurred while sending the request.
info :   The response ended prematurely.
info :   PUT https://nuget.pkg.github.com/sweethaven-village/
info : An error was encountered when fetching 'PUT https://nuget.pkg.github.com/sweethaven-village/'. The request will now be retried.
info : An error occurred while sending the request.
info :   The response ended prematurely.
info :   PUT https://nuget.pkg.github.com/sweethaven-village/
error: An error occurred while sending the request.
error:   The response ended prematurely.

When HttpClientHandler is used, publishing sometimes fails on the server side like this:

info : Pushing Chicken.1.0.0-g5e6a3076b547632358aac5994f5b6e0fcff14bd6-ubuntu-latest-2020-07-24-004434.nupkg to 'https://nuget.pkg.github.com/sweethaven-village'...
info :   PUT https://nuget.pkg.github.com/sweethaven-village/
warn : Missing nuget package in request
info :   BadRequest https://nuget.pkg.github.com/sweethaven-village/ 71ms
error: Response status code does not indicate success: 400 (Bad Request).

For this particular package, the first error seems to happen on macos-latest and the second on ubuntu-latest. I've seen the first error happen on all platforms for different packages.

DamienDennehy commented 4 years ago

After 9 failures in a row using DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER seems to have worked. Thank you @jcansdale. If it happens again I'll post any of the 400 warnings so this can be investigated more.

jcansdale commented 4 years ago

@DamienDennehy interesting, thanks.

Which OS are you using? The intermittent errors I was seeing when using this option where on Linux.

DamienDennehy commented 4 years ago

Yeah it was Linux - Ubuntu. I'm using GitHub Actions to push and it was failing. Here's the push step:

` deploy-packages:

needs: build

if: github.ref == 'refs/heads/master'

runs-on: ubuntu-latest

env:
    DOTNET_NOLOGO: true
    DOTNET_SKIP_FIRST_TIME_EXPERIENCE: true

steps:
- name: Setup .NET Core
  uses: actions/setup-dotnet@v1
  with:
    dotnet-version: 3.1.301
    source-url: https://nuget.pkg.github.com/ORG_NAME/index.json    
  env:
    NUGET_AUTH_TOKEN: ${{secrets.GITHUB_TOKEN}}

- name: Download Artifacts 
  uses: actions/download-artifact@v2
  with:
    name: ubuntu-latest
    path: ./artifacts

- name: Push to GitHub Packages
  run: dotnet nuget push ./artifacts/**/*.nupkg --skip-duplicate --no-symbols true   
  env:
    DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLE: false   `

jasonycw commented 4 years ago

I am having the same issue on Windows PowerShell 7 when using

dotnet nuget add source https://nuget.pkg.github.com/OWNER/index.json --name "GitHub package" --username DEV --password $GITHUB_ACCESS_TOKEN --store-password-in-clear-text
dotnet nuget push "ThePackage.1.5.14.nupkg" --source "GitHub package" --skip-duplicate

99% of the time, the dotnet nuget push will return

warn : No API Key was provided and no API Key could be found for 'https://nuget.pkg.github.com/OWNER'. To save an API Key for a source use the 'setApiKey' command.
Pushing ThePackage.1.5.14.nupkg to 'https://nuget.pkg.github.com/OWNER'...
  PUT https://nuget.pkg.github.com/OWNER/
An error was encountered when fetching 'PUT https://nuget.pkg.github.com/OWNER/'. The request will now be retried.
An error occurred while sending the request.
  The response ended prematurely.
  PUT https://nuget.pkg.github.com/OWNER/
An error was encountered when fetching 'PUT https://nuget.pkg.github.com/OWNER/'. The request will now be retried.
An error occurred while sending the request.
  The response ended prematurely.
  PUT https://nuget.pkg.github.com/OWNER/
error: An error occurred while sending the request.
error:   The response ended prematurely.

But it seems that if I don't use dotnet nuget push but just nuget push, it will work most of the time on Windows

nuget sources add -Source https://nuget.pkg.github.com/OWNER/index.json -Name "GitHub package" -Username DEV -Password GITHUB_ACCESS_TOKEN
nuget push "ThePackage.1.5.14.nupkg" -Source "GitHub package" -SkipDuplicate

Reference: https://github.com/NuGet/Home/issues/8580#issuecomment-531997703 NuGet Version: 5.2.0.6090

So I retried dotnet nuget push in docker, same error as the one on Windows showed up. Then I added DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER=0 with docker,

FROM mcr.microsoft.com/dotnet/core/sdk:3.1-alpine AS base
ARG TECTAC_GITHUB_ACCESS_TOKEN

ENV DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER=0
RUN dotnet nuget add source https://nuget.pkg.github.com/OWNER/index.json --name "GitHub package" --username DEV --password $GITHUB_ACCESS_TOKEN --store-password-in-clear-text
RUN find . -name "*.nupkg" | while read f; do dotnet nuget push ${f} --source "GitHub package" --skip-duplicate; done;

It returned different error when it ran dotnet nuget push

find: ./proc/8/task/17/fdinfo/109: No such file or directory
find: ./proc/8/task/18/fdinfo/107: No such file or directory
find: ./proc/8/task/19/fd/109: No such file or directory
find: ./proc/8/task/19/fdinfo/107: No such file or directory
error: Unable to load the service index for source https://nuget.pkg.github.com/OWNER/index.json.
error:   The type initializer for 'System.Net.Http.CurlHandler' threw an exception.
error:   The type initializer for 'Http' threw an exception.
error:   The type initializer for 'HttpInitializer' threw an exception.
error:   Unable to load shared library 'System.Net.Http.Native' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: Error loading shared library libSystem.Net.Http.Native: No such file or directory

And if I use the latest nuget CLI (5.7.0.6726) with mono on docker

FROM mcr.microsoft.com/dotnet/core/sdk:3.1-alpine AS base
ARG GITHUB_ACCESS_TOKEN

RUN apk upgrade && \
    apk add --no-cache --update-cache mono --repository http://dl-cdn.alpinelinux.org/alpine/edge/testing && \
        cert-sync /etc/ssl/certs/ca-certificates.crt
ADD https://dist.nuget.org/win-x86-commandline/latest/nuget.exe /usr/local/bin/nuget.exe
ENV nuget="mono /usr/local/bin/nuget.exe"
ENV DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER=0

RUN $nuget sources add -Source https://nuget.pkg.github.com/OWNER/index.json -Name "GitHub package" -Username DEV -Password $GITHUB_ACCESS_TOKEN -StorePasswordInClearText

RUN find . -name "*.nupkg" | while read f; do $nuget push ${f} -Source "GitHub package" -SkipDuplicate; done;

I get

WARNING: No API Key was provided and no API Key could be found for 'https://nuget.pkg.github.com/OWNER'. To save an API Key for a source use the 'setApiKey' command.
Pushing TEC.EtgDotNetCommon.EventStore.1.5.14.nupkg to 'https://nuget.pkg.github.com/OWNER'...
  PUT https://nuget.pkg.github.com/OWNER/
An error was encountered when fetching 'PUT https://nuget.pkg.github.com/OWNER/'. The request will now be retried.
An error occurred while sending the request
  Error getting response stream (ReadDoneAsync2): ReceiveFailure
  PUT https://nuget.pkg.github.com/OWNER/
An error was encountered when fetching 'PUT https://nuget.pkg.github.com/OWNER/'. The request will now be retried.
An error occurred while sending the request
  Error getting response stream (ReadDoneAsync2): ReceiveFailure
  PUT https://nuget.pkg.github.com/OWNER/
Error getting response stream (ReadDoneAsync2): ReceiveFailure

jcansdale commented 4 years ago

@jasonycw,

Thanks for sharing your investigations!

But it seems that if I don't use dotnet nuget push but just nuget push, it will work most of the time on Windows

You say nuget push works most of the time on Windows. I thought this was the only configuration that works consistently. I'm interested to know what error is shows when it fails?

jcansdale commented 4 years ago

@RehanSaeed,

I've forked the NuGet/NuGet.Client repo and created a GitHub Actions based integration test for this issue, see: https://github.com/jcansdale-test/NuGet.Client/blob/dev/.github/workflows/dotnet-core.yml

You can see publishing randomly fails on macos-latest: https://github.com/jcansdale-test/NuGet.Client/actions/runs/237534394

If I change TransferEncodingChunked to false, it seems to work consistently: https://github.com/jcansdale-test/NuGet.Client/actions/runs/237559628

I wonder who might be able to investigate this further?

jasonycw commented 4 years ago

@jcansdale I didn't save the only time it failed, but since then, all nuget push works. That's why I said "most of the time" instead of "all the time".

However since it was just for testing purposes and in my use case, I need to push through docker but both dotnet nuget push and nuget push don't work. In the end, I used gpr instead of dotnet nuget or nuget

ARG TECTAC_GITHUB_ACCESS_TOKEN

RUN dotnet tool install -g gpr --no-cache -v q
RUN dotnet nuget add source https://nuget.pkg.github.com/OWNER/index.json --name "GitHub package" --username DEV --password $GITHUB_ACCESS_TOKEN --store-password-in-clear-text

RUN find . -name "*.nupkg" | while read f; do gpr push --api-key $TECTAC_GITHUB_ACCESS_TOKEN ${f}; done;

jcansdale commented 4 years ago

Hi @RehanSaeed,

You should find the following works fine now!

dotnet nuget push .\windows-latest\*.nupkg --api-key ${{ github.token }} --source GitHub --skip-duplicate

For some reason basic auth (password in the nuget.config file) still doesn't work consistently. We've added a warning to encourage users to use --api-key whenever basic auth is used.

It's a mystery why --api-key works but basic auth fails. They both appear to use chunked encoding. However, if I tweak the NuGet source to disable chunked encoding, this seems to fix publishing using basic auth. 😕

Using the --api-key option is a better solution for a number of reasons, so this it the fix/warning the packages team are going with.

Thanks for your persistence on this issue! 🙏

gitfool commented 4 years ago

@jcansdale is the change to support api key documented somewhere?

gitfool commented 4 years ago

FWIW, it finally got a mention in the GitHub Changelog but the docs for Configuring dotnet CLI for use with GitHub Packages still don't mention it. 🤔

EmanueleCiriachi commented 3 years ago

I have the same problem. I am getting

warn : No destination repository detected. Ensure the source project has a 'RepositoryUrl' property defined. If you're using
a nuspec file, ensure that it has a repository element with the required 'type' and 'url' attributes.

  BadRequest https://nuget.pkg.github.com/gkDigitalEcosystem/ 278ms
error: Response status code does not indicate success: 400 (Bad Request).

Why does it complain about a 'RepositoryUrl' property? Do I really need a VS project? I merely have a .nupkg that I wanted to store.

zivkan commented 10 months ago

TL;DR try running dotnet nuget locals http-cache --clear, before every push, and see if that resolves flakey issues

A few months ago I was investigating a customer issue, and tried to use GPR for something. I was getting auth issues from GPR, so I opened Fiddler, and I found that NuGet wasn't sending credentials, because GPR wasn't sending the WWW-Authenticate header on its HTTP 401 responses to unauthenticated requests, in violation of RFC 7235. I reported this to GitHub support, and they said an internal issue was created, but that's all the visibility I have. Maybe it's fixed by now, I'm not sure.

When pushing to V3 NuGet feeds, NuGet gets the "service index", to find the push URL, and then pushes the nupkg to that URL. However, NuGet will cache the service index, like all other GET requests that is not search, for up to 30 minutes. When the server index is stale, NuGet will request the service index, without authentication, GPR will respond with a 401, including the WWW-Authenticate header, so .NET's HTTP client knows how to auth on the second request, and then the third request will be to the push endpoint, apparently with auth headers. However, when the service index is cached, then the first two requests aren't made, and .NET's HTTP Client doesn't know which auth scheme to use (basic, bearer, digest, etc).

Therefore, if the push failures are due to missing auth, try deleting the HTTP cache just before every push, using dotnet nuget locals http-cache --clear. If this works for you, perhaps contacting github support to tell them about the server side bug will help them prioritize getting it fixed.

PieterjanDeClippel commented 1 month ago

This issue was most likely fixed in .NET 7. I don't experience this problem anymore. This is an example workflow

NuGet / Home