Nuget push binary caching fails for python

[anders-wind]

Describe the bug I'm trying to use the github nuget binary caching integration and while it works for most of our package it fails for python (dependency of pybind11)

Pushing python3_x64-linux.3.10.1-vcpkgc87aa8919988de817c495b22160d65c4accc42bdcfffbd763e4566758c070e28.nupkg to 'https://nuget.pkg.github.com/my-org'...
  PUT https://nuget.pkg.github.com/my-org/
An error was encountered when fetching 'PUT https://nuget.pkg.github.com/my-org/'. The request will now be retried.
Error while copying content to a stream.
  Unable to read data from the transport connection: The socket has been shut down.
  The socket has been shut down
  PUT https://nuget.pkg.github.com/my-org/
An error was encountered when fetching 'PUT https://nuget.pkg.github.com/my-org/'. The request will now be retried.
Error while copying content to a stream.
  Unable to read data from the transport connection: The socket has been shut down.
  The socket has been shut down
  PUT https://nuget.pkg.github.com/my-org/
Error while copying content to a stream.
  Unable to read data from the transport connection: The socket has been shut down.
  The socket has been shut down

I ran cmake with: -DVCPKG_INSTALL_OPTIONS=--debug

Here's our manifest

{
    "name": "epex",
    "version-semver": "0.0.1",
    "dependencies": [
        "nlohmann-json",
        "frozen",
        "pybind11",
        {
            "name": "abseil",
            "features": [
                "cxx17"
            ],
            "version>=": "20211102.1"
        },
        {
            "name": "fmt",
            "version>=": "8.1.1"
        },
        "spdlog",
        "gtest",
        "benchmark"
    ],
    "builtin-baseline": "c0d667ad6bb2d68890f33ded8b2e5bc182ab01fb"
}

and I have the following in my github action:

VCPKG_BINARY_SOURCES: 'default;nuget,GitHub,readwrite'

Again it works for most packages, but not for python3.

Environment

• OS: Linux ubuntu-latest
• Compiler: clang-10

To Reproduce setup nuget credentials in github actions and configure vcpkg to use it. add pybind11 as vcpkg dependency in manifest. run cmake and see cache pushing fails.

Expected behavior python3 dependency would be correctly pushed to the binary cache

Failure logs

Building package python3[core]:x64-linux...
-- Installing port from location: /home/runner/work/EPEXcpp/EPEXcpp/vcpkg/buildtrees/versioning_/versions/python3/c9b705608afc0485c03effb5aaa439f027b20af4
[DEBUG] 139823004247872: popen(/usr/local/bin/cmake "-DALL_FEATURES=deprecated-win7-support;" -DCURRENT_PORT_DIR=/home/runner/work/EPEXcpp/EPEXcpp/vcpkg/buildtrees/versioning_/versions/python3/c9b705608afc0485c03effb5aaa439f027b20af4 -D_HOST_TRIPLET=x64-linux -DFEATURES=core -DPORT=python3 -DVCPKG_USE_HEAD_VERSION=0 -D_VCPKG_DOWNLOAD_TOOL=BUILT_IN -D_VCPKG_EDITABLE=0 -D_VCPKG_NO_DOWNLOADS=0 -DCMD=BUILD -DDOWNLOADS=/home/runner/work/EPEXcpp/EPEXcpp/vcpkg/downloads -DTARGET_TRIPLET=x64-linux -DTARGET_TRIPLET_FILE=/home/runner/work/EPEXcpp/EPEXcpp/vcpkg/triplets/x64-linux.cmake -DVCPKG_BASE_VERSION=2022-02-01 -DVCPKG_CONCURRENCY=3 -DVCPKG_PLATFORM_TOOLSET=external -DGIT=/usr/bin/git -DVCPKG_ROOT_DIR=/home/runner/work/EPEXcpp/EPEXcpp/vcpkg -DPACKAGES_DIR=/home/runner/work/EPEXcpp/EPEXcpp/vcpkg/packages -DBUILDTREES_DIR=/home/runner/work/EPEXcpp/EPEXcpp/vcpkg/buildtrees -D_VCPKG_INSTALLED_DIR=/home/runner/work/EPEXcpp/EPEXcpp/build/temp.linux-x86_64-3.8/vcpkg_installed -DDOWNLOADS=/home/runner/work/EPEXcpp/EPEXcpp/vcpkg/downloads -DVCPKG_MANIFEST_INSTALL=OFF -P /home/runner/work/EPEXcpp/EPEXcpp/vcpkg/scripts/ports.cmake 2>&1)
-- Downloading https://github.com/python/cpython/archive/v3.10.1.tar.gz -> python-cpython-v3.10.1.tar.gz...
-- Extracting source /home/runner/work/EPEXcpp/EPEXcpp/vcpkg/downloads/python-cpython-v3.10.1.tar.gz
-- Applying patch 0001-static-library.patch
-- Applying patch 0002-use-vcpkg-zlib.patch
-- Applying patch 0003-devendor-external-dependencies.patch
-- Applying patch 0004-dont-copy-vcruntime.patch
-- Applying patch 0005-only-build-required-projects.patch
-- Using source at /home/runner/work/EPEXcpp/EPEXcpp/vcpkg/buildtrees/python3/src/v3.10.1-7ec7e9e4c8.clean
-- Getting CMake variables for x64-linux-dbg
-- Getting CMake variables for x64-linux-rel
-- Configuring x64-linux-dbg
-- Configuring x64-linux-rel
-- Building x64-linux-dbg
-- Installing x64-linux-dbg
-- Building x64-linux-rel
-- Installing x64-linux-rel
-- Installing: /home/runner/work/EPEXcpp/EPEXcpp/vcpkg/packages/python3_x64-linux/share/python3/copyright
-- Fixing pkgconfig file: /home/runner/work/EPEXcpp/EPEXcpp/vcpkg/packages/python3_x64-linux/lib/pkgconfig/python-3.10-embed.pc
-- Fixing pkgconfig file: /home/runner/work/EPEXcpp/EPEXcpp/vcpkg/packages/python3_x64-linux/lib/pkgconfig/python-3.10.pc
-- Fixing pkgconfig file: /home/runner/work/EPEXcpp/EPEXcpp/vcpkg/packages/python3_x64-linux/debug/lib/pkgconfig/python-3.10-embed.pc
-- Fixing pkgconfig file: /home/runner/work/EPEXcpp/EPEXcpp/vcpkg/packages/python3_x64-linux/debug/lib/pkgconfig/python-3.10.pc
-- Installing: /home/runner/work/EPEXcpp/EPEXcpp/vcpkg/packages/python3_x64-linux/share/python3/usage
-- Installing cmake wrappers
[DEBUG] 139823004247872: cmd_execute_and_stream_data() returned 0 after 237497821 us
-- Performing post-build validation
-- Performing post-build validation done
[DEBUG] system(cd /home/runner/work/EPEXcpp/EPEXcpp/vcpkg/packages/python3_x64-linux && zip --quiet -y -r /home/runner/work/EPEXcpp/EPEXcpp/vcpkg/buildtrees/python3/x64-linux.zip *)
[DEBUG] cmd_execute() returned 0 after 14252988 us
Stored binary cache: /home/runner/.cache/vcpkg/archives/c8/c87aa8919988de817c495b22160d65c4accc42bdcfffbd763e4566758c070e28.zip
[DEBUG] 139823004247872: popen(/usr/bin/mono /home/runner/work/EPEXcpp/EPEXcpp/vcpkg/downloads/tools/nuget-5.11.0-linux/nuget.exe pack /home/runner/work/EPEXcpp/EPEXcpp/vcpkg/buildtrees/python3/x64-linux.nuspec -OutputDirectory /home/runner/work/EPEXcpp/EPEXcpp/vcpkg/buildtrees -NoDefaultExcludes -ForceEnglishOutput -NonInteractive 2>&1)
[DEBUG] 139823004247872: cmd_execute_and_stream_data() returned 0 after 16391572 us
Attempting to build package from 'x64-linux.nuspec'.
WARNING: NU5103: The folder 'lib/pkgconfig/python-3.10-embed.pc' under 'lib' is not recognized as a valid framework name or a supported culture identifier. Rename it to a valid framework name or culture identifier.
WARNING: NU5103: The folder 'lib/pkgconfig/python-3.10.pc' under 'lib' is not recognized as a valid framework name or a supported culture identifier. Rename it to a valid framework name or culture identifier.
WARNING: NU5103: The folder 'lib/python3.10/enum.py' under 'lib' is not recognized as a valid framework name or a supported culture identifier. Rename it to a valid framework name or culture identifier.
WARNING: NU5103: The folder 'lib/python3.10/turtle.py' under 'lib' is not recognized as a valid framework name or a supported culture identifier. Rename it to a valid framework name or culture identifier.
WARNING: NU5103: The folder 'lib/python3.10/xdrlib.py' under 'lib' is not recognized as a valid framework name or a supported culture identifier. Rename it to a valid framework name or culture identifier.
... (a million lines like the ones above)
...
...
...
WARNING: NU5123: The file 'debug/lib/python3.10/site-packages/pip/_vendor/urllib3/contrib/_securetransport/__pycache__/__init__.cpython-310.pyc' path, name, or both are too long. Your package might not work without long file path support. Please shorten the file path or file name.
WARNING: NU5128: Some target frameworks declared in the dependencies group of the nuspec and the lib/ref folder do not have exact matches in the other location. Consult the list of actions below:
- Add a dependency group for pkgconfig0.0 to the nuspec
- Add a dependency group for python3.100.0 to the nuspec
Successfully created package '/home/runner/work/EPEXcpp/EPEXcpp/vcpkg/buildtrees/python3_x64-linux.3.10.1-vcpkgc87aa8919988de817c495b22160d65c4accc42bdcfffbd763e4566758c070e28.nupkg'.
Uploading binaries for python3:x64-linux to NuGet source GitHub.
[DEBUG] 139823004247872: popen(/usr/bin/mono /home/runner/work/EPEXcpp/EPEXcpp/vcpkg/downloads/tools/nuget-5.11.0-linux/nuget.exe push /home/runner/work/EPEXcpp/EPEXcpp/vcpkg/buildtrees/python3_x64-linux.3.10.1-vcpkgc87aa8919988de817c495b22160d65c4accc42bdcfffbd763e4566758c070e28.nupkg -ForceEnglishOutput -Timeout 100 -Source GitHub -NonInteractive 2>&1)
[DEBUG] 139823004247872: cmd_execute_and_stream_data() returned 1 after  3021775 us
WARNING: No API Key was provided and no API Key could be found for 'https://nuget.pkg.github.com/twig-energy'. To save an API Key for a source use the 'setApiKey' command.
Pushing python3_x64-linux.3.10.1-vcpkgc87aa8919988de817c495b22160d65c4accc42bdcfffbd763e4566758c070e28.nupkg to 'https://nuget.pkg.github.com/twig-energy'...
  PUT https://nuget.pkg.github.com/twig-energy/
An error was encountered when fetching 'PUT https://nuget.pkg.github.com/twig-energy/'. The request will now be retried.
Error while copying content to a stream.
  Unable to read data from the transport connection: The socket has been shut down.
  The socket has been shut down
  PUT https://nuget.pkg.github.com/twig-energy/
An error was encountered when fetching 'PUT https://nuget.pkg.github.com/twig-energy/'. The request will now be retried.
Error while copying content to a stream.
  Unable to read data from the transport connection: The socket has been shut down.
  The socket has been shut down
  PUT https://nuget.pkg.github.com/twig-energy/
Error while copying content to a stream.
  Unable to read data from the transport connection: The socket has been shut down.
  The socket has been shut down
Pushing NuGet to GitHub failed. Use --debug for more information.
Installing package python3[core]:x64-linux...
Elapsed time for package python3:x64-linux: 4.568 min
Starting package 17/18: pybind11:x64-linux
Installing package pybind11[core]:x64-linux...
Elapsed time for package pybind11:x64-linux: 13.81 ms
Starting package 18/18: spdlog:x64-linux
Installing package spdlog[core]:x64-linux...
Elapsed time for package spdlog:x64-linux: 26.78 ms

Total elapsed time: 4.888 min

[anders-wind]

And we should have plenty of space etc

[mathisloge]

I think this might be a thing: To save an API Key for a source use the 'setApiKey' command.

https://docs.microsoft.com/de-de/nuget/reference/cli-reference/cli-ref-setapikey

      - name: "Setup NuGet apikey"
        shell: "bash"
        run: >
          `./vcpkg/vcpkg fetch nuget | tail -n 1`
          setapikey "<NUGET_REGISTRY_PAT>" -Source "<NUGET_REGISTRY>"

[anders-wind]

But why would it be required for pushing python3 but not the 17 other packages which are successfully pushed?

[JackBoosY]

cc @BillyONeal

[BillyONeal]

For helping investigation: python3 is 106MB, and ~35MB zipped.

Unfortunately it is blowing up inside the command

/usr/bin/mono /home/runner/work/EPEXcpp/EPEXcpp/vcpkg/downloads/tools/nuget-5.11.0-linux/nuget.exe push /home/runner/work/EPEXcpp/EPEXcpp/vcpkg/buildtrees/python3_x64-linux.3.10.1-vcpkgc87aa8919988de817c495b22160d65c4accc42bdcfffbd763e4566758c070e28.nupkg -ForceEnglishOutput -Timeout 100 -Source GitHub -NonInteractive 2>&1

meaning our options for direct diagnosis are limited.

[anders-wind]

Is there anything I can do to provide more input? I am not sure what the requires:discussion label entails

[anders-wind]

https://github.com/NuGet/Home/issues/9775#issuecomment-714509211 ^seems relevant, apparently using api key is more stable than username/password. Is this something we could get for vcpkg?

[mathisloge]

https://docs.microsoft.com/de-de/nuget/reference/cli-reference/cli-ref-setapikey

      - name: "Setup NuGet apikey"
        shell: "bash"
        run: >
          `./vcpkg/vcpkg fetch nuget | tail -n 1`
          setapikey "<NUGET_REGISTRY_PAT>" -Source "<NUGET_REGISTRY>"

@anders-wind have you tried this?

[BillyONeal]

Related, maybe: https://github.com/microsoft/vcpkg/pull/22191

[JackBoosY]

Hi guys, can anyone test whether https://github.com/microsoft/vcpkg/pull/22191 works?

Thanks.

[anders-wind]

Im trying out the setapikey setup. Im getting Your request could not be authenticated by the GitHub Packages service. Please ensure your access token is valid and has the appropriate scopes configured.

        mono `./vcpkg/vcpkg fetch nuget | tail -n 1` sources add -source "https://nuget.pkg.github.com/twig-energy/index.json" -name "GitHub";
        mono `./vcpkg/vcpkg fetch nuget | tail -n 1` setapikey "${{ secrets.PACKAGES_ACCESS_TOKEN }}" -source "https://nuget.pkg.github.com/twig-energy/index.json";

The secrets.PACKAGES_ACCESS_TOKEN is a personal access token from my account. I also tried with secrets.GITHUB_TOKEN

[mathisloge]

If the packages are stored in a different repo, your PAT need to have the read and write packages persmissions.

If it is the same repo, github token works,but might fail if multiple repos are using binary caching if there isnt a global repo to store the build packages

[anders-wind]

These are the permissions I gave the key. The path I'm pointing to is just the organization url and not the entire repo url. Should I change that?

[mathisloge]

Yeah, you need a repo as a location. I recommend just to create a new repo e.g. vcpkg-cache and use that repo everywhere. That works quite well with GHA.

Permissions are looking good.

[anders-wind]

Okay ill try that :)

[anders-wind]

Okay, I got the setapikey working - it actually enables consistent pushes, but for whatever reason it does not provide neccesary permissions to download from the registry. So I need to input both username/password and the apikey

        mono `./vcpkg/vcpkg fetch nuget | tail -n 1` sources add -source "https://nuget.pkg.github.com/twig-energy/index.json" -storepasswordincleartext -name "GitHub" -username "twig-energy" -password "${{ secrets.GITHUB_TOKEN }}"
        mono `./vcpkg/vcpkg fetch nuget | tail -n 1` setapikey "${{ secrets.PACKAGES_ACCESS_TOKEN }}" -source "https://nuget.pkg.github.com/twig-energy/index.json"

@mathisloge I tried adding a new repo but got a bunch of other errors so I returned to the org setup

[mathisloge]

yeah username + password + apikey. IIRC the first one is for github basic auth to get access to the package manifests and the second one for uploading

[anders-wind]

I need to try if the GITHUB_TOKEN can also be used for the setapikey :) But yea hope the documentation will be updated - but this issue can be closed I guess :) Thanks for the help!