Provide a shortcut for uploading (syncing) local binarycaching to remote sources

[reitowo]

Is your feature request related to a problem? Please describe. I want to add a github nuget binary caching source for accelerating workflow build speed, and I found the only way to upload to the github package is to disable the default source and rebuild all ports again.

Proposed solution Provide a switch in environment variable like VCPKG_BINARY_SYNC_SOURCES. If the archive is found in any source, try to upload to all sources with write enabled.

For example, if set VCPKG_BINARY_SOURCES=clear;default,readwrite;nuget,GitHub,readwrite, VCPKG_BINARY_SYNC_SOURCES=1 If the cache archive is retrieved from the local default path, upload (sync) it to the nuget source.

[autoantwort]

The problem with the nuget packages that are currently generated is that they contains information that are not available in the zips from the default binary cache. So you can't create a nuget package afterwards from the default source. You could reduce the information the nuget package so that this is possible in the future.

[reitowo]

The problem with the nuget packages that are currently generated is that they contains information that are not available in the zips from the default binary cache. So you can't create a nuget package afterwards from the default source.

You could reduce the information the nuget package so that this is possible in the future.

What specifically is missing? Can it be added to the default source otherwise?

Oh.. I just realized that the VCPKG_BINARY_NUGET_CACHE is literally a nuget cache, not saving to default local source...

I thought binarycaching should be uniform formatted... (shouldn't it be?)

Why binarycaching is not designed in a interface implementation schema and keep the file format same across all backends?

[autoantwort]

What specifically is missing?

I think it is mainly info.compiler_id and info.compiler_version. But maybe we can simply omit that.

Why binarycaching is not designed in a interface implementation schema and keep the file format same across all backends?

It is, except the nuget backend

[reitowo]

If it is possible, I'll try implementing it.

[BillyONeal]

I thought binarycaching should be uniform formatted... (shouldn't it be?)

For example, we speak the NuGet protocol which means it needs to be a valid .nupkg. The zips the other backends use should be more or less the same.

[github-actions[bot]]

This is an automated message. Per our repo policy, stale issues get closed if there has been no activity in the past 180 days. The issue will be automatically closed in 14 days. If you wish to keep this issue open, please add a new comment.

[autoantwort]

Keep open

[github-actions[bot]]

This is an automated message. Per our repo policy, stale issues get closed if there has been no activity in the past 180 days. The issue will be automatically closed in 14 days. If you wish to keep this issue open, please add a new comment.

[autoantwort]

Keep open

[Crzyrndm]

The problem with the nuget packages that are currently generated is that they contains information that are not available in the zips from the default binary cache. So you can't create a nuget package afterwards from the default source. You could reduce the information the nuget package so that this is possible in the future.

This feature would be significantly beneficial for non-nuget package sources as well. Notably, the current inability to push cached builds to a previously checked cache precludes being able to have a tiered caching system (i.e. Local (fast) cache + Remote (slow) cache).

The issue is that if the package is found in the remote cache, it is never bumped to the local cache for fast subsequent access and that mostly invalidates having the fast cache (if the slow cache is shared between machines, only one machine ever gets the "fast path" for each package build and caching results are significantly inconsistent).

Pretty much all the examples I can think of use the default/filesystem as the fast path and a networked provider for the slower layer (shared drive/Google drive/One drive/dropbox, AWS/GCP/Azure/...). Multi-device/user workflows and cloud CI services with caching similar to GHA cache (local to the branch) benefit significantly if this would work

[github-actions[bot]]

This is an automated message. Per our repo policy, stale issues get closed if there has been no activity in the past 180 days. The issue will be automatically closed in 14 days. If you wish to keep this issue open, please add a new comment.

[autoantwort]

One could add the no-stale label