Registry.baseline: WHY?

[reitowo]

Is your feature request related to a problem? Please describe. Many of us are personal developers, and requiring users to provide a baseline is kind of complicated. Because updating an registry repository itself is already frustrating (https://github.com/microsoft/vcpkg/discussions/17132), and now we also have to update the latest commit id in all projects' vcpkg-configuration.json, which is hard to maintain. I think it will be convenient if vcpkg use HEAD or latest commit as default behaviour.

Proposed solution When baseline field not presented in the registry, use latest commit directly.

[Chlorie]

This.

I really hope vcpkg could enable us developers to live at head. Also, well-behaved registries already maintain all the information for older versions, automatically using newer git registries does not conflict with the need for holding the versions for some specific libraries -- we can write those version constraints in the manifest vcpkg.json files anyways.

I hope that the previous behavior could be re-enabled by setting baseline to HEAD or something like that.

Thanks.

[ras0219-msft]

We do want to make it as easy as possible for developers to use the latest and greatest versions of their dependencies. However, we also believe that repeatable builds are absolutely essential for security and reliability of such a critical infrastructure component as your package manager. This is why we have baselines -- they allow you to easily update your entire dependency set to the latest versions with a one-line change, but you can always go back in time and compare with what exactly worked before.

Absolutely agreed that we need to do more to support the maintenance of git registries, with better tooling and documentation. We also hope to provide a super easy way to update your baselines via some cli command like vcpkg update.

If you absolutely must have dynamic behavior (such as a smoke-test CI pipeline in addition to your main pipeline), you can use something like git fetch <repo> HEAD && git rev-parse FETCH_HEAD which will get the latest version. You can use this to write out a very simple vcpkg-configuration.json with the latest baselines, leaving your vcpkg.json completely untouched.

[reitowo]

We do want to make it as easy as possible for developers to use the latest and greatest versions of their dependencies. However, we also believe that repeatable builds are absolutely essential for security and reliability of such a critical infrastructure component as your package manager. This is why we have baselines -- they allow you to easily update your entire dependency set to the latest versions with a one-line change, but you can always go back in time and compare with what exactly worked before.

Absolutely agreed that we need to do more to support the maintenance of git registries, with better tooling and documentation. We also hope to provide a super easy way to update your baselines via some cli command like vcpkg update.

If you absolutely must have dynamic behavior (such as a smoke-test CI pipeline in addition to your main pipeline), you can use something like git fetch <repo> HEAD && git rev-parse FETCH_HEAD which will get the latest version. You can use this to write out a very simple vcpkg-configuration.json with the latest baselines, leaving your vcpkg.json completely untouched.

Completely reasonable. We sure can use git fetch <repo> HEAD && git rev-parse FETCH_HEAD or write an vcpkg update command, and how about this: (https://github.com/microsoft/vcpkg-tool/pull/76)

{
    "registries": [
        {
            "kind": "git",
            "repository": "https://github.com/cnSchwarzer/vcpkg-registry.git",
            "packages": [ "bup", "k1ee" ],
            "baseline": "HEAD"
        }
    ]
}

When user explicitly wants baseline to keep track with HEAD, we just use the HEAD as commit SHA. Because sometimes we update our vcpkg-registry repo so often, and even do an vcpkg update every time we update is kind of complicated.

I agree with the idea about vcpkg update, that requires user to manually update the commit SHA to make sure the project build process could be reproduced exactly. However using HEAD is an ease to developers, maybe we just need to explicit use an commit SHA in our main or release branches to keep integrity, and use HEAD in our dev branches for easy use?

[ras0219]

maybe we just need to explicit use an commit SHA in our main or release branches to keep integrity

In my experience, this is very cumbersome with the way that git works -- merging forward and back from main to dev will certainly result in merge conflicts or accidentally releasing "HEAD" and forever making that release unreproducible.

There's also a potential performance problem about when vcpkg should refetch the baseline: Every vcpkg install? When you purge vcpkg_installed? Some other scheme?

I can possibly see enabling HEAD behind an experimental feature flag for now that prints nasty warnings to the user.

[reitowo]

Is it possible to specify a branch's HEAD? @ras0219

[ras0219]

Is it possible to specify a branch's HEAD?

Sorry for the delayed response; I don't know what you mean by the question.

[reitowo]

Is it possible to specify a branch's HEAD?

Sorry for the delayed response; I don't know what you mean by the question.

In my experience, this is very cumbersome with the way that git works -- merging forward and back from main to dev will certainly result in merge conflicts or accidentally releasing "HEAD" and forever making that release unreproducible.

Instead of HEAD that may cause unexpected problems because of merging dev and main, can we provide an extra field that let user specify a branch's name, that vcpkg will use that branch's latest commit. This will no longer require us to manually acquire commit hash.

[ras0219-msft]

https://github.com/microsoft/vcpkg-tool/pull/99 is the feature you're looking for, assuming you mean the catalog fetching. We still do not have plans in the short term to implement 100% unstable/floating baselines

[reitowo]

Yes, that's exactly the feature I'm looking forward! Thank you! Also thanks @strega-nil