search

Azure / azure-sdk-for-cpp

This repository is for active development of the Azure SDK for C++. For consumers of the SDK we recommend visiting our versioned developer docs at https://azure.github.io/azure-sdk-for-cpp.
MIT License
181 stars 127 forks source link

Improve end customer testing of acquiring packages that ship as beta, combined with other packages that are GA. #6060

Open ahsonkhan opened 1 month ago

ahsonkhan commented 1 month ago

It is important for customers to be able to adopt our specific beta packages, easily, so they can provide valuable feedback on new features, while keeping the rest of the dependency graph stable.

For example, have a test job (running in CI/nightly) where the customer can successfully build an application using a GA'd version of azure-core-cpp, with a beta version of azure-identity-cpp (or any other beta client package), acquiring them via vcpkg.

Such end-user package acquisition testing will help catch build issues that require patching the beta registry. An example of that was here: https://github.com/Azure/azure-sdk-vcpkg-betas/commit/ad5bdf2169956569826f25f11c6aa9294948e7bc https://github.com/Azure/azure-sdk-for-cpp/pull/6042

Currently, our CI pipeline tests and verifies all sources in main build well together, and the CI in the vcpkg main registry tests to ensure all GA'd packages work well. But there's a gap where beta packages are mixed with GA'd packages.

A way to simulate that scenario and repro the issue is by building an application which only contains the specific beta package being tested in its vcpkg-configuration.json, and not the entire list of packages we ship (for example, filtering this down to just azure-identity-cpp results in vcpkg getting the azure-core-cpp dependency from the GA registry): https://github.com/Azure/azure-sdk-vcpkg-betas/blob/62c56807f81f0cae7c6ee97a3eb4edc79f2c5481/vcpkg-configuration.json#L13-L30

cc @RickWinter, @ronniegeraghty

LarryOsterman commented 1 month ago

I would go well beyond this and have stages in the nightly pipeline which test: 1) Building a sample application in the repo that uses a representative set of functionality for each package. This sample would pull from the main vcpkg registry using current GA bits. 2) Building the same sample applications in the repo using the beta registry.

Note that in each case, a powershell script should be used on a clean windows-2022/Ubuntu 22.04 image which: 1) Acquires vcpkg 2) Configures vcpkg to use the beta registry if needed 3) Builds the sample

The key part here is that we have an artifact checked into our tree which automates the acquisition process for clients of the SDK. That way customers have an example of live code that performs the acquisition and NOT documentation online.

ahsonkhan commented 1 month ago

Absolutely!

One subtlety that I do want to emphasize, along with what you shared Larry, is that the mix of beta and GA packages tested together is important here. Not just only GAs or only betas, separately (which we already have some testing of those, in silos). Using the identity beta with the core GA is an example of what uncovered the types of build issues we saw here (using core beta wouldn't have hit it).

The automation you called out is also an important aspect to this.