HenokAddis
commented
4 years ago This should not be too complicated to achieve, maybe we can settle on this tomorrow.
- The other proposal from @jeffersonking was to leverage our current ring structure (Insiders and Production) and push to PROD more frequently now. We can move internal people to PROD who we want to be more stable, but SOMEONE needs to be in a ring to tell us if things are broken. We cannot test all scenarios perfectly every time, and I believe our BETA teams need to help us out.
All-in-all, once we get some of these “shipblockers” closed out and have decided on a model for this… I think we should roll out to a broader audience in our “PROD” rings. People need to try this out and give feedback and I’d rather have a larger pool than burn my cycles getting 5 people to continually do this.
ArlenTokatlian
Today, we have Insiders and Production versions of the SARIF Viewers in both Visual Studio and VSCode. Unfortunately, given that we have much of our new infrastructure/features in the Insiders ring, everyone in Windows will also be living on that ring. Since every PR that is merged with master will get pushed to Insiders, everyone is on Canary.
With customers new to the experience and previous potential breaking changes making it to Production, we want to be more careful what features we want to release out to customers.
[EASY] Send out test bits prior to publishing
Before we push to Insiders, we validate that people have gone through a test matrix (Generation of test matrix is TBD; driven by Brian Collinwood from an email).
Create a New Release Pipeline Ring
In our build/release pipeline, we could add an additional ring structure layer before Insiders and Production called “Canary.” This would allow us to get some users on the freshest bits to stress-test some scenarios without breaking the world on any PR. This will also allow us to batch-release features/bug-fixes and share release notes with customers.
Create a Maximum Allowed Version Check via WAVE
This idea will enable us to add a feature flag in WAVE that puts people either in a "Canary" or "Insiders" state. Based on their state, there is a max approved version: Canary = update at PR time; Insiders = Do not update beyond max version.
Even though today, we batch PRs into a release, we don't have good early validation from customers. We can see that based on the creation of this issue, which is likely sourced because it was only tested on Mac. prevents us from releasing broken bits by accident. The testing would still fall on the folks who have flipped to "Canary" and get early builds.
We would need to:
Create Canary in addition to Insiders/Default settings
Update how we send releases to send on every submitted PR
Update version number revs to account for each PR being a new version
Create an max version config file
Create logic to only update to the max version if in Insiders
The current model is to produce a side build with the version model (x.0.0-y). This does not work because SARIF wants to update on reload since the version is less than the full version name in Insiders. I just tested this by installing 3.0.0-1 and it immediately tries.