search

microsoft / fast

The adaptive interface system for modern web experiences.
https://www.fast.design
Other
9.3k stars 598 forks source link

Proposal: Improvements to the release channel strategy #3872

Closed awentzel closed 3 years ago

awentzel commented 4 years ago

This is an initial stab at improving our release process with the goal to improve our process based on our existing experiences and feedback over the past couple of months. Please take the time to understand and contribute to the process so we can maintain the best product quality possible while improving developer and operational efficiencies as well.

Release Challenges (Packages)

It's cumbersome and disruptive to partner commitments, collaborator and contributor support, operations, and engaging in growth opportunities.   As a solution, we're proposing a consistent release cycle on which our community may depend on.

Release milestones are difficult to determine what has been committed and must be completed in each release.  

Process 

Publishing Packages

Publishing of NPM packages to be done on nightly/weekly cadence enabled through GH Actions running on a cron schedule and/or merge to master. These packages will be released into different channels for which engineering picks up each morning to eliminate blocking productivity. These packages will then be deployed during the release process at the time of deployment to coincide with the documentation release. Additionally, documentation on pre-releases is still available on NPM or on staging sites which will no longer be secured by Azure Active Directory.

Beachball looks promising on better supporting our requirements.

Publish to NPM

Links for inspiration.

Release channels

Milestone 2021-3 Releases

Milestone changes (future improvements) Subscoping for tooling packages, Jane?

npm publish --tag `dev`

For details https://docs.npmjs.com/adding-dist-tags-to-packages#:~:text=Publishing%20a%20package%20with%20a%20dist-tag%20%C2%A7%20By,navigate%20to%20the%20root%20directory%20of%20your%20package

This PR is related to #3889 #3543 #3660

chrisdholt commented 4 years ago

Working my way through, but I think a monthly release is way too much time between things going in and being released. Just from an internal partner standpoint, we're already stretched at every other week / every two weeks.

From previous conversations, we've discussed regular, automated nightly releases, and I would wager that would be preferred to a manual release by a majority of folks. If we had continual release which included v-diffing and unit tests as part of our build process would that be preferred to the extensive manual testing throughout the month above?

The only other thing I'm trying to reconcile is how the schedule works with other commitments we have for the product teams some of us support. Looking at this in isolation for FAST, it seems like a reasonable schedule. But when I consider that folks on my team internally have other commitments during a given week or month, and I include the potential for external contributors and their workflow - this seems difficult to stick to.

EisenbergEffect commented 4 years ago

Just had a quick convo with @awentzel about introducing different release channels to help with this. Aaron will circle back with you @chrisdholt to discuss and then update the issue with more details based on that.

awentzel commented 4 years ago

This proposal does not address the publishing of NPM packages. This schedule is intended for site releases and the schedule is distributed over an entire month to minimize work disruptions. It ensures uninterrupted work and dedicates time for TDD to improve quality while also allowing for stabilization after release.

awentzel commented 4 years ago

I've added multiple sections to further outline NPM packaging process, the use of Containers with Docker, and the use of scripts to spin up and down temporary app deployments for each PR. Beachball, could be used for NPM publishing on nightly cadence. @chrisdholt @EisenbergEffect @nicholasrice @janechu @radium-v for review and feedback, anyone else is also free to provide feedback.

awentzel commented 4 years ago

@microsoft/fast-collaborators please review prior to release 12. Thanks

EisenbergEffect commented 4 years ago

The highlights for me that I like are:

The only real hesitation I have is over week 3 in the cycle. I think that's likely to remain a dev-focused week based on sheer quantity of work and inability to actually make use of everyone testing. If we freeze the release first thing on Monday of week 4, then we could use all day Monday for testing and maybe even Tuesday if needed, with a release happening either Tuesday or Wednesday depending on how that goes. Then sites pushing out on Thursday.

janechu commented 4 years ago

For me the site release should coincide with the package release, ideally it would be same day or shortly after package publish because we intend on putting the published package version number up on the site as well. I think we also eventually will need a published changelog on the site. Correct me if I'm misinterpreting because it seems like it's being suggested that NPM release is every 2 weeks and sites are monthly.

I agree with @EisenbergEffect the move to docker is going to make the process a lot better as well as using nightly releases. I also don't mind the dedicated release branch method.

chrisdholt commented 4 years ago

I agree with @EisenbergEffect on the timelines here:

The only real hesitation I have is over week 3 in the cycle. I think that's likely to remain a dev-focused week based on sheer quantity of work and inability to actually make use of everyone testing. If we freeze the release first thing on Monday of week 4, then we could use all day Monday for testing and maybe even Tuesday if needed, with a release happening either Tuesday or Wednesday depending on how that goes. Then sites pushing out on Thursday.

As I mentioned above, I don't think part of the team will be able to join and freeze commitments for a dedicated testing sprint. I think some UI automation or vdiff tests for the components AND the sites which run on a regular cadence would make testing more of a part of the development cadence. Again, I don't want to play down the importance of testing, but the current plan to set aside a couple days to do just that - it's not feasible for part of the team due to internal commitments.

I'd also agree with @janechu on having the package release coincide with the npm publish. If the proposed timeline is still once a month, I'm unsure of a monthly cycle as opposed to our current cycle. I think a nightly will help relieve the pressure we've felt recently to "not miss", which seems to be at least some of the reason (along with the need for more testing) to extend this release to once a month.

EisenbergEffect commented 4 years ago

Agree on releasing docs along side packages. 💯

janechu commented 4 years ago

Just a note, we could put the version number on the production site for:

Once staging is opened up, the nightly releases would accumulate and we could include both:

I think we could also include a changelog for these releases. This way consumers of our nightly releases can see exactly what they're getting.

It would also be nice to include nightly releases on the prod site, but I'm not sure how we can do that without an automatic deployment and a separate compilation to grab the information for nightly releases. Which in the short term might be a hassle.

awentzel commented 4 years ago

Feedback has been re-incorporated into the top detail section. Noting that Testing and Releasing "weeks" does not infer must spend the entire week doing the activity, only that the hour of testing must be performed. This work should result in no issues or new issues to be triaged and prioritized for this release or future release. This "week" then affords time for completing the commitment. The following week similarity for releasing. The hour or two of releasing should be performed assuming testing is 100% complete and any follow up work for that week has been completed. The release will be performed and completed that Monday. If anything comes up the remainder of the week we have time to resolve. This time also allows for stability and monitoring to ensure no issues come about the release.

awentzel commented 4 years ago

@janechu nightly releases for sites would be a long term goal as require major infrastructure changes and based on full automation which I'm in favor of. Staging will continue to deploy every time a PR is merged into master. There are just a lot of moving parts to get this in so I'd rather focus on improving the current process to solve current challenges. Vdiff etc are major feature improvements that would include many details in themselves and out of scope for this process strategy.

awentzel commented 4 years ago

Added "Publish to NPM and GitHub Packages" and "Release channels" section in the issue details.

awentzel commented 4 years ago

I've updated the schedule, deleted sections not currently in-scope. Please provide opinions on NPM publishing cadence, issues, and requirements.

awentzel commented 4 years ago

I'm proposing we release npm packages upon every merge to master. dev and next tags would update on every merge to master where a minor/patch has been incremented. latest would update manually as part of monthly release cadence until a container strategy is employed whereby staging can also be tested locally and thus reducing risk of issues in staging/production and paving the way for the fully automated release of websites upon merging to master. I even wonder if staging could be eliminated as part of the infrastructure, instead of release new sites for each PR for testing/sharing. This then makes site updates part of the PR review process. Merging to master would then destroy that infrastructure and upgrade production.

EisenbergEffect commented 4 years ago

This sounds like a good plan. I think we just need to figure out the details of whether we can use beachball for the dev/next releases and how the package versions and depdencies would work.

awentzel commented 4 years ago

@kenotron, I have a couple question's on Beachball I'm hoping you can help with. Does it support multiple channels or formatting, it looks like it supports 'canary' now, but can't find the documentation. For example, something like TypeScript where we can have three channels dev next and latest For example, dev=1.0.1-dev.20201026 next=1.0.1-dev.202010.26 and latest 1.0.0.

kenotron commented 4 years ago

To answer the main question: beachball does take a "--tag", you can also have pair that with a "--no-push" to not touch the master branch. Another kind of flow would be to have several branches to have beachball publish diff & push to a next or dev branch.

Aside: It's not documented because it's sort of a beta feature right now. Collectively I really think we're all abusing the "prerelease" - symbol for the versions. The Semver 2 spec actually encourages things like build number / date formatted stuff to be +. I am not sure who else uses that format currently.

awentzel commented 4 years ago

With Semver here's what matters for sake of this proposal and my recommendation to follow Semver 2.0 with both pre-release and metadata, also shown in "For example".

defined as: x=major=backward incompatible api changes y=minor=backward compatible new features z=patch=backward compatible bug fixes -tag=unstable may not be compatible +meta=preferencial, I'm proposing data format yyyymmdd for intuitiveness like TypeScript.

formatted as: x.y.z-tag+yyymmdd

For example: 2.3.24-dev+20201027

@microsoft/fast-collaborators please provide your thoughts.

awentzel commented 4 years ago

We also will want to probably include a checkbox to ensure that the manual "beachball change" has been used to generate the /changes folder/files which are used by the CI to determine how/what is published using beachball publish.

For example:

chrisdholt commented 4 years ago

In my opinion I think the build should just fail there. There should always be a change file even if it’s just a chore. I’m not sure the checkbox is going to accomplish more than a failed build will. The more checkboxes we get the less important they seem to me

awentzel commented 4 years ago

Two new requirements:

  1. build should fail if no /changes directory has been generated, which is the output from running beachball change
  2. beachball should support metadata creation. Request submitted here. It does not currently. I recommend simply using a single tag for pre-releases using x.y.z-next and then when we get additional functionality we can add the +yyyymmss and next tag.
awentzel commented 4 years ago

I understand this can be tested with beachball canary --tag dev --registry http://localhost:4873 though still need to try and it and adding it here for documentation when I pick this task up again soon.

EisenbergEffect commented 4 years ago

@kenotron Would you take a PR to add the metadata features that @awentzel needs? If you can point me to the relevant locations in the codebase, I may be able to squeeze out some time to put that together. We're anxious to get this going for our project and happy to make necessary contributions back to Beachball to enable that.

awentzel commented 4 years ago

@EisenbergEffect from previous conversations with @kenotron you can find more details on its issue here.

awentzel commented 4 years ago

I believe everyone is in agreement to how this could be done. However, beachball requires another feature to fully implement around support for semver metadata. Until this is implemented we could not fully implement this release strategy.

EisenbergEffect commented 4 years ago

Let's not close this. We do intend to implement this as soon as we can get the Beachball PR in.

awentzel commented 3 years ago

The hardest part of this work is completed. I've started a new issue #4626 for the remainder of the future when the dependency is completed by Beachball. All important remaining details from this have been moved to the new issue. The rest is now considered complete. Closing this issue.