Nutrition Framework - Proper Release Testing

iffy-pi / apple-shortcuts

A repository for handling the updates and versioning of my apple shortcuts

https://iffy-pi.github.io/apple-shortcuts/

2 stars 0 forks source link

Nutrition Framework - Proper Release Testing #62

Open iffy-pi opened 3 months ago

iffy-pi commented 3 months ago

We need to define proper release testing cycles when publishing new version of Nutrition Framework. Too many times have we run into releasing bugs that should have been caught. Traditionally, it's been difficult to do it because the process cannot be automated. But it seems I have to take on the manual work of making sure everything is working on new versions.

We will use this issue to track how we can create a rigid test cycle and a suite of functionality to test for each shortcut involved in the framework

iffy-pi commented 3 months ago

As of now, some areas that we need to test:

Test the public installer updating with the beta pipeline to make sure it picks up new updates
Test the prerelease installer with the beta pipeline to make sure it doesn't request updates
Test the prelease installer installs with the beta pipeline to make sure it works and downloads required shortcuts as expected
Do some basic functionality checks with this
(After promoting changes to main pipeline) Test public installer updating with main pipeline to make sure it picks up new updates
Test public installer with installing to make sure it works

iffy-pi commented 3 months ago

In general we need to look at:

How we are ensuring users get the correct update experience and install experience
Ensuring all shortcuts function as appropriately (Tests for each shortcut in the system to verify their functionality)
For a shortcut release, run the tests associated with all the shortcuts