straussmaximilian
commented
4 years ago For the Workflow test, we should set up something that runs on Linux and Mac OS X. A nice solution for Windows would be to run these tests on self-hosted cloud runners. One way to achieve this would be to use an Auto Scaling Group on AWS: https://dev.to/jimmydqv/github-self-hosted-runners-on-aws-part-2-ec2-3jhj
This however would not work for Mac Os X - but maybe we could use something like this: https://www.macstadium.com
swillems
To have a maintainable package, automated tests and performance benchmarks are crucial. I have the following tests in mind, considering a versioning scheme where we are using (X-Y-Z): X-Major, Y-Minor, Z-Patch. In terms of branching we would have a master/dev and feature branches.
Unit Tests:
Simple function tests within nbdev. They should be run for every push on every branch. Duration ~ minutes
Workflow test:
Tests to run a full pipeline (i.e., perform a search on HeLa Thermo / Bruker data). We could run them for every version, even minor on dev). Duration <1h We would auto-create a settings template for the current version and replace the file_path with the respective filenames)
Integration test:
Test to try all possible (or at least most) combinations of settings. This is something we could do for every Minor version. Duration will be several hours.
Installer Test:
I think shipping is very crucial, and we should have one-click installers ready for each patch. To compile an installer takes approx. <10 minutes so this could be done for every push on the dev branch.
UI Test:
This is a very difficult test but very important to keep a userbase. Implementing a new feature and then the GUI doesn't work anymore. The current settings scheme is very flexible so that the core functionality should be tested with the Workflow test. A proper GUI test would probably include using tools like
pyautoguithat automatically "clicks" through workflows. If we want to be fancy, we could also use this to make a screenshoted documentation for each version automatically. Ideally I think this would be for every push on the dev branch.Performance test:
The workflow test from above will not allow us to give a good estimate on performance. We will get execution time and proteins and peptides, but we should also consider metrics like quantification accuracy. For this, we should use multi-species samples with known mixing ratios that are computationally more demanding, and I would hence consider as a different kind of test. The idea would be to have a set of PRIDE datasets like PXD010012, which we always re-run. As we could the analysis results from the repository, we would also have a baseline to compare the results to. Depending on the number of datasets this could take This is something we could potentially do for every minor version.
List of potential performance test sets:
Implementation
For running those tests, I will use GitHub-Actions self-hosted runners. This would allow us to use powerful workstations to run the tests.
Ideally, we can also set up runners for each Windows / Linux and Mac.
At some point, one could also make the testing results more explorable, i.e., pushing the results to a db and having a little dashboard app that shows performance over version/time.
Also, note that we can always trigger the tests manually.
Let me know if you would suggest additional or think the current test set should be optimized.
Styleguide
We should also add an automatic style test.