mattorp
commented
3 years ago Implementation suggestion
- A new Typescript repo that contains scripts, scenes, and test definitions to perform the specified tests
- Anyone can clone it to perform tests locally
- The scripts should skip any tests already performed or marked as in progress for the same or similar system (this is easier to handle on Macs, Windows requires some segmentation to group similar systems)
- The tester should be able to toggle scenes under development (developers might want to test during development)
- The test results should be stored in CSV for (later) analysis
- Overviews of the different systems/components/scenes performance should be updated on new test results
- Let users know if a system/component fits their needs
- Let developers know how their scenes perform across systems
- CLI to run tests, access results and generate tests from a performance (tracks MIDI and converts to a test definition)
- GUI to view results and create new tests
Define tests
Test results
- Resolution
- Most/least taxing controls
Stored as mean, median, standard deviation, min, max for the scene in general:
- FPS
- CPU/GPU usage
Convenience functions
- Test if a control affects performance
For deciding on which system/component to buy it would be nice to have access to comprehensive results of different scenes + controls settings. Also for developers to test the performance across systems.
To handle continuous testing, a script to pull, perform and push any test requests not yet completed for a given setup would be helpful. Anyone willing to lend compute power could clone the repo and run the tests automatically at night.