CI/CD that spans ALL modes

[coleramos425]

Describe the suggestion CI/CD that spans ALL modes in Omniperf. Obtaining access to test nodes with Instinct cards that can be used in this pipeline.

Justification At the moment CI/CD is incomplete in Omniperf - it only covers a handful of analyze mode command combinations. Without access to hardware to test Omniperf's profiling capabilities, profile mode is completely overlooked. If we obtain access to a test node, our code coverage would increase tremendously.

Additionally, there may be some room to re-think the procedure for the test cases we add/cover. At the moment, I don't think there's much of a methodology to the random list of command combinations tested. (i.e. Adding testing for --gui flag, checking values of output tables between runs, etc.)

Implementation

1. Steal an HPC Fund node from Karl
2. Access what hardware we can spare for this "testing node"
3. Some level of IT required to establish connection between workflow files and HPC Fund
4. Come to an agreement on how list of test cases will be generated

Additional Notes May want to loop @JoseSantosAMD into this as he's the engineer who set existing CI/CD up

Originally posted by @coleramos425 in https://github.com/AMDResearch/omniperf/discussions/153#discussioncomment-6806778

[coleramos425]

TODO:

• [x] An Excel spreadsheet breaking down a testing matrix

• Should include columns for mode, SoC, name of test, and arguments

• [x] Add column for "Expected Behavior"

• Agree on what behavior should is to be monitored to warrant a passing mark

• [x] Add column for "Test Apps" to above table

• [x] Set an implementation plan

• How can we encapsulate code to keep things clean/maintainable

• Separating different testing into different files

[coleramos425]

Complete