search

openshmem-org / tests-sos

Sandia OpenSHMEM unit tests and performance testing suite
Other
6 stars 11 forks source link

How to check for pass/fail when running SOS tests #38

Closed tmh97 closed 4 months ago

tmh97 commented 4 months ago

Hey SOS folks,

Cornelis has been working on improving our SOS testing, performance analysis, etc. We would be very grateful if you can help us answer the following:

When running the SOS tests, what is the best way to determine if a test passes or fails? There are some tests that output things, and some that are quite silent. Is relying on the return code enough to ensure a valid pass/fail determination? What is the strategy you folks employ in your own internal validation?

Thanks in advance for any help!

cc @jack-morrison

davidozog commented 4 months ago

Hi @tmh97, thanks for the question.

Yes indeed, you can rely on the exit code to indicate whether any particular test has passed (exit code is 0) or failed (exit code is non-zero). For the SOS repo, this is how the Github workflow/action determines which "rows" in the testing matrix have passed. You can check out the entire workflow script here: https://github.com/Sandia-OpenSHMEM/SOS/blob/main/.github/workflows/ci.yml If you're unfamiliar, a prerequisite to understanding that file/script is some details in the Github docs, e.g.: https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions That said, using Github is not at all required for these tests, for example we used Travis CI for a long time and also have had success with Jenkins. But right now we rely almost entirely on Github Actions for CI/CD.

In other words, what you see within the SOS "Actions" tab (can you see those details, btw? Perhaps permissions need adjustment?) represents our primary strategy for validating SOS. However, we also manually run "make check" on various systems of interest (including OPA/OPX) and check the "shmem_perf_suite" measurements for performance regressions. This process is not automated yet, but we are working towards it, perhaps targeting internal CI that would use our own self-hosted Github runners. For now, we manually check the platforms of interest every SOS release cycle in addition to the Github Action tests.

Does that help?

jack-morrison commented 4 months ago

Hi @davidozog, thanks for this! Like Tom mentioned, we're also working to improve our testing workflow and this is exactly the info we were looking for. I can confirm that we can see the GH Actions runs, which will surely also be a helpful reference. Please feel free to close this issue out.

davidozog commented 4 months ago

Sure thing @jack-morrison. Yes, I'll close this issue and plop the above info in a Wiki page on tests-sos, and we can clean/update that as needed going forward.