Add unit and regression tests for conformance tests

SovereignCloudStack / standards

SCS standards in a machine readable format

https://scs.community/

Creative Commons Attribution Share Alike 4.0 International

34 stars 23 forks source link

Add unit and regression tests for conformance tests #531

Closed martinmo closed 6 months ago

martinmo commented 6 months ago

This PR adds basic unit and regression test infrastructure for our conformance tests in ./Tests using the pytest tool.

Included are

a new GitHub workflow that runs pytest tests and blocks merging if tests fail,
unit and regression tests for the most important bits of the version policy and flavor naming conformance tests,
a section in Tests/README.md explaining how to use the unit tests,
bug fixes for bugs found by the new tests,
and last but not least some refactorings I couldn't resist to perform already.

Resolves #476.

mbuechse commented 6 months ago

This seems to be progressing nicely. Will you remove the Draft state soon?

martinmo commented 6 months ago

This seems to be progressing nicely. Will you remove the Draft state soon?

Thank you! It is ready for review now. I've updated the PR description accordingly.

I took a bit longer because of layer 8 problem on my part – I spent a lot of time debuggig an issue which turned out to be caused by a tiny typo :see_no_evil:

martinmo commented 6 months ago

The markdown linter complains, I'm on it...

martinmo commented 6 months ago

Converted to draft again to block merging, I found another bug when testing with minikube, unfortunately...

CRITICAL: '<=' not supported between instances of 'NoneType' and 'K8sVersion'

I'm currently investigating.

martinmo commented 6 months ago

The issue is now resolved. TL;DR: it was bad handling of the case when the lower version of the affected version range was not specified. (The bigger picture is that the VersionRange class is not the best abstraction to cover all use cases – but because this PR is scoped to the tests, I don't want to change it completely. This will be done in #526.)

mbuechse commented 6 months ago

You are GO for launch! https://www.youtube.com/watch?v=lMtWWls4oas