[Bug] Spec test for 'rolling_downgrade' runs the test multiple times

cnti-testcatalog / testsuite

📞📱☎️📡🌐 Cloud Native Telecom Initiative (CNTI) Test Catalog is a tool to check for and provide feedback on the use of K8s + cloud native best practices in networking applications and platforms

https://wiki.lfnetworking.org/display/LN/Test+Catalog

Apache License 2.0

169 stars 70 forks source link

[Bug] Spec test for 'rolling_downgrade' runs the test multiple times #1946

Open kosstennbl opened 3 months ago

kosstennbl commented 3 months ago

https://github.com/cnti-testcatalog/testsuite/blob/83f08b67171cfeeac84e0e4d2af60ff521ef1393/spec/workload/configuration_spec.cr#L149-L154

'rolling_downgrade' should pass when valid version is given test has call of cnf_testsuite in a retry block, which could cause multiple calls of the same test. This seems like an incorrect behavior of spec test, maybe retries should be removed?

kosstennbl commented 2 months ago

Same issue in compatibility_spec.cr