search

cnti-testcatalog / testsuite

📞📱☎️📡🌐 Cloud Native Telecom Initiative (CNTI) Test Catalog is a tool to check for and provide feedback on the use of K8s + cloud native best practices in networking applications and platforms
https://wiki.lfnetworking.org/display/LN/Test+Catalog
Apache License 2.0
169 stars 70 forks source link

Test exclusion during certification no longer working (e.g. Cert ~testname) #2048

Closed Smitholi67 closed 3 weeks ago

Smitholi67 commented 1 month ago

Describe the bug Prior to release 1.2.0, it was possible to execute the cert command and excluding specific tests (e.g. Cert ~testname). We have noticed that this no longer works.

To Reproduce Run either Cert or the new Cert Essential with a specific test that you would like to exclude (for example Cert ~oran_e2_connection).

Expected behavior The excluded test should not execute if working properly.

Screenshots If applicable, add screenshots to help explain your problem.

Device (please complete the following information):

How will this be tested? aka Acceptance Criteria (optional)

(optional: unnecessary for things like spelling errors and such)

Once this issue is address how will the fix be verified?

Additional context Add any other context about the problem here.

NOTE: you can enable higher logging level output via the command line or env var. to help with debugging

# cmd line
./cnf-testsuite -l debug test

# make sure to use -- if running from source
crystal src/cnf-testsuite.cr -- -l debug test

# env var
LOGLEVEL=DEBUG ./cnf-testsuite test

Also setting the verbose option for many tasks will add extra output to help with debugging

crystal src/cnf-testsuite.cr test_name verbose

Check usage documentation for more info about invoking commands and logging

agentpoyo commented 1 month ago

I confirmed this is broken. Perhaps the refactoring of the points.yml layout caused this to break. We'll need to investigate.

martin-mat commented 1 month ago

it is broken by this PR https://github.com/cnti-testcatalog/testsuite/pull/2012

because test/task execution was changed from SAM dependencies to explicitly calling t.invoke(task) based on content of points.yml

In order to make it working again, an alternative approach needs to be implemented (exclusion list as a parameter?). Or change current implementation of cert task.

martin-mat commented 3 weeks ago

Functionality now available as implemented in #2051 . Note that the usage is different now.