search

cnti-testcatalog / testsuite

📞📱☎️📡🌐 Cloud Native Telecom Initiative (CNTI) Test Catalog is a tool to check for and provide feedback on the use of K8s + cloud native best practices in networking applications and platforms
https://wiki.lfnetworking.org/display/LN/Test+Catalog
Apache License 2.0
169 stars 70 forks source link

[BUG] Falsely PASSED result of specialized_init_system on nginx (sample-cnfs/sample-ndn-privileged) #1959

Closed martin-mat closed 2 months ago

martin-mat commented 2 months ago

logs.txt Describe the bug Test execution returns falsely positive result of specialized_init_system on nginx (sample-cnfs/sample-ndn-privileged)

To Reproduce It is probably environment-related issue. But, regardless of the environment issues, tests should never falsely pass.

Reproduction on my environment:

$ ./cnf-testsuite cnf_setup cnf-path=./sample-cnfs/sample-ndn-privileged $ ./cnf-testsuite specialized_init_system ✔️ 🏆PASSED: Containers use specialized init systems 🖥️ 🚀 E, [2024-04-08 22:33:16 +02:00 #1327438] ERROR -- cnf-testsuite: container_id not found for: docker://9bf0d727e866d7b2220e10e76a996eef6f6af363e9b2ea20d0e3b7da94b73d95

the error logged probably gives a clue that something went wrong, but regardless the tests should never pass in case of error.

Expected behavior In case of environment issue, the tests should report an error. In worst case, the tests should fail. It should never pass.

martin-mat commented 2 months ago

Note that the error is caused by using docker container engine instead of containerd. But the scope of the bug report is reporting PASSED despite the error.

martin-mat commented 2 months ago

Potentially multiple tests are impacted (from microservice category and maybe others) - those which defaults to Passed unless the code finds a reason to fail. In case that something unexpected happens, they may falsely pass.