cnti-testcatalog / testsuite

📞📱☎️📡🌐 Cloud Native Telecom Initiative (CNTI) Test Catalog is a tool to check for and provide feedback on the use of K8s + cloud native best practices in networking applications and platforms
https://wiki.lfnetworking.org/display/LN/Test+Catalog
Apache License 2.0
174 stars 72 forks source link

[BUG] Update generate_config to add container_names, or deprecate and update the tests that require it #1413

Open denverwilliams opened 2 years ago

denverwilliams commented 2 years ago

Describe the bug We should update the generate_config command to support adding & populating the container_names argument if we continue to use it. Currently, only the rolling_update, rolling_downgrade and rolling_version_change tests need this argument and they're currently excluded from the certification? So, as long as these tests remain excluded, this ticket doesn't need to be addressed.

To Reproduce Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

Expected behavior A clear and concise description of what you expected to happen.

Screenshots If applicable, add screenshots to help explain your problem.

Device (please complete the following information):

How will this be tested? aka Acceptance Criteria (optional)

(optional: unnecessary for things like spelling errors and such)

Once this issue is address how will the fix be verified?

Additional context Add any other context about the problem here.


NOTE: you can enable higher logging level output via the command line or env var. to help with debugging

# cmd line
./cnf-testsuite -l debug test

# make sure to use -- if running from source
crystal src/cnf-testsuite.cr -- -l debug test

# env var
LOGLEVEL=DEBUG ./cnf-testsuite test

Also setting the verbose option for many tasks will add extra output to help with debugging

crystal src/cnf-testsuite.cr test_name verbose

Check usage documentation for more info about invoking commands and logging

lixuna commented 8 months ago

@HashNuke @agentpoyo @denverwilliams is still issue still relevant for CNTI?

HashNuke commented 8 months ago

Yes issue still relevant. Tagging appropriately for contributor visibility.

Notes for contributors

The generate_config helps generate a cnf-testsuite.yml file if the cnf is a helm chart or a manifest directory. It does not currently generate any config options relevant for running the rolling_* tests as mentioned in the ticket description.

spec/setup_spec.cr has some automated tests that have example usage of this command.

We could generate these fields as comments in the resulting config file, along with example values and display a note in the command's output that these fields need to be set appropriately for the relevant tests to run (instead of being skipped).

If we go any further and try to fetch previous versions of the container images in order to set useful values for the rolling_* config fields, then we would have to handle a few more scenarios (Example: What to do if the images are from a private registry? Which containers do we use for the test?)

kosstennbl commented 2 months ago

Even with new config generation (#2138) - this issue will still stay still relevant. As I mentioned in #1321, I think that these tests should be redesigned, and possibly, after this redesign, additional info in CNF config wouldn't be needed. On the other hand, I think, that config generation primarily serves for new testsuite users, and adding configuration for this small, non-cert test group is not that necessary.