cnti-testcatalog / testsuite

📞📱☎️📡🌐 Cloud Native Telecom Initiative (CNTI) Test Catalog is a tool to check for and provide feedback on the use of K8s + cloud native best practices in networking applications and platforms
https://wiki.lfnetworking.org/display/LN/Test+Catalog
Apache License 2.0
174 stars 72 forks source link

Enhance the YAML result log with detail about failed or skipped test #2155

Open sysarch-repo opened 2 months ago

sysarch-repo commented 2 months ago

Is your feature request related to a problem? Please describe. The YAML result log can aid compliance with regulatory requirements, ensuring traceability and accountability in case audits or reviews are necessary. Example of the information stored per test:

items:
- name: increase_decrease_capacity
  status: failed
  type: essential
  points: 0
- name: node_drain
  status: skipped
  type: essential
  points: 0
- name: privileged_containers
  status: passed
  type: essential
  points: 100

In case of a failed, skipped or n/a test, there is no details about the root cause or failed resources stored in the YAML file. The stdout/stderr must be looked up to obtain this information which is a problem in machine-to-machine communications.

Describe the solution you'd like Add information in the YAML file for failed, skipped and n/a tests. Examples:

Failed test:

details: "Failed to increase replicas from 1 to 3"
exception: "ERROR -- cnf-testsuite: Pod scaling has timed-out" 
details: "Users with cluster-admin RBAC permissions found"
failed_resources: ["ServiceAccount sonobuoy-serviceaccount"]

skipped test: details: "node_drain chaos test requires the cluster to have at least two schedulable nodes"

n/a test: details: "Pods are not using SELinux"

Remediation info needs not be captured in the YAML file.

Describe alternatives you've considered Redirect the stdout/stderr output of the CNF testsuite to a file and parse the obtained data. This is ineffective as the stdout/stderr information does not seem to follow a common schema,

How will this be tested? aka Acceptance Criteria (optional)

(optional: unnecessary for things like some version bumps etc.)

Once this issue is addressed how will the fix be verified?

Additional context Add any other context or screenshots about the feature request here.