search

cnti-testcatalog / testsuite

📞📱☎️📡🌐 Cloud Native Telecom Initiative (CNTI) Test Catalog is a tool to check for and provide feedback on the use of K8s + cloud native best practices in networking applications and platforms
https://wiki.lfnetworking.org/display/LN/Test+Catalog
Apache License 2.0
174 stars 72 forks source link

[BUG] Fix or skip tests that fail when running against a Kubespray cluster. #1275

Closed denverwilliams closed 6 months ago

denverwilliams commented 2 years ago

Describe the bug The cnf-testsuite was developed and CI tested only against a Kind Cluster. Users will likely be using other K8s Platforms. We should ensure incompatible tests are either skipped or updated so they do not fail when running against a cluster that was no created using KIND. To Reproduce Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

Expected behavior A clear and concise description of what you expected to happen.

Screenshots If applicable, add screenshots to help explain your problem.

Device (please complete the following information):

How will this be tested? aka Acceptance Criteria (optional)

(optional: unnecessary for things like spelling errors and such)

Once this issue is address how will the fix be verified?

Additional context Add any other context about the problem here.

NOTE: you can enable higher logging level output via the command line or env var. to help with debugging

# cmd line
./cnf-testsuite -l debug test

# make sure to use -- if running from source
crystal src/cnf-testsuite.cr -- -l debug test

# env var
LOGLEVEL=DEBUG ./cnf-testsuite test

Also setting the verbose option for many tasks will add extra output to help with debugging

crystal src/cnf-testsuite.cr test_name verbose

Check usage documentation for more info about invoking commands and logging

denverwilliams commented 2 years ago

Spec Log with all failing tests:

2022-03-16T02:27:57.4613306Z Failures:
2022-03-16T02:27:57.4613454Z 
2022-03-16T02:27:57.4613943Z   1) Utils 'task_runner' should run a test against a single cnf if passed a cnf-config argument even if there are multiple cnfs installed
2022-03-16T02:27:57.4614899Z      Failure/Error: (task_response).should eq("✖️  FAILED: Found privileged containers: [\"cluster-tools\", \"coredns\", \"kube-proxy\"]".colorize(:red))
2022-03-16T02:27:57.4615294Z 
2022-03-16T02:27:57.4615710Z        Expected: "✖️  FAILED: Found privileged containers: [\"cluster-tools\", \"coredns\", \"kube-proxy\"]"
2022-03-16T02:27:57.4616624Z             got: "✖️  FAILED: Found privileged containers: [\"cluster-tools\", \"coredns\", \"calico-node\", \"kube-proxy\", \"node-cache\"]"
2022-03-16T02:27:57.4616983Z 
2022-03-16T02:27:57.4617124Z      # spec/utils/utils_spec.cr:202
2022-03-16T02:27:57.4617320Z 
2022-03-16T02:27:57.4617734Z   2) Platform Observability 'node_exporter' should detect the named release of the installed node_exporter
2022-03-16T02:27:57.4618375Z      Failure/Error: (/(PASSED){1}.*(Your platform is using the){1}.*(release for the node exporter){1}/ =~ response_s).should_not be_nil
2022-03-16T02:27:57.4618718Z 
2022-03-16T02:27:57.4618854Z        Expected: nil not to be nil
2022-03-16T02:27:57.4619039Z 
2022-03-16T02:27:57.4619200Z      # spec/platform/observability_spec.cr:46
2022-03-16T02:27:57.4619414Z 
2022-03-16T02:27:57.4619829Z   3) Platform Observability 'metrics_server' should detect the named release of the installed metrics_server
2022-03-16T02:27:57.4620296Z      Failure/Error: $?.success?.should be_true
2022-03-16T02:27:57.4620580Z 
2022-03-16T02:27:57.4620694Z        Expected: true
2022-03-16T02:27:57.4620957Z             got: false
2022-03-16T02:27:57.4621116Z 
2022-03-16T02:27:57.4621278Z      # spec/platform/observability_spec.cr:82
2022-03-16T02:27:57.4621487Z 
2022-03-16T02:27:57.4621816Z   4) Cluster API 'clusterapi_enabled' should pass if cluster api is installed
2022-03-16T02:27:57.4622302Z      Failure/Error: (/Cluster API is enabled/ =~ response_s).should_not be_nil
2022-03-16T02:27:57.4622563Z 
2022-03-16T02:27:57.4622693Z        Expected: nil not to be nil
2022-03-16T02:27:57.4622875Z 
2022-03-16T02:27:57.4623030Z      # spec/platform/cluster_api_spec.cr:24
2022-03-16T02:27:57.4623233Z 
2022-03-16T02:27:57.4623617Z   5) Setup 'cnf_setup/cnf_cleanup' should install/cleanup with cnf-path arg as alias for cnf-config
2022-03-16T02:27:57.4627329Z      Failure/Error: (/Successfully setup coredns/ =~ response_s).should_not be_nil
2022-03-16T02:27:57.4627645Z 
2022-03-16T02:27:57.4627799Z        Expected: nil not to be nil
2022-03-16T02:27:57.4628014Z 
2022-03-16T02:27:57.4628264Z      # spec/setup_spec.cr:87
2022-03-16T02:27:57.4628447Z 
2022-03-16T02:27:57.4629012Z   6) Setup 'cnf_setup/cnf_cleanup' should install/cleanup a cnf with a cnf-testsuite.yml
2022-03-16T02:27:57.4629528Z      Failure/Error: (/Successfully setup coredns/ =~ response_s).should_not be_nil
2022-03-16T02:27:57.4629796Z 
2022-03-16T02:27:57.4629927Z        Expected: nil not to be nil
2022-03-16T02:27:57.4630112Z 
2022-03-16T02:27:57.4630240Z      # spec/setup_spec.cr:101
2022-03-16T02:27:57.4630421Z 
2022-03-16T02:27:57.4630841Z   7) Setup 'cnf_setup/cnf_cleanup' should work with cnf-testsuite.yml that has no directory associated with it
2022-03-16T02:27:57.4631381Z      Failure/Error: (/Successfully setup coredns/ =~ response_s).should_not be_nil
2022-03-16T02:27:57.4631648Z 
2022-03-16T02:27:57.4631782Z        Expected: nil not to be nil
2022-03-16T02:27:57.4631971Z 
2022-03-16T02:27:57.4632094Z      # spec/setup_spec.cr:116
2022-03-16T02:27:57.4632285Z 
2022-03-16T02:27:57.4632623Z   8) CnfTestSuite 'testsuite all' should run the configuration lifecycle tests
2022-03-16T02:27:57.4634889Z      Failure/Error: (/PASSED: Helm readiness probe found/ =~ response_s).should_not be_nil
2022-03-16T02:27:57.4635456Z 
2022-03-16T02:27:57.4635873Z        Expected: nil not to be nil
2022-03-16T02:27:57.4636076Z 
2022-03-16T02:27:57.4636285Z      # spec/cnf_testsuite_all/cnf_testsuite_config_lifecycle_spec.cr:20
2022-03-16T02:27:57.4636540Z 
2022-03-16T02:27:57.4636891Z   9) CnfTestSuite 'testsuite all' should run all the microservice tests
2022-03-16T02:27:57.4640847Z      Failure/Error: (CNFManager::Points.all_result_test_names(CNFManager::Points.final_cnf_results_yml).sort).should eq(["reasonable_startup_time", "reasonable_image_size"].sort)
2022-03-16T02:27:57.4641819Z 
2022-03-16T02:27:57.4642195Z        Expected: ["reasonable_image_size", "reasonable_startup_time"]
2022-03-16T02:27:57.4643883Z             got: ["cni_compatible", "container_sock_mounts", "dangerous_capabilities", "database_persistence", "elastic_volumes", "external_ips", "host_pid_ipc_privileges", "hostpath_mounts", "immutable_file_systems", "ingress_egress_blocked", "insecure_capabilities", "linux_hardening", "log_output", "network_policies", "no_local_volume_configuration", "non_root_containers", "non_root_user", "open_metrics", "privilege_escalation", "privileged_containers", "prometheus_traffic", "reasonable_image_size", "reasonable_startup_time", "require_labels", "resource_policies", "routed_logs", "service_discovery", "single_process_type", "symlink_file_system", "tracing"]
2022-03-16T02:27:57.4645386Z 
2022-03-16T02:27:57.4645758Z      # spec/cnf_testsuite_all/cnf_testsuite_microservice_spec.cr:22
2022-03-16T02:27:57.4646877Z 
2022-03-16T02:27:57.4647791Z  10) CnfTestSuite 'all' should run the workloads test suite
2022-03-16T02:27:57.4648805Z      Failure/Error: (CNFManager::Points.all_result_test_names(CNFManager::Points.final_cnf_results_yml).sort).should eq(["volume_hostpath_not_found", "privileged", "increase_capacity", "decrease_capacity", "install_script_helm", "helm_chart_valid", "helm_chart_published"].sort)
2022-03-16T02:27:57.4649747Z 
2022-03-16T02:27:57.4651108Z        Expected: ["decrease_capacity", "helm_chart_published", "helm_chart_valid", "increase_capacity", "install_script_helm", "privileged", "volume_hostpath_not_found"]
2022-03-16T02:27:57.4652913Z             got: ["cni_compatible", "container_sock_mounts", "dangerous_capabilities", "database_persistence", "decrease_capacity", "elastic_volumes", "external_ips", "helm_chart_published", "helm_chart_valid", "host_pid_ipc_privileges", "hostpath_mounts", "immutable_file_systems", "increase_capacity", "ingress_egress_blocked", "insecure_capabilities", "install_script_helm", "linux_hardening", "log_output", "network_policies", "no_local_volume_configuration", "non_root_containers", "non_root_user", "open_metrics", "privilege_escalation", "privileged", "privileged_containers", "prometheus_traffic", "require_labels", "resource_policies", "routed_logs", "service_discovery", "single_process_type", "symlink_file_system", "tracing", "volume_hostpath_not_found"]
2022-03-16T02:27:57.4654492Z 
2022-03-16T02:27:57.4654845Z      # spec/cnf_testsuite_all/cnf_testsuite_spec.cr:29
2022-03-16T02:27:57.4655151Z 
2022-03-16T02:27:57.4655654Z  11) AirGap 'cnf_setup/cnf_cleanup' should install/cleanup a cnf helm chart in airgapped mode
2022-03-16T02:27:57.4656480Z      Failure/Error: (/Successfully setup coredns/ =~ response_s).should_not be_nil
2022-03-16T02:27:57.4656780Z 
2022-03-16T02:27:57.4656916Z        Expected: nil not to be nil
2022-03-16T02:27:57.4657117Z 
2022-03-16T02:27:57.4658324Z      # spec/airgap_task_spec.cr:67
2022-03-16T02:27:57.4658875Z 
2022-03-16T02:27:57.4659361Z  12) AirGap 'cnf_setup/cnf_cleanup' should install/cleanup a cnf helm directory in airgapped mode
2022-03-16T02:27:57.4659885Z      Failure/Error: (/Successfully setup coredns/ =~ response_s).should_not be_nil
2022-03-16T02:27:57.4660166Z 
2022-03-16T02:27:57.4660304Z        Expected: nil not to be nil
2022-03-16T02:27:57.4661795Z 
2022-03-16T02:27:57.4661939Z      # spec/airgap_task_spec.cr:83
2022-03-16T02:27:57.4662165Z 
2022-03-16T02:27:57.4662586Z  13) Microservice 'shared_database' should pass if no database is used by two microservices
2022-03-16T02:27:57.4663041Z      Failure/Error: $?.success?.should be_true
2022-03-16T02:27:57.4664470Z 
2022-03-16T02:27:57.4664594Z        Expected: true
2022-03-16T02:27:57.4664864Z             got: false
2022-03-16T02:27:57.4665021Z 
2022-03-16T02:27:57.4665174Z      # spec/workload/microservice_spec.cr:21
2022-03-16T02:27:57.4665381Z 
2022-03-16T02:27:57.4665927Z  14) Microservice 'shared_database' should pass if one service connects to a database but other non-service connections are made to the database
2022-03-16T02:27:57.4666463Z      Failure/Error: $?.success?.should be_true
2022-03-16T02:27:57.4666668Z 
2022-03-16T02:27:57.4667808Z        Expected: true
2022-03-16T02:27:57.4668097Z             got: false
2022-03-16T02:27:57.4668424Z 
2022-03-16T02:27:57.4668586Z      # spec/workload/microservice_spec.cr:34
2022-03-16T02:27:57.4668798Z 
2022-03-16T02:27:57.4670357Z  15) Microservice 'shared_database' should fail if two services on the cluster connect to the same database
2022-03-16T02:27:57.4670864Z      Failure/Error: $?.success?.should be_true
2022-03-16T02:27:57.4671070Z 
2022-03-16T02:27:57.4671179Z        Expected: true
2022-03-16T02:27:57.4671439Z             got: false
2022-03-16T02:27:57.4671596Z 
2022-03-16T02:27:57.4671751Z      # spec/workload/microservice_spec.cr:47
2022-03-16T02:27:57.4671955Z 
2022-03-16T02:27:57.4674241Z  16) Resilience Disk Fill Chaos 'disk_fill' A 'Good' CNF should not crash when disk fill occurs
2022-03-16T02:27:57.4674771Z      Failure/Error: (/PASSED: disk_fill chaos test passed/ =~ response_s).should_not be_nil
2022-03-16T02:27:57.4675060Z 
2022-03-16T02:27:57.4675191Z        Expected: nil not to be nil
2022-03-16T02:27:57.4675376Z 
2022-03-16T02:27:57.4675693Z      # spec/workload/resilience/disk_fill_spec.cr:22
2022-03-16T02:27:57.4675917Z 
2022-03-16T02:27:57.4676336Z  17) Resilience pod memory hog Chaos 'pod_memory_hog' A 'Good' CNF should not crash when pod memory hog occurs
2022-03-16T02:27:57.4676893Z      Failure/Error: (/PASSED: pod_memory_hog chaos test passed/ =~ response_s).should_not be_nil
2022-03-16T02:27:57.4677183Z 
2022-03-16T02:27:57.4677314Z        Expected: nil not to be nil
2022-03-16T02:27:57.4677497Z 
2022-03-16T02:27:57.4677677Z      # spec/workload/resilience/pod_memory_hog_spec.cr:22
2022-03-16T02:27:57.4677907Z 
2022-03-16T02:27:57.4678388Z  18) Resilience Pod Network corruption Chaos 'pod_network_corruption' A 'Good' CNF should not crash when network corruption occurs
2022-03-16T02:27:57.4679276Z      Failure/Error: (/PASSED: pod_network_corruption chaos test passed/ =~ response_s).should_not be_nil
2022-03-16T02:27:57.4679577Z 
2022-03-16T02:27:57.4679709Z        Expected: nil not to be nil
2022-03-16T02:27:57.4679892Z 
2022-03-16T02:27:57.4680098Z      # spec/workload/resilience/pod_network_corruption_spec.cr:22
2022-03-16T02:27:57.4680356Z 
2022-03-16T02:27:57.4680842Z  19) Resilience Pod Network duplication Chaos 'pod_network_duplication' A 'Good' CNF should not crash when network duplication occurs
2022-03-16T02:27:57.4681505Z      Failure/Error: (/PASSED: pod_network_duplication chaos test passed/ =~ response_s).should_not be_nil
2022-03-16T02:27:57.4681815Z 
2022-03-16T02:27:57.4681947Z        Expected: nil not to be nil
2022-03-16T02:27:57.4682130Z 
2022-03-16T02:27:57.4682327Z      # spec/workload/resilience/pod_network_duplication_spec.cr:22
2022-03-16T02:27:57.4682578Z 
2022-03-16T02:27:57.4683021Z  20) Resilience Pod Network Latency Chaos 'pod_network_latency' A 'Good' CNF should not crash when network latency occurs
2022-03-16T02:27:57.4683599Z      Failure/Error: (/PASSED: pod_network_latency chaos test passed/ =~ response_s).should_not be_nil
2022-03-16T02:27:57.4683898Z 
2022-03-16T02:27:57.4684028Z        Expected: nil not to be nil
2022-03-16T02:27:57.4684216Z 
2022-03-16T02:27:57.4684408Z      # spec/workload/resilience/pod_network_latency_spec.cr:22
2022-03-16T02:27:57.4684648Z 
2022-03-16T02:27:57.4685038Z  21) Resilience pod delete Chaos 'pod_io_stress' A 'Good' CNF should not crash when pod delete occurs
2022-03-16T02:27:57.4685578Z      Failure/Error: (/PASSED: pod_io_stress chaos test passed/ =~ response_s).should_not be_nil
2022-03-16T02:27:57.4685865Z 
2022-03-16T02:27:57.4685994Z        Expected: nil not to be nil
2022-03-16T02:27:57.4686177Z 
2022-03-16T02:27:57.4686369Z      # spec/workload/resilience/pod_io_stress_spec.cr:22
2022-03-16T02:27:57.4686596Z 
2022-03-16T02:27:57.4686876Z  22) Security 'non_root_user' should pass with a non-root cnf
2022-03-16T02:27:57.4687307Z      Failure/Error: (/Root user not found/ =~ response_s).should_not be_nil
2022-03-16T02:27:57.4687561Z 
2022-03-16T02:27:57.4687690Z        Expected: nil not to be nil
2022-03-16T02:27:57.4687874Z 
2022-03-16T02:27:57.4688127Z      # spec/workload/security_spec.cr:13
2022-03-16T02:27:57.4688333Z 
2022-03-16T02:27:57.4688598Z  23) Security 'non_root_user' should fail with a root cnf
2022-03-16T02:27:57.4689008Z      Failure/Error: (/Root user found/ =~ response_s).should_not be_nil
2022-03-16T02:27:57.4689251Z 
2022-03-16T02:27:57.4689379Z        Expected: nil not to be nil
2022-03-16T02:27:57.4689560Z 
2022-03-16T02:27:57.4689702Z      # spec/workload/security_spec.cr:27
2022-03-16T02:27:57.4689900Z 
2022-03-16T02:27:57.4690297Z  24) Security 'resource_policies' should pass on a cnf that has containers with resource limits defined
2022-03-16T02:27:57.4690859Z      Failure/Error: (/PASSED: Containers have resource limits defined/ =~ response_s).should_not be_nil
2022-03-16T02:27:57.4691156Z 
2022-03-16T02:27:57.4691285Z        Expected: nil not to be nil
2022-03-16T02:27:57.4691468Z 
2022-03-16T02:27:57.4691613Z      # spec/workload/security_spec.cr:207
2022-03-16T02:27:57.4691813Z 
2022-03-16T02:27:57.4692190Z  25) Security 'hostpath_mounts' should pass when the cnf has no containers with hostPath mounts
2022-03-16T02:27:57.4692828Z      Failure/Error: (/FAILED: Found containers with hostPath mounts/ =~ response_s).should be_nil
2022-03-16T02:27:57.4693117Z 
2022-03-16T02:27:57.4693239Z        Expected: 2065 to be nil
2022-03-16T02:27:57.4693413Z 
2022-03-16T02:27:57.4693558Z      # spec/workload/security_spec.cr:299
2022-03-16T02:27:57.4693771Z 
2022-03-16T02:27:57.4694157Z  26) Private Registry: Rolling 'rolling_update' should pass if using local registry and a port
2022-03-16T02:27:57.4694652Z      Failure/Error: (/Passed/ =~ response_s).should_not be_nil
2022-03-16T02:27:57.4694885Z 
2022-03-16T02:27:57.4695012Z        Expected: nil not to be nil
2022-03-16T02:27:57.4695193Z 
2022-03-16T02:27:57.4695337Z      # spec/workload/registry_spec.cr:96
2022-03-16T02:27:57.4695534Z 
2022-03-16T02:27:57.4695924Z  27) Private Registry: Rolling 'rolling_downgrade' should pass if using local registry and a port
2022-03-16T02:27:57.4696408Z      Failure/Error: (/Passed/ =~ response_s).should_not be_nil
2022-03-16T02:27:57.4696641Z 
2022-03-16T02:27:57.4696769Z        Expected: nil not to be nil
2022-03-16T02:27:57.4696951Z 
2022-03-16T02:27:57.4697096Z      # spec/workload/registry_spec.cr:110
2022-03-16T02:27:57.4697294Z 
2022-03-16T02:27:57.4697697Z  28) Private Registry: Rolling 'rolling_version_change' should pass if using local registry and a port
2022-03-16T02:27:57.4698180Z      Failure/Error: (/Passed/ =~ response_s).should_not be_nil
2022-03-16T02:27:57.4698413Z 
2022-03-16T02:27:57.4698544Z        Expected: nil not to be nil
2022-03-16T02:27:57.4698728Z 
2022-03-16T02:27:57.4698876Z      # spec/workload/registry_spec.cr:124
2022-03-16T02:27:57.4699075Z 
2022-03-16T02:27:57.4699382Z  29) State 'elastic_volume' should pass if the cnf uses an elastic volume
2022-03-16T02:27:57.4699864Z      Failure/Error: (/PASSED: Elastic Volumes Used/ =~ response_s).should_not be_nil
2022-03-16T02:27:57.4700138Z 
2022-03-16T02:27:57.4700265Z        Expected: nil not to be nil
2022-03-16T02:27:57.4700468Z 
2022-03-16T02:27:57.4700606Z      # spec/workload/state_spec.cr:21
2022-03-16T02:27:57.4700796Z 
2022-03-16T02:27:57.4701243Z  30) State 'database_persistence' should pass if the cnf uses a database that uses an elastic volume with a stateful set
2022-03-16T02:27:57.4701835Z      Failure/Error: (/PASSED: Elastic Volumes and Statefulsets Used/ =~ response_s).should_not be_nil
2022-03-16T02:27:57.4702127Z 
2022-03-16T02:27:57.4702256Z        Expected: nil not to be nil
2022-03-16T02:27:57.4702436Z 
2022-03-16T02:27:57.4702570Z      # spec/workload/state_spec.cr:42
2022-03-16T02:27:57.4702760Z 
2022-03-16T02:27:57.4703071Z  31) Scalability 'scalability' should run all of the scalability tests
2022-03-16T02:27:57.4703482Z      Failure/Error: $?.success?.should be_true
2022-03-16T02:27:57.4703690Z 
2022-03-16T02:27:57.4703798Z        Expected: true
2022-03-16T02:27:57.4704065Z             got: false
2022-03-16T02:27:57.4704223Z 
2022-03-16T02:27:57.4704449Z      # spec/workload/scalability_spec.cr:25
2022-03-16T02:27:57.4704660Z 
2022-03-16T02:27:57.4705038Z  32) CnfTestSuite 'require_labels' should pass if a cnf has the app.kubernetes.io/name label
2022-03-16T02:27:57.4712316Z      Failure/Error: (/PASSED: Pods have the app.kubernetes.io\/name label/ =~ response_s).should_not be_nil
2022-03-16T02:27:57.4712637Z 
2022-03-16T02:27:57.4712768Z        Expected: nil not to be nil
2022-03-16T02:27:57.4712949Z 
2022-03-16T02:27:57.4713108Z      # spec/workload/configuration_spec.cr:444
2022-03-16T02:27:57.4713320Z 
2022-03-16T02:27:57.4713957Z  33) ReleaseManager '#ReleaseManager::GithubReleaseManager.upsert_release' should return the upserted release and asset response
2022-03-16T02:27:57.4714332Z 
2022-03-16T02:27:57.4781896Z        Missing hash key: "id" (KeyError)
2022-03-16T02:27:57.4782556Z          from /usr/share/crystal/src/hash.cr:1027:11 in '[]'
2022-03-16T02:27:57.4783070Z          from /usr/share/crystal/src/json/any.cr:102:7 in '[]'
2022-03-16T02:27:57.4783616Z          from src/tasks/utils/release_manager.cr:129:33 in 'upsert_release'
2022-03-16T02:27:57.4784396Z          from spec/utils/release_manager_spec.cr:47:30 in '->'
2022-03-16T02:27:57.4784933Z          from /usr/share/crystal/src/primitives.cr:255:3 in 'internal_run'
2022-03-16T02:27:57.4785469Z          from /usr/share/crystal/src/spec/example.cr:33:16 in 'run'
2022-03-16T02:27:57.4786006Z          from /usr/share/crystal/src/spec/context.cr:18:23 in 'internal_run'
2022-03-16T02:27:57.4786537Z          from /usr/share/crystal/src/spec/context.cr:330:7 in 'run'
2022-03-16T02:27:57.4787066Z          from /usr/share/crystal/src/spec/context.cr:18:23 in 'internal_run'
2022-03-16T02:27:57.4787586Z          from /usr/share/crystal/src/spec/context.cr:147:7 in 'run'
2022-03-16T02:27:57.4788078Z          from /usr/share/crystal/src/spec/dsl.cr:274:7 in '->'
2022-03-16T02:27:57.4788584Z          from /usr/share/crystal/src/primitives.cr:255:3 in 'run'
2022-03-16T02:27:57.4789095Z          from /usr/share/crystal/src/crystal/main.cr:45:14 in 'main'
2022-03-16T02:27:57.4789605Z          from /usr/share/crystal/src/crystal/main.cr:119:3 in 'main'
2022-03-16T02:27:57.4789949Z          from __libc_start_main
2022-03-16T02:27:57.4790218Z          from _start
2022-03-16T02:27:57.4790469Z          from ???
2022-03-16T02:27:57.4790704Z        
2022-03-16T02:27:57.4790845Z 
2022-03-16T02:27:57.4791421Z  34) ReleaseManager '#ReleaseManager::GithubReleaseManager.delete_release' should delete the release from the found_id
2022-03-16T02:27:57.4791783Z 
2022-03-16T02:27:57.4828310Z        Missing hash key: "id" (KeyError)
2022-03-16T02:27:57.4828907Z          from /usr/share/crystal/src/hash.cr:1027:11 in '[]'
2022-03-16T02:27:57.4829393Z          from /usr/share/crystal/src/json/any.cr:102:7 in '[]'
2022-03-16T02:27:57.4829919Z          from src/tasks/utils/release_manager.cr:129:33 in 'upsert_release'
2022-03-16T02:27:57.4830443Z          from spec/utils/release_manager_spec.cr:65:30 in '->'
2022-03-16T02:27:57.4830980Z          from /usr/share/crystal/src/primitives.cr:255:3 in 'internal_run'
2022-03-16T02:27:57.4831512Z          from /usr/share/crystal/src/spec/example.cr:33:16 in 'run'
2022-03-16T02:27:57.4832042Z          from /usr/share/crystal/src/spec/context.cr:18:23 in 'internal_run'
2022-03-16T02:27:57.4832565Z          from /usr/share/crystal/src/spec/context.cr:330:7 in 'run'
2022-03-16T02:27:57.4833093Z          from /usr/share/crystal/src/spec/context.cr:18:23 in 'internal_run'
2022-03-16T02:27:57.4833612Z          from /usr/share/crystal/src/spec/context.cr:147:7 in 'run'
2022-03-16T02:27:57.4834100Z          from /usr/share/crystal/src/spec/dsl.cr:274:7 in '->'
2022-03-16T02:27:57.4834600Z          from /usr/share/crystal/src/primitives.cr:255:3 in 'run'
2022-03-16T02:27:57.4835112Z          from /usr/share/crystal/src/crystal/main.cr:45:14 in 'main'
2022-03-16T02:27:57.4835615Z          from /usr/share/crystal/src/crystal/main.cr:119:3 in 'main'
2022-03-16T02:27:57.4835963Z          from __libc_start_main
2022-03-16T02:27:57.4836455Z          from _start
2022-03-16T02:27:57.4836712Z          from ???
2022-03-16T02:27:57.4836949Z        
2022-03-16T02:27:57.4837089Z 
2022-03-16T02:27:57.4837555Z  35) Platform Observability 'prometheus_adapter' should detect the named release of the installed prometheus_adapter
2022-03-16T02:27:57.4837915Z 
2022-03-16T02:27:57.4879590Z        Helm install error: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "APIService" in version "apiregistration.k8s.io/v1beta1" (Helm::InstallationFailed)
2022-03-16T02:27:57.4880810Z          from lib/helm/helm.cr:306:7 in 'install'
2022-03-16T02:27:57.4883603Z          from spec/platform/observability_spec.cr:57:16 in '->'
2022-03-16T02:27:57.4884677Z          from /usr/share/crystal/src/primitives.cr:255:3 in 'internal_run'
2022-03-16T02:27:57.4885263Z          from /usr/share/crystal/src/spec/example.cr:33:16 in 'run'
2022-03-16T02:27:57.4885803Z          from /usr/share/crystal/src/spec/context.cr:18:23 in 'internal_run'
2022-03-16T02:27:57.4886818Z          from /usr/share/crystal/src/spec/context.cr:330:7 in 'run'
2022-03-16T02:27:57.4887358Z          from /usr/share/crystal/src/spec/context.cr:18:23 in 'internal_run'
2022-03-16T02:27:57.4887888Z          from /usr/share/crystal/src/spec/context.cr:147:7 in 'run'
2022-03-16T02:27:57.4888386Z          from /usr/share/crystal/src/spec/dsl.cr:274:7 in '->'
2022-03-16T02:27:57.4888890Z          from /usr/share/crystal/src/primitives.cr:255:3 in 'run'
2022-03-16T02:27:57.4889395Z          from /usr/share/crystal/src/crystal/main.cr:45:14 in 'main'
2022-03-16T02:27:57.4889912Z          from /usr/share/crystal/src/crystal/main.cr:119:3 in 'main'
2022-03-16T02:27:57.4890261Z          from __libc_start_main
2022-03-16T02:27:57.4890532Z          from _start
2022-03-16T02:27:57.4890786Z          from ???
2022-03-16T02:27:57.4891024Z        
2022-03-16T02:27:57.4891167Z 
2022-03-16T02:27:57.4891280Z Finished in 21:09 minutes
2022-03-16T02:27:57.4891605Z 247 examples, 32 failures, 3 errors, 0 pending
2022-03-16T02:27:57.4891828Z 
2022-03-16T02:27:57.4891937Z Failed examples:
2022-03-16T02:27:57.4892103Z 
2022-03-16T02:27:57.4892713Z crystal spec spec/utils/utils_spec.cr:171 # Utils 'task_runner' should run a test against a single cnf if passed a cnf-config argument even if there are multiple cnfs installed
2022-03-16T02:27:57.4893672Z crystal spec spec/platform/observability_spec.cr:29 # Platform Observability 'node_exporter' should detect the named release of the installed node_exporter
2022-03-16T02:27:57.4894616Z crystal spec spec/platform/observability_spec.cr:70 # Platform Observability 'metrics_server' should detect the named release of the installed metrics_server
2022-03-16T02:27:57.4895460Z crystal spec spec/platform/cluster_api_spec.cr:17 # Cluster API 'clusterapi_enabled' should pass if cluster api is installed
2022-03-16T02:27:57.4896261Z crystal spec spec/setup_spec.cr:82 # Setup 'cnf_setup/cnf_cleanup' should install/cleanup with cnf-path arg as alias for cnf-config
2022-03-16T02:27:57.4897048Z crystal spec spec/setup_spec.cr:96 # Setup 'cnf_setup/cnf_cleanup' should install/cleanup a cnf with a cnf-testsuite.yml
2022-03-16T02:27:57.4897862Z crystal spec spec/setup_spec.cr:110 # Setup 'cnf_setup/cnf_cleanup' should work with cnf-testsuite.yml that has no directory associated with it
2022-03-16T02:27:57.4898748Z crystal spec spec/cnf_testsuite_all/cnf_testsuite_config_lifecycle_spec.cr:16 # CnfTestSuite 'testsuite all' should run the configuration lifecycle tests
2022-03-16T02:27:57.4899626Z crystal spec spec/cnf_testsuite_all/cnf_testsuite_microservice_spec.cr:16 # CnfTestSuite 'testsuite all' should run all the microservice tests
2022-03-16T02:27:57.4900418Z crystal spec spec/cnf_testsuite_all/cnf_testsuite_spec.cr:16 # CnfTestSuite 'all' should run the workloads test suite
2022-03-16T02:27:57.4901202Z crystal spec spec/airgap_task_spec.cr:56 # AirGap 'cnf_setup/cnf_cleanup' should install/cleanup a cnf helm chart in airgapped mode
2022-03-16T02:27:57.4902150Z crystal spec spec/airgap_task_spec.cr:76 # AirGap 'cnf_setup/cnf_cleanup' should install/cleanup a cnf helm directory in airgapped mode
2022-03-16T02:27:57.4902999Z crystal spec spec/workload/microservice_spec.cr:16 # Microservice 'shared_database' should pass if no database is used by two microservices
2022-03-16T02:27:57.4904020Z crystal spec spec/workload/microservice_spec.cr:29 # Microservice 'shared_database' should pass if one service connects to a database but other non-service connections are made to the database
2022-03-16T02:27:57.4905009Z crystal spec spec/workload/microservice_spec.cr:42 # Microservice 'shared_database' should fail if two services on the cluster connect to the same database
2022-03-16T02:27:57.4905914Z crystal spec spec/workload/resilience/disk_fill_spec.cr:15 # Resilience Disk Fill Chaos 'disk_fill' A 'Good' CNF should not crash when disk fill occurs
2022-03-16T02:27:57.4906865Z crystal spec spec/workload/resilience/pod_memory_hog_spec.cr:15 # Resilience pod memory hog Chaos 'pod_memory_hog' A 'Good' CNF should not crash when pod memory hog occurs
2022-03-16T02:27:57.4908005Z crystal spec spec/workload/resilience/pod_network_corruption_spec.cr:15 # Resilience Pod Network corruption Chaos 'pod_network_corruption' A 'Good' CNF should not crash when network corruption occurs
2022-03-16T02:27:57.4909134Z crystal spec spec/workload/resilience/pod_network_duplication_spec.cr:15 # Resilience Pod Network duplication Chaos 'pod_network_duplication' A 'Good' CNF should not crash when network duplication occurs
2022-03-16T02:27:57.4910210Z crystal spec spec/workload/resilience/pod_network_latency_spec.cr:15 # Resilience Pod Network Latency Chaos 'pod_network_latency' A 'Good' CNF should not crash when network latency occurs
2022-03-16T02:27:57.4911200Z crystal spec spec/workload/resilience/pod_io_stress_spec.cr:15 # Resilience pod delete Chaos 'pod_io_stress' A 'Good' CNF should not crash when pod delete occurs
2022-03-16T02:27:57.4911988Z crystal spec spec/workload/security_spec.cr:7 # Security 'non_root_user' should pass with a non-root cnf
2022-03-16T02:27:57.4912688Z crystal spec spec/workload/security_spec.cr:21 # Security 'non_root_user' should fail with a root cnf
2022-03-16T02:27:57.4913488Z crystal spec spec/workload/security_spec.cr:200 # Security 'resource_policies' should pass on a cnf that has containers with resource limits defined
2022-03-16T02:27:57.4914346Z crystal spec spec/workload/security_spec.cr:291 # Security 'hostpath_mounts' should pass when the cnf has no containers with hostPath mounts
2022-03-16T02:27:57.4915188Z crystal spec spec/workload/registry_spec.cr:88 # Private Registry: Rolling 'rolling_update' should pass if using local registry and a port
2022-03-16T02:27:57.4916045Z crystal spec spec/workload/registry_spec.cr:102 # Private Registry: Rolling 'rolling_downgrade' should pass if using local registry and a port
2022-03-16T02:27:57.4916917Z crystal spec spec/workload/registry_spec.cr:116 # Private Registry: Rolling 'rolling_version_change' should pass if using local registry and a port
2022-03-16T02:27:57.4917709Z crystal spec spec/workload/state_spec.cr:15 # State 'elastic_volume' should pass if the cnf uses an elastic volume
2022-03-16T02:27:57.4918807Z crystal spec spec/workload/state_spec.cr:28 # State 'database_persistence' should pass if the cnf uses a database that uses an elastic volume with a stateful set
2022-03-16T02:27:57.4919637Z crystal spec spec/workload/scalability_spec.cr:16 # Scalability 'scalability' should run all of the scalability tests
2022-03-16T02:27:57.4920487Z crystal spec spec/workload/configuration_spec.cr:437 # CnfTestSuite 'require_labels' should pass if a cnf has the app.kubernetes.io/name label
2022-03-16T02:27:57.4921621Z crystal spec spec/utils/release_manager_spec.cr:43 # ReleaseManager '#ReleaseManager::GithubReleaseManager.upsert_release' should return the upserted release and asset response
2022-03-16T02:27:57.4922809Z crystal spec spec/utils/release_manager_spec.cr:61 # ReleaseManager '#ReleaseManager::GithubReleaseManager.delete_release' should delete the release from the found_id
2022-03-16T02:27:57.4924033Z crystal spec spec/platform/observability_spec.cr:53 # Platform Observability 'prometheus_adapter' should detect the named release of the installed prometheus_adapter
taylor commented 2 years ago

Estimate of effort 8

lixuna commented 2 years ago

If this issue is for all tests, please create smaller issues for:

  1. cert essential
  2. cert normal
  3. cert bonus
  4. tests not included in cert
taylor commented 2 years ago

Following up with some more details to what @lixuna says above ☝🏻

@denverwilliams @agentpoyo please split this ticket so that we have 3 total tickets for implementation of the kubespray support fixes covering the 3 types of certification tests:

This ticket can be used for which ever one is most appropriate, for example if the current commits listed are covering a large portion of one type of cert tests (eg. normal) then this ticket can be used for that and 2 other tickets should be created for the other types.

The implementation should be prioritized to have essential as the first priority to complete. Then normal tests followed by bonus tests.

Non-cert tests should be deferred for now until the ones in the certification are completed. A new ticket for non-certification tests can be created for that work.

lixuna commented 8 months ago

@taylor can this issue be closed?