search

cnti-testcatalog / testsuite

📞📱☎️📡🌐 Cloud Native Telecom Initiative (CNTI) Test Catalog is a tool to check for and provide feedback on the use of K8s + cloud native best practices in networking applications and platforms
https://wiki.lfnetworking.org/display/LN/Test+Catalog
Apache License 2.0
169 stars 70 forks source link

[BUG] "resource_policies" (and 4 additional) test crashes #1999

Closed martin-mat closed 1 month ago

martin-mat commented 2 months ago

Describe the bug The upstream Kubescape test has been renamed causing the testsuite resource_policies test to fail

"resource_policies" test crashes with an error:

$ ./cnf-testsuite resource_policies -l debug
I, [2024-04-23 13:42:07 +00:00 #1388680]  INFO -- cnf-testsuite: kubescape_framework_download
I, [2024-04-23 13:42:07 +00:00 #1388680]  INFO -- cnf-testsuite: install_kubescape
I, [2024-04-23 13:42:07 +00:00 #1388680]  INFO -- cnf-testsuite: scan command: /home/ubuntu/.cnf-testsuite/tools/kubescape/kubescape scan framework nsa --use-from /home/ubuntu/.cnf-testsuite/tools/kubescape/nsa.json --exclude-namespaces kube-system,kube-public,kube-node-lease,local-path-storage,litmus,cnf-testsuite --format json --output kubescape_results.json
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: output:
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: stderr: {"level":"info","ts":"2024-04-23T13:42:07Z","msg":"ARMO security scanner starting"}
{"level":"warn","ts":"2024-04-23T13:42:08Z","msg":"current version 'v2.0.158' is not updated to the latest release: 'v3.0.0'"}
{"level":"warn","ts":"2024-04-23T13:42:08Z","msg":"Kubernetes cluster nodes scanning is disabled. This is required to collect valuable data for certain controls. You can enable it using  the --enable-host-scan flag"}
{"level":"warn","ts":"2024-04-23T13:42:08Z","msg":"Deprecated format version","run":"--format-version=v2"}
{"level":"info","ts":"2024-04-23T13:42:15Z","msg":"Downloading/Loading policy definitions"}
{"level":"info","ts":"2024-04-23T13:42:15Z","msg":"Downloaded/Loaded policy"}
{"level":"info","ts":"2024-04-23T13:42:15Z","msg":"Accessing Kubernetes objects"}
{"level":"info","ts":"2024-04-23T13:42:15Z","msg":"Accessed to Kubernetes objects"}
{"level":"info","ts":"2024-04-23T13:42:15Z","msg":"Scanning","cluster":"cnf-setup"}
{"level":"error","ts":"2024-04-23T13:42:16Z","msg":"in 'runRegoOnSingleRule', failed to compile rule, name: linux-hardening, reason: 1 error occurred: linux-hardening:23: rego_parse_error: functions must use = operator (not := operator)\n\tis_unsafe_obj(obj) := fix_paths {\n\t                   ^"}
{"level":"error","ts":"2024-04-23T13:42:16Z","msg":"in 'runRegoOnSingleRule', failed to compile rule, name: linux-hardening, reason: 1 error occurred: linux-hardening:23: rego_parse_error: functions must use = operator (not := operator)\n\tis_unsafe_obj(obj) := fix_paths {\n\t                   ^"}
{"level":"info","ts":"2024-04-23T13:42:16Z","msg":"Done scanning","cluster":"cnf-setup"}

Overall risk-score (0- Excellent, 100- All failed): 7

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Scan results have not been submitted: run kubescape with the '--submit' flag
Sign up for free: https://portal.armo.cloud/account/sign-up?utm_source=GitHub&utm_medium=CLI&utm_campaign=no_submit
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

🕵️  Run with '--verbose'/'-v' flag for detailed resources view

I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: task_runner args: #<Sam::Args:0x7fbc5997c480 @arr=[], @named_args={}>
D, [2024-04-23 13:42:16 +00:00 #1388680] DEBUG -- cnf-testsuite: Results.file created: results/cnf-testsuite-results-20240423-134216-579.yml
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: cnf_config_list
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: find: find cnfs/* -name "cnf-testsuite.yml"
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: find response: ["cnfs/coredns/cnf-testsuite.yml"]
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: ensure_cnf_installed?  true
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: check_cnf_config args: #<Sam::Args:0x7fbc5997c480 @arr=[], @named_args={}>
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: check_cnf_config cnf:
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: cnf_config_list
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: find: find cnfs/* -name "cnf-testsuite.yml"
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: find response: ["cnfs/coredns/cnf-testsuite.yml"]
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: CNF configs found: 1
D, [2024-04-23 13:42:16 +00:00 #1388680] DEBUG -- cnf-testsuite: single_task_runner args: #<Sam::Args:0x7fbc5902ada0 @arr=[], @named_args={"cnf-config" => "cnfs/coredns/cnf-testsuite.yml"}>
D, [2024-04-23 13:42:16 +00:00 #1388680] DEBUG -- cnf-testsuite: parse_config_yml config_yml_path: cnfs/coredns/cnf-testsuite.yml
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: airgapped: false
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: generate_tar_mode: false
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: ensure_cnf_testsuite_yml_path
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: generate_and_set_release_name
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: generate_and_set_release_name config_yml_path: cnfs/coredns/cnf-testsuite.yml
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: airgapped mode: false
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: generate_tar_mode: false
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: ensure_cnf_testsuite_yml_path
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: ensure_cnf_testsuite_yml_dir
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: parsed_config_file: cnfs/coredns/cnf-testsuite.yml
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: src_helm_directory:
D, [2024-04-23 13:42:16 +00:00 #1388680] DEBUG -- cnf-testsuite: predefined_release_name: coredns
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: parsed_config_file: cnfs/coredns/cnf-testsuite.yml
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: cnf_installation_method
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: cnf_installation_method config: #<Totem::Config:0x7fbc5c283640>
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: cnf_installation_method config: cnfs/coredns/cnf-testsuite.yml
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: directory_parameter_split :
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: directory_parameter_split :
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: directory :  parameters:
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: release_name: coredns
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: helm_directory:
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: manifest_directory:
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: Building helm_directory and manifest_directory full paths
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: full_helm_directory: /home/ubuntu/cnf-testsuite/cnfs/coredns/ exists? true
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: full_manifest_directory: /home/ubuntu/cnf-testsuite/cnfs/coredns/ exists? true
D, [2024-04-23 13:42:16 +00:00 #1388680] DEBUG -- cnf-testsuite: install type count install_type: helm_chart
D, [2024-04-23 13:42:16 +00:00 #1388680] DEBUG -- cnf-testsuite: install_type: helm_directory not found in cnfs/coredns/cnf-testsuite.yml
D, [2024-04-23 13:42:16 +00:00 #1388680] DEBUG -- cnf-testsuite: install_type: manifest_directory not found in cnfs/coredns/cnf-testsuite.yml
D, [2024-04-23 13:42:16 +00:00 #1388680] DEBUG -- cnf-testsuite: installation_type_count: 1
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: cnf_destination_dir config_file: cnfs/coredns/cnf-testsuite.yml
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: parsed_config_file: cnfs/coredns/cnf-testsuite.yml
D, [2024-04-23 13:42:16 +00:00 #1388680] DEBUG -- cnf-testsuite: cnf_destination_dir parsed_config_file config: #<Totem::Config:0x7fbc5c283460>
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: release_name: coredns
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: cnf destination dir: /home/ubuntu/cnf-testsuite/cnfs/coredns
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: ensure_cnf_testsuite_yml_dir
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: USING EXPORTED CHART PATH
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite-resource_policies: Starting test
D, [2024-04-23 13:42:16 +00:00 #1388680] DEBUG -- cnf-testsuite-resource_policies: cnf_config: #<CNFManager::Config:0x7fbc5c2898c0>
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: kubescape parse
I, [2024-04-23 13:42:16 +00:00 #1388680]  INFO -- cnf-testsuite: kubescape test_by_test_name
E, [2024-04-23 13:42:16 +00:00 #1388680] ERROR -- cnf-testsuite: Cast from Array(JSON::Any) to Hash(K, V) failed, at /usr/share/crystal/src/json/any.cr:274:5:274
E, [2024-04-23 13:42:17 +00:00 #1388680] ERROR -- cnf-testsuite: /usr/share/crystal/src/json/any.cr:273:3 in 'as_h'
E, [2024-04-23 13:42:17 +00:00 #1388680] ERROR -- cnf-testsuite: src/tasks/utils/kubescape.cr:69:7 in 'parse'
E, [2024-04-23 13:42:17 +00:00 #1388680] ERROR -- cnf-testsuite: src/tasks/utils/kubescape.cr:51:5 in 'parse_test_report'
E, [2024-04-23 13:42:17 +00:00 #1388680] ERROR -- cnf-testsuite: src/tasks/workload/security.cr:314:19 in '->'
E, [2024-04-23 13:42:17 +00:00 #1388680] ERROR -- cnf-testsuite: /usr/share/crystal/src/log/log.cr:36:3 in 'all_cnfs_task_runner'
E, [2024-04-23 13:42:17 +00:00 #1388680] ERROR -- cnf-testsuite: src/tasks/utils/task.cr:38:9 in 'task_runner:task'
E, [2024-04-23 13:42:17 +00:00 #1388680] ERROR -- cnf-testsuite: src/tasks/workload/security.cr:311:3 in '->'
E, [2024-04-23 13:42:17 +00:00 #1388680] ERROR -- cnf-testsuite: lib/sam/src/sam/task.cr:54:39 in 'call'
E, [2024-04-23 13:42:17 +00:00 #1388680] ERROR -- cnf-testsuite: lib/sam/src/sam/execution.cr:20:7 in 'invoke'
E, [2024-04-23 13:42:17 +00:00 #1388680] ERROR -- cnf-testsuite: lib/sam/src/sam.cr:35:5 in 'invoke'
E, [2024-04-23 13:42:17 +00:00 #1388680] ERROR -- cnf-testsuite: lib/sam/src/sam.cr:53:7 in 'process_tasks'
E, [2024-04-23 13:42:17 +00:00 #1388680] ERROR -- cnf-testsuite: src/cnf-testsuite.cr:132:3 in '__crystal_main'
E, [2024-04-23 13:42:17 +00:00 #1388680] ERROR -- cnf-testsuite: /usr/share/crystal/src/crystal/main.cr:129:5 in 'main_user_code'
E, [2024-04-23 13:42:17 +00:00 #1388680] ERROR -- cnf-testsuite: /usr/share/crystal/src/crystal/main.cr:115:7 in 'main'
E, [2024-04-23 13:42:17 +00:00 #1388680] ERROR -- cnf-testsuite: /usr/share/crystal/src/crystal/main.cr:141:3 in 'main'
E, [2024-04-23 13:42:17 +00:00 #1388680] ERROR -- cnf-testsuite: /lib/x86_64-linux-gnu/libc.so.6 in '??'
E, [2024-04-23 13:42:17 +00:00 #1388680] ERROR -- cnf-testsuite: /lib/x86_64-linux-gnu/libc.so.6 in '__libc_start_main'
E, [2024-04-23 13:42:17 +00:00 #1388680] ERROR -- cnf-testsuite: ./cnf-testsuite in '_start'
E, [2024-04-23 13:42:17 +00:00 #1388680] ERROR -- cnf-testsuite: ???
D, [2024-04-23 13:42:17 +00:00 #1388680] DEBUG -- cnf-testsuite: update_yml results: {"name" => "cnf testsuite", "testsuite_version" => "<%= CnfTestSuite::VERSION %>", "status" => nil, "points" => nil, "exit_code" => 0, "items" => []}
D, [2024-04-23 13:42:17 +00:00 #1388680] DEBUG -- cnf-testsuite: update_yml parsed_new_yml: {"name" => "cnf testsuite", "testsuite_version" => "<%= CnfTestSuite::VERSION %>", "status" => nil, "points" => nil, "exit_code" => 2, "items" => []}
I, [2024-04-23 13:42:17 +00:00 #1388680]  INFO -- cnf-testsuite: exception with skipped exit code
I, [2024-04-23 13:42:17 +00:00 #1388680]  INFO -- cnf-testsuite: results yaml: {"name" => "cnf testsuite", "testsuite_version" => "<%= CnfTestSuite::VERSION %>", "status" => nil, "points" => nil, "exit_code" => 2, "items" => []}
$

To Reproduce

Expected behavior the tests passes

Note1: The test may pass in case that old/workinging kubescape is not wiped. This may be also a reason why github actions did not detect this.

Note2: the issue was most probably introduced by this change.

1992

Note3: "resource_policies" is an essential certification test.

horecoli commented 2 months ago

I quickly analyzed this issue, and it seems that the issue is caused by this change: https://github.com/cnti-testcatalog/testsuite/pull/1992

I compared the old nsa.json with the new nsa.json taken after #1992, and unfortunately, the new nsa.json does not define a test called "Resource policies." As a result, the parsing is failing because there are no results for that test in the output JSON from Kubescape.

taylor commented 2 months ago

@HashNuke @martin-mat @wavell this test was removed upstream in this PR https://github.com/kubescape/regolibrary/pull/586

PR to remove was closed without merging. Test renamed. See @HashNuke comments below regarding

HashNuke commented 2 months ago

Looks like the test was renamed to Resource limits.

The file was renamed twice. The history can be tracked via the new file's link.

controls/resourcepolicies.json -> controls/resourcelimits.json
controls/resourcelimits.json -> controls/C-0009-resourcelimits.json
lixuna commented 2 months ago

Documentation for resource limits at: https://hub.armosec.io/docs/c-0009

taylor commented 2 months ago

See https://hub.armosec.io/docs/c-0009 which is linked from https://github.com/cnti-testcatalog/testsuite/blob/main/docs/LIST_OF_TESTS.md#resource-policies

martin-mat commented 2 months ago

resource_policies is not the only test impacted. Additional:

hostpath_mounts (also cert/essential) removed by this commit: https://github.com/kubescape/regolibrary/commit/e909c92840bce4cf8cbde9b2ad0f8819a7023d05

and 3 platform security tests: control_plane_hardening cluster_admin exposed_dashboard

horecoli commented 2 months ago

In the new version of nsaframework.json for kubescape (available at https://github.com/armosec/regolibrary/releases/download/v1.0.316/nsa ), there is not even a Resource limits test. It appears that this test has been split into two separate tests:

{ "controlID": "C-0270", "patch": { "name": "Ensure CPU limits are set" } }, { "controlID": "C-0271", "patch": { "name": "Ensure memory limits are set" } }

The split was done for that here: https://github.com/kubescape/regolibrary/pull/594

So, I suggest splitting it in the testsuite as well. This will make it more consistent and understandable for users to know which limits are missing, whether it is for CPU or memory.

HashNuke commented 2 months ago

Got some fixes in this branch that resolves this issue - https://github.com/cnti-testcatalog/testsuite/tree/kubescape-version-check

I'll let the build run and see if there is anything else to resolve.

horecoli commented 2 months ago

Okay, @HashNuke, I have seen your changes, so I will let you fix this issue inside that branch.

HashNuke commented 2 months ago

The changes required to fix the kubescape-related issues are in this PR - https://github.com/cnti-testcatalog/testsuite/pull/2004

Without these combined fixes, the main branch build would just fail. So they all have to go in together.

taylor commented 2 months ago

(copying from https://github.com/cnti-testcatalog/testsuite/pull/2004)

resource_policies test is split into: cpu_limits and memory_limits tests

Because the resouce_policies was an essential test the parts (memory and CPU) that are now two tests have been tagged as essential tests.

@martin-mat @horecoli @HashNuke

kosstennbl commented 2 months ago

Resource limits: Splitting the test seems like a reasonable decision, we were discussing this inside the team and came to the same solution (before Akash's PR).

platform:exposed_dashboard removal: I'm not sure if we should look for ways to replace it later or not. From deleted entry in Rationale.md - this test was existing to cover for a vulnerability in old versions of dashboard. If so - this test might be not needed anymore at all.

Using control id to run tests: Nice solution, didn't even knew that there's a possibility to do it like that in Kubescape. Following question: maybe we could move all tests to using control ID's instead of NSA framework? We require only single test results for each testsuite task, is there a reason to run the whole framework each time?