search

cnti-testcatalog / testsuite

📞📱☎️📡🌐 Cloud Native Telecom Initiative (CNTI) Test Catalog is a tool to check for and provide feedback on the use of K8s + cloud native best practices in networking applications and platforms
https://wiki.lfnetworking.org/display/LN/Test+Catalog
Apache License 2.0
169 stars 70 forks source link

[BUG] helm_deploy doesn't test the right configmap if cnf_config is only set by cnf_setup #1957

Open collivier opened 2 months ago

collivier commented 2 months ago

Describe the bug

helm_deploy doesn't test the right configmap (nginx) if cnf_config isn't set in its call. cnf-config has to be falsely set in the helm_deploy call too to make it work (please note the test fails if you run the certification as described)

To Reproduce

$ ./cnf-testsuite setup 
$ ./cnf-testsuite cnf_setup cnf-config=./example-cnfs/coredns/cnf-testsuite.yml
$ ./cnf-testsuite helm_deploy -l debug
...
I, [2024-04-08 17:37:41 +02:00 #88472]  INFO -- cnf-testsuite: check_cnf_config args: #<Sam::Args:0x7faa5e1eec60 @arr=[], @named_args={"cnf-config" => "cnfs/nginx/cnf-testsuite.yml"}>
I, [2024-04-08 17:37:41 +02:00 #88472]  INFO -- cnf-testsuite: check_cnf_config cnf: cnfs/nginx
I, [2024-04-08 17:37:41 +02:00 #88472]  INFO -- cnf-testsuite: KubectlClient::Get.configmap command: kubectl get configmap cnf-testsuite-nginx-startup-information -o json
D, [2024-04-08 17:37:41 +02:00 #88472] DEBUG -- cnf-testsuite: KubectlClient::Get.configmap output: 
I, [2024-04-08 17:37:41 +02:00 #88472]  INFO -- cnf-testsuite: KubectlClient::Get.configmap stderr: Error from server (NotFound): configmaps "cnf-testsuite-nginx-startup-information" not found

E, [2024-04-08 17:37:41 +02:00 #88472] ERROR -- cnf-testsuite: Missing hash key: "data"
E, [2024-04-08 17:37:42 +02:00 #88472] ERROR -- cnf-testsuite: /usr/lib/crystal/lib/hash.cr:1077:11 in '[]'
E, [2024-04-08 17:37:42 +02:00 #88472] ERROR -- cnf-testsuite: /usr/lib/crystal/lib/json/any.cr:103:7 in '[]'
E, [2024-04-08 17:37:42 +02:00 #88472] ERROR -- cnf-testsuite: src/tasks/workload/compatibility.cr:418:19 in '->'
E, [2024-04-08 17:37:42 +02:00 #88472] ERROR -- cnf-testsuite: /usr/lib/crystal/lib/log/log.cr:36:3 in 'all_cnfs_task_runner'
E, [2024-04-08 17:37:42 +02:00 #88472] ERROR -- cnf-testsuite: src/tasks/utils/task.cr:38:9 in 'task_runner'
E, [2024-04-08 17:37:42 +02:00 #88472] ERROR -- cnf-testsuite: src/tasks/workload/compatibility.cr:410:3 in '->'
E, [2024-04-08 17:37:42 +02:00 #88472] ERROR -- cnf-testsuite: lib/sam/src/sam/task.cr:54:39 in 'call'
E, [2024-04-08 17:37:42 +02:00 #88472] ERROR -- cnf-testsuite: lib/sam/src/sam/execution.cr:20:7 in 'invoke'
E, [2024-04-08 17:37:42 +02:00 #88472] ERROR -- cnf-testsuite: lib/sam/src/sam.cr:35:5 in 'invoke'
E, [2024-04-08 17:37:42 +02:00 #88472] ERROR -- cnf-testsuite: lib/sam/src/sam.cr:53:7 in 'process_tasks'
E, [2024-04-08 17:37:42 +02:00 #88472] ERROR -- cnf-testsuite: src/cnf-testsuite.cr:132:3 in '__crystal_main'
E, [2024-04-08 17:37:42 +02:00 #88472] ERROR -- cnf-testsuite: /usr/lib/crystal/lib/crystal/main.cr:115:5 in 'main_user_code'
E, [2024-04-08 17:37:42 +02:00 #88472] ERROR -- cnf-testsuite: /usr/lib/crystal/lib/crystal/main.cr:101:7 in 'main'
E, [2024-04-08 17:37:42 +02:00 #88472] ERROR -- cnf-testsuite: /usr/lib/crystal/lib/crystal/main.cr:127:3 in 'main'
E, [2024-04-08 17:37:42 +02:00 #88472] ERROR -- cnf-testsuite: /lib/x86_64-linux-gnu/libc.so.6 in '??'
E, [2024-04-08 17:37:42 +02:00 #88472] ERROR -- cnf-testsuite: /lib/x86_64-linux-gnu/libc.so.6 in '__libc_start_main'
E, [2024-04-08 17:37:42 +02:00 #88472] ERROR -- cnf-testsuite: ./cnf-testsuite in '_start'
E, [2024-04-08 17:37:42 +02:00 #88472] ERROR -- cnf-testsuite: ???

Calling ./cnf-testsuite helm_deploy cnf-config=./example-cnfs/coredns/cnf-testsuite.yml works 

./cnf-testsuite setup
./cnf-testsuite cnf_setup cnf-config=./example-cnfs/coredns/cnf-testsuite.yml
./cnf-testsuite helm_deploy cnf-config=./example-cnfs/coredns/cnf-testsuite.yml

Expected behavior

helm_deploy must pass as described in CNTI certification instructions https://github.com/lfn-cnti/certification/blob/main/instructions.md

collivier commented 2 months ago

@martin-mat as discussed in the last meeting.

HashNuke commented 2 months ago

This is a valid issue. Thank you for reporting this.

helm_deploy test should not require the cnf-config option to be passed to the command.

martin-mat commented 2 months ago

I cannot reproduce it:

$ ./cnf-testsuite helm_deploy -l debug
I, [2024-04-08 19:29:20 +02:00 #1005342]  INFO -- cnf-testsuite: task_runner args: #<Sam::Args:0x7f0bcdc67640 @arr=[], @named_args={}>
D, [2024-04-08 19:29:20 +02:00 #1005342] DEBUG -- cnf-testsuite: Results.file created: results/cnf-testsuite-results-20240408-192920-890.yml
I, [2024-04-08 19:29:20 +02:00 #1005342]  INFO -- cnf-testsuite: check_cnf_config args: #<Sam::Args:0x7f0bcdc67640 @arr=[], @named_args={}>
I, [2024-04-08 19:29:20 +02:00 #1005342]  INFO -- cnf-testsuite: check_cnf_config cnf:
I, [2024-04-08 19:29:20 +02:00 #1005342]  INFO -- cnf-testsuite: cnf_config_list
I, [2024-04-08 19:29:20 +02:00 #1005342]  INFO -- cnf-testsuite: find: find cnfs/* -name "cnf-testsuite.yml"
I, [2024-04-08 19:29:20 +02:00 #1005342]  INFO -- cnf-testsuite: find response: ["cnfs/coredns/cnf-testsuite.yml"]
I, [2024-04-08 19:29:20 +02:00 #1005342]  INFO -- cnf-testsuite: CNF configs found: 1
D, [2024-04-08 19:29:20 +02:00 #1005342] DEBUG -- cnf-testsuite: single_task_runner args: #<Sam::Args:0x7f0bcdc672c0 @arr=[], @named_args={"cnf-config" => "cnfs/coredns/cnf-testsuite.yml"}>
D, [2024-04-08 19:29:20 +02:00 #1005342] DEBUG -- cnf-testsuite: parse_config_yml config_yml_path: cnfs/coredns/cnf-testsuite.yml
I, [2024-04-08 19:29:20 +02:00 #1005342]  INFO -- cnf-testsuite: airgapped: false
I, [2024-04-08 19:29:20 +02:00 #1005342]  INFO -- cnf-testsuite: generate_tar_mode: false
I, [2024-04-08 19:29:20 +02:00 #1005342]  INFO -- cnf-testsuite: ensure_cnf_testsuite_yml_path
I, [2024-04-08 19:29:20 +02:00 #1005342]  INFO -- cnf-testsuite: generate_and_set_release_name
I, [2024-04-08 19:29:20 +02:00 #1005342]  INFO -- cnf-testsuite: generate_and_set_release_name config_yml_path: cnfs/coredns/cnf-testsuite.yml
I, [2024-04-08 19:29:20 +02:00 #1005342]  INFO -- cnf-testsuite: airgapped mode: false
I, [2024-04-08 19:29:20 +02:00 #1005342]  INFO -- cnf-testsuite: generate_tar_mode: false
I, [2024-04-08 19:29:20 +02:00 #1005342]  INFO -- cnf-testsuite: ensure_cnf_testsuite_yml_path
I, [2024-04-08 19:29:20 +02:00 #1005342]  INFO -- cnf-testsuite: ensure_cnf_testsuite_yml_dir
I, [2024-04-08 19:29:20 +02:00 #1005342]  INFO -- cnf-testsuite: parsed_config_file: cnfs/coredns/cnf-testsuite.yml
I, [2024-04-08 19:29:20 +02:00 #1005342]  INFO -- cnf-testsuite: src_helm_directory:
D, [2024-04-08 19:29:20 +02:00 #1005342] DEBUG -- cnf-testsuite: predefined_release_name: coredns
I, [2024-04-08 19:29:20 +02:00 #1005342]  INFO -- cnf-testsuite: parsed_config_file: cnfs/coredns/cnf-testsuite.yml
I, [2024-04-08 19:29:20 +02:00 #1005342]  INFO -- cnf-testsuite: cnf_installation_method
I, [2024-04-08 19:29:20 +02:00 #1005342]  INFO -- cnf-testsuite: cnf_installation_method config: #<Totem::Config:0x7f0bcd3bcb40>
I, [2024-04-08 19:29:20 +02:00 #1005342]  INFO -- cnf-testsuite: cnf_installation_method config: cnfs/coredns/cnf-testsuite.yml
I, [2024-04-08 19:29:20 +02:00 #1005342]  INFO -- cnf-testsuite: directory_parameter_split :
I, [2024-04-08 19:29:20 +02:00 #1005342]  INFO -- cnf-testsuite: directory_parameter_split :
I, [2024-04-08 19:29:20 +02:00 #1005342]  INFO -- cnf-testsuite: directory :  parameters:
I, [2024-04-08 19:29:20 +02:00 #1005342]  INFO -- cnf-testsuite: release_name: coredns
I, [2024-04-08 19:29:20 +02:00 #1005342]  INFO -- cnf-testsuite: helm_directory:
I, [2024-04-08 19:29:20 +02:00 #1005342]  INFO -- cnf-testsuite: manifest_directory:
I, [2024-04-08 19:29:20 +02:00 #1005342]  INFO -- cnf-testsuite: Building helm_directory and manifest_directory full paths
I, [2024-04-08 19:29:20 +02:00 #1005342]  INFO -- cnf-testsuite: full_helm_directory: /home/matyamar/src/cnf-testsuite-mod/cnfs/coredns/ exists? true
I, [2024-04-08 19:29:20 +02:00 #1005342]  INFO -- cnf-testsuite: full_manifest_directory: /home/matyamar/src/cnf-testsuite-mod/cnfs/coredns/ exists? true
D, [2024-04-08 19:29:20 +02:00 #1005342] DEBUG -- cnf-testsuite: install type count install_type: helm_chart
D, [2024-04-08 19:29:20 +02:00 #1005342] DEBUG -- cnf-testsuite: install_type: helm_directory not found in cnfs/coredns/cnf-testsuite.yml
D, [2024-04-08 19:29:20 +02:00 #1005342] DEBUG -- cnf-testsuite: install_type: manifest_directory not found in cnfs/coredns/cnf-testsuite.yml
D, [2024-04-08 19:29:20 +02:00 #1005342] DEBUG -- cnf-testsuite: installation_type_count: 1
I, [2024-04-08 19:29:20 +02:00 #1005342]  INFO -- cnf-testsuite: cnf_destination_dir config_file: cnfs/coredns/cnf-testsuite.yml
I, [2024-04-08 19:29:20 +02:00 #1005342]  INFO -- cnf-testsuite: parsed_config_file: cnfs/coredns/cnf-testsuite.yml
D, [2024-04-08 19:29:20 +02:00 #1005342] DEBUG -- cnf-testsuite: cnf_destination_dir parsed_config_file config: #<Totem::Config:0x7f0bcd3bcaa0>
I, [2024-04-08 19:29:20 +02:00 #1005342]  INFO -- cnf-testsuite: release_name: coredns
I, [2024-04-08 19:29:20 +02:00 #1005342]  INFO -- cnf-testsuite: cnf destination dir: /home/matyamar/src/cnf-testsuite-mod/cnfs/coredns
I, [2024-04-08 19:29:20 +02:00 #1005342]  INFO -- cnf-testsuite: ensure_cnf_testsuite_yml_dir
I, [2024-04-08 19:29:20 +02:00 #1005342]  INFO -- cnf-testsuite: USING EXPORTED CHART PATH
I, [2024-04-08 19:29:20 +02:00 #1005342]  INFO -- cnf-testsuite-helm_deploy: Starting test
D, [2024-04-08 19:29:20 +02:00 #1005342] DEBUG -- cnf-testsuite-helm_deploy: cnf_config: #<CNFManager::Config:0x7f0bcf47f8c0>
I, [2024-04-08 19:29:20 +02:00 #1005342]  INFO -- cnf-testsuite: check_cnf_config args: #<Sam::Args:0x7f0bcdc672c0 @arr=[], @named_args={"cnf-config" => "cnfs/coredns/cnf-testsuite.yml"}>
I, [2024-04-08 19:29:20 +02:00 #1005342]  INFO -- cnf-testsuite: check_cnf_config cnf: cnfs/coredns
I, [2024-04-08 19:29:20 +02:00 #1005342]  INFO -- cnf-testsuite: KubectlClient::Get.configmap command: kubectl get configmap cnf-testsuite-coredns-startup-information -o json
✔️  PASSED: Helm deploy successful ⚙🛠️⬆☁
D, [2024-04-08 19:29:21 +02:00 #1005342] DEBUG -- cnf-testsuite: KubectlClient::Get.configmap output: {
    "apiVersion": "v1",
    "data": {
        "e2_found": "false",
        "helm_used": "true",
        "startup_time": "30",
        "tracing_used": "false"
    },
    "immutable": true,
    "kind": "ConfigMap",
    "metadata": {
        "annotations": {
            "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"data\":{\"e2_found\":\"false\",\"helm_used\":\"true\",\"startup_time\":\"30\",\"tracing_used\":\"false\"},\"immutable\":true,\"kind\":\"ConfigMap\",\"metadata\":{\"annotations\":{},\"name\":\"cnf-testsuite-coredns-startup-information\",\"namespace\":\"default\"}}\n"
        },
        "creationTimestamp": "2024-04-08T17:11:55Z",
        "name": "cnf-testsuite-coredns-startup-information",
        "namespace": "default",
        "resourceVersion": "118760",
        "uid": "951eb004-f68d-49c4-b7e0-9223129da597"
    }
}

D, [2024-04-08 19:29:21 +02:00 #1005342] DEBUG -- cnf-testsuite: task helm_deploy emoji: ⚙🛠️⬆☁
I, [2024-04-08 19:29:21 +02:00 #1005342]  INFO -- cnf-testsuite: task_type_by_task
I, [2024-04-08 19:29:21 +02:00 #1005342]  INFO -- cnf-testsuite: points: {"name" => "helm_deploy", "emoji" => "⚙🛠️⬆☁", "tags" => ["compatibility", "dynamic", "workload", "cert", "normal"]}
D, [2024-04-08 19:29:21 +02:00 #1005342] DEBUG -- cnf-testsuite: points tags: ["compatibility", "dynamic", "workload", "cert", "normal"]
I, [2024-04-08 19:29:21 +02:00 #1005342]  INFO -- cnf-testsuite: resp: ["compatibility", "dynamic", "workload", "cert", "normal"]
I, [2024-04-08 19:29:21 +02:00 #1005342]  INFO -- cnf-testsuite: task_type x: compatibility acc:
I, [2024-04-08 19:29:21 +02:00 #1005342]  INFO -- cnf-testsuite: task_type x: dynamic acc:
I, [2024-04-08 19:29:21 +02:00 #1005342]  INFO -- cnf-testsuite: task_type x: workload acc:
I, [2024-04-08 19:29:21 +02:00 #1005342]  INFO -- cnf-testsuite: task_type x: cert acc:
I, [2024-04-08 19:29:21 +02:00 #1005342]  INFO -- cnf-testsuite: task_type x: normal acc: cert
I, [2024-04-08 19:29:21 +02:00 #1005342]  INFO -- cnf-testsuite: task_type: normal
I, [2024-04-08 19:29:21 +02:00 #1005342]  INFO -- cnf-testsuite: task_points: task: helm_deploy is worth: 5 points
I, [2024-04-08 19:29:21 +02:00 #1005342]  INFO -- cnf-testsuite: cmd: /home/matyamar/src/cnf-testsuite-mod/cnf-testsuite helm_deploy
I, [2024-04-08 19:29:21 +02:00 #1005342]  INFO -- cnf-testsuite-helm_deploy: task_runtime=208; start_time=2024-04-08 17:29:20 UTC; end_time:2024-04-08 17:29:21 UTC
I, [2024-04-08 19:29:21 +02:00 #1005342]  INFO -- cnf-testsuite: task_type_by_task
I, [2024-04-08 19:29:21 +02:00 #1005342]  INFO -- cnf-testsuite: points: {"name" => "helm_deploy", "emoji" => "⚙🛠️⬆☁", "tags" => ["compatibility", "dynamic", "workload", "cert", "normal"]}
D, [2024-04-08 19:29:21 +02:00 #1005342] DEBUG -- cnf-testsuite: points tags: ["compatibility", "dynamic", "workload", "cert", "normal"]
I, [2024-04-08 19:29:21 +02:00 #1005342]  INFO -- cnf-testsuite: resp: ["compatibility", "dynamic", "workload", "cert", "normal"]
I, [2024-04-08 19:29:21 +02:00 #1005342]  INFO -- cnf-testsuite: task_type x: compatibility acc:
I, [2024-04-08 19:29:21 +02:00 #1005342]  INFO -- cnf-testsuite: task_type x: dynamic acc:
I, [2024-04-08 19:29:21 +02:00 #1005342]  INFO -- cnf-testsuite: task_type x: workload acc:
I, [2024-04-08 19:29:21 +02:00 #1005342]  INFO -- cnf-testsuite: task_type x: cert acc:
I, [2024-04-08 19:29:21 +02:00 #1005342]  INFO -- cnf-testsuite: task_type x: normal acc: cert
I, [2024-04-08 19:29:21 +02:00 #1005342]  INFO -- cnf-testsuite: task_type: normal
I, [2024-04-08 19:29:21 +02:00 #1005342]  INFO -- cnf-testsuite: upsert_task: task: helm_deploy has status: passed and is awarded: 5 points. Runtime: 208 seconds
I, [2024-04-08 19:29:21 +02:00 #1005342]  INFO -- cnf-testsuite: results yaml: {"name" => "cnf testsuite", "testsuite_version" => "cert_20-2024-04-08-192920-3054d806", "status" => nil, "command" => "/home/matyamar/src/cnf-testsuite-mod/cnf-testsuite helm_deploy", "points" => nil, "exit_code" => 0, "items" => [{"name" => "helm_deploy", "status" => "passed", "type" => "normal", "points" => 5}]}

Questions:

collivier commented 2 months ago

@martin-mat I'm using main. Please reproduce it by calling

$ ./cnf-testsuite setup 
$ ./cnf-testsuite cnf_setup cnf-config=./example-cnfs/coredns/cnf-testsuite.yml
$ ./cnf-testsuite helm_deploy -l debug

Please find the same failure cnf_cleanup included. Please note another bug here as the sucess is falsy (configmap not found) 

$ ./cnf-testsuite cnf_cleanup cnf-config=example-cnfs/coredns
Successfully cleaned up coredns
$ ./cnf-testsuite cnf_setup cnf-config=example-cnfs/coredns
Successfully created directories for cnf-testsuite
cnf-testsuite namespace already exists on the Kubernetes cluster
cnf setup online mode
Successfully setup coredns
cnf setup online mode complete
cedric@debian:~/Devs/CNTI/testsuite$ ./cnf-testsuite helm_deploy 
✔️  PASSED: Helm deploy successful ⚙🛠️⬆☁
E, [2024-04-09 09:16:36 +02:00 #17841] ERROR -- cnf-testsuite: Missing hash key: "data"
E, [2024-04-09 09:16:36 +02:00 #17841] ERROR -- cnf-testsuite: /usr/lib/crystal/lib/hash.cr:1077:11 in '[]'
E, [2024-04-09 09:16:36 +02:00 #17841] ERROR -- cnf-testsuite: /usr/lib/crystal/lib/json/any.cr:103:7 in '[]'
E, [2024-04-09 09:16:36 +02:00 #17841] ERROR -- cnf-testsuite: src/tasks/workload/compatibility.cr:418:19 in '->'
E, [2024-04-09 09:16:36 +02:00 #17841] ERROR -- cnf-testsuite: /usr/lib/crystal/lib/log/log.cr:36:3 in 'all_cnfs_task_runner'
E, [2024-04-09 09:16:36 +02:00 #17841] ERROR -- cnf-testsuite: src/tasks/utils/task.cr:38:9 in 'task_runner'
E, [2024-04-09 09:16:36 +02:00 #17841] ERROR -- cnf-testsuite: src/tasks/workload/compatibility.cr:410:3 in '->'
E, [2024-04-09 09:16:36 +02:00 #17841] ERROR -- cnf-testsuite: lib/sam/src/sam/task.cr:54:39 in 'call'
E, [2024-04-09 09:16:36 +02:00 #17841] ERROR -- cnf-testsuite: lib/sam/src/sam/execution.cr:20:7 in 'invoke'
E, [2024-04-09 09:16:36 +02:00 #17841] ERROR -- cnf-testsuite: lib/sam/src/sam.cr:35:5 in 'invoke'
E, [2024-04-09 09:16:36 +02:00 #17841] ERROR -- cnf-testsuite: lib/sam/src/sam.cr:53:7 in 'process_tasks'
E, [2024-04-09 09:16:36 +02:00 #17841] ERROR -- cnf-testsuite: src/cnf-testsuite.cr:132:3 in '__crystal_main'
E, [2024-04-09 09:16:36 +02:00 #17841] ERROR -- cnf-testsuite: /usr/lib/crystal/lib/crystal/main.cr:115:5 in 'main_user_code'
E, [2024-04-09 09:16:36 +02:00 #17841] ERROR -- cnf-testsuite: /usr/lib/crystal/lib/crystal/main.cr:101:7 in 'main'
E, [2024-04-09 09:16:36 +02:00 #17841] ERROR -- cnf-testsuite: /usr/lib/crystal/lib/crystal/main.cr:127:3 in 'main'
E, [2024-04-09 09:16:36 +02:00 #17841] ERROR -- cnf-testsuite: /lib/x86_64-linux-gnu/libc.so.6 in '??'
E, [2024-04-09 09:16:36 +02:00 #17841] ERROR -- cnf-testsuite: /lib/x86_64-linux-gnu/libc.so.6 in '__libc_start_main'
E, [2024-04-09 09:16:36 +02:00 #17841] ERROR -- cnf-testsuite: ./cnf-testsuite in '_start'
E, [2024-04-09 09:16:36 +02:00 #17841] ERROR -- cnf-testsuite: ???
$
martin-mat commented 2 months ago

I tried to reproduce but no success. Can you provide debug logs from the failure? cedric@debian:~/Devs/CNTI/testsuite$ ./cnf-testsuite helm_deploy -l debug

collivier commented 2 months ago

@martin-mat https://github.com/cnti-testcatalog/testsuite/issues/1957#issue-2231573776