search

cnti-testcatalog / testsuite

📞📱☎️📡🌐 Cloud Native Telecom Initiative (CNTI) Test Catalog is a tool to check for and provide feedback on the use of K8s + cloud native best practices in networking applications and platforms
https://wiki.lfnetworking.org/display/LN/Test+Catalog
Apache License 2.0
169 stars 70 forks source link

[BUG] disk_fill case cannot use for statefultset resource #1844

Open pengwenwu opened 7 months ago

pengwenwu commented 7 months ago

Describe the bug When the statefulset application runs the disk_fill test case, it keeps waiting cyclically. It is found that the chaos engine fails to create a runner and an error is reported.

Warning  ChaosResourcesOperationFailed  23s   chaos-operator  (app indentification) Unable to filter app by specified info

An error is reported during the query of chaos-operator-ce:

ERROR   controller.chaosengine  Reconciler error        {"reconciler group": "litmuschaos.io", "reconciler kind": "ChaosEngine", "name": "dsp-config-center-34", "namespace": "kube-system", "error": "resource type 'deployment', err: no deployments apps with matching labels app=dsp-config-center"}

Check whether the statefulset resource cannot be tested.

Expected behavior Can return early, or directly pass the test, instead of waiting for the result cyclically.

Device (please complete the following information):

How will this be tested? aka Acceptance Criteria (optional)

(optional: unnecessary for things like spelling errors and such)

Once this issue is address how will the fix be verified?

Additional context Add any other context about the problem here.

NOTE: you can enable higher logging level output via the command line or env var. to help with debugging

# cmd line
./cnf-testsuite disk_fill -l debug offline=true wait_count=300 verbose

# debug log
```bash
I, [2023-11-22 16:14:09 UTC #56040]  INFO -- cnf-testsuite-wait_for_test: Checking experiment status  kubectl get chaosengine.litmuschaos.io dsp-config-center-17  -n kube-system -o jsonpath='{.status.experiments[0].status}'
I, [2023-11-22 16:14:09 UTC #56040]  INFO -- cnf-testsuite: pod-delete experiment status: Waiting for Job Creation
I, [2023-11-22 16:14:09 UTC #56040]  INFO -- cnf-testsuite: experimentStatus Waiting for Job Creation
I, [2023-11-22 16:14:11 UTC #56040]  INFO -- cnf-testsuite-wait_for_test: Checking experiment status  kubectl get chaosengine.litmuschaos.io dsp-config-center-17  -n kube-system -o jsonpath='{.status.experiments[0].status}'
I, [2023-11-22 16:14:11 UTC #56040]  INFO -- cnf-testsuite-wait_for_test: status_code: 0
I, [2023-11-22 16:14:11 UTC #56040]  INFO -- cnf-testsuite-wait_for_test: Checking experiment status  kubectl get chaosengine.litmuschaos.io dsp-config-center-17  -n kube-system -o jsonpath='{.status.experiments[0].status}'
I, [2023-11-22 16:14:11 UTC #56040]  INFO -- cnf-testsuite: pod-delete experiment status: Running
I, [2023-11-22 16:14:11 UTC #56040]  INFO -- cnf-testsuite: experimentStatus Running
I, [2023-11-22 16:14:13 UTC #56040]  INFO -- cnf-testsuite-wait_for_test: Checking experiment status  kubectl get chaosengine.litmuschaos.io dsp-config-center-17  -n kube-system -o jsonpath='{.status.experiments[0].status}'
I, [2023-11-22 16:14:13 UTC #56040]  INFO -- cnf-testsuite-wait_for_test: status_code: 0
I, [2023-11-22 16:14:13 UTC #56040]  INFO -- cnf-testsuite-wait_for_test: Checking experiment status  kubectl get chaosengine.litmuschaos.io dsp-config-center-17  -n kube-system -o jsonpath='{.status.experiments[0].status}'
I, [2023-11-22 16:14:13 UTC #56040]  INFO -- cnf-testsuite: pod-delete experiment status: Running
I, [2023-11-22 16:14:13 UTC #56040]  INFO -- cnf-testsuite: experimentStatus Running
I, [2023-11-22 16:14:15 UTC #56040]  INFO -- cnf-testsuite-wait_for_test: Checking experiment status  kubectl get chaosengine.litmuschaos.io dsp-config-center-17  -n kube-system -o jsonpath='{.status.experiments[0].status}'
I, [2023-11-22 16:14:15 UTC #56040]  INFO -- cnf-testsuite-wait_for_test: status_code: 0
I, [2023-11-22 16:14:15 UTC #56040]  INFO -- cnf-testsuite-wait_for_test: Checking experiment status  kubectl get chaosengine.litmuschaos.io dsp-config-center-17  -n kube-system -o jsonpath='{.status.experiments[0].status}'
I, [2023-11-22 16:14:15 UTC #56040]  INFO -- cnf-testsuite: pod-delete experiment status: Running
I, [2023-11-22 16:14:15 UTC #56040]  INFO -- cnf-testsuite: experimentStatus Running
I, [2023-11-22 16:14:17 UTC #56040]  INFO -- cnf-testsuite-wait_for_test: Checking experiment status  kubectl get chaosengine.litmuschaos.io dsp-config-center-17  -n kube-system -o jsonpath='{.status.experiments[0].status}'
I, [2023-11-22 16:14:17 UTC #56040]  INFO -- cnf-testsuite-wait_for_test: status_code: 0
I, [2023-11-22 16:14:17 UTC #56040]  INFO -- cnf-testsuite-wait_for_test: Checking experiment status  kubectl get chaosengine.litmuschaos.io dsp-config-center-17  -n kube-system -o jsonpath='{.status.experiments[0].status}'
I, [2023-11-22 16:14:17 UTC #56040]  INFO -- cnf-testsuite: pod-delete experiment status: Running
I, [2023-11-22 16:14:17 UTC #56040]  INFO -- cnf-testsuite: experimentStatus Running
I, [2023-11-22 16:14:19 UTC #56040]  INFO -- cnf-testsuite-wait_for_test: Checking experiment status  kubectl get chaosengine.litmuschaos.io dsp-config-center-17  -n kube-system -o jsonpath='{.status.experiments[0].status}'
I, [2023-11-22 16:14:20 UTC #56040]  INFO -- cnf-testsuite-wait_for_test: status_code: 0
agentpoyo commented 7 months ago

@pengwenwu

What type of k8s cluster? How many nodes? What version of the testsuite are you using currently? eg. ./cnf-testsuite --version

Any other information you can share on setup or if it's a CNF that's accessible, we can attempt to reproduce the issue to determine what's going on.

Thanks again.

pengwenwu commented 7 months ago

@pengwenwu

What type of k8s cluster? How many nodes? What version of the testsuite are you using currently? eg. ./cnf-testsuite --version

Any other information you can share on setup or if it's a CNF that's accessible, we can attempt to reproduce the issue to determine what's going on.

Thanks again.

k8s version: v1.25.3 with 3 nodes CNF TestSuite version: v0.42.2

Sorry, screenshots or other app-related information cannot be uploaded due to security requirements.

You can try setup to register a statefulset application.

agentpoyo commented 7 months ago

CNF TestSuite version: v0.42.2

Thanks for the info.

Can you attempt another test using the latest version https://github.com/cncf/cnf-testsuite/releases/tag/v0.45.0 as we fixed some bugs surrounding the chaos tests that might address the issue you are seeing.

Thanks.

pengwenwu commented 7 months ago

CNF TestSuite version: v0.42.2

Thanks for the info.

Can you attempt another test using the latest version https://github.com/cncf/cnf-testsuite/releases/tag/v0.45.0 as we fixed some bugs surrounding the chaos tests that might address the issue you are seeing.

Thanks.

I tried using cnf-testsuite v0.45.0 and still have this problem. In addition, because the waiting time of wait_for_test is increased to wait_count ≥ 1800, the execution time is longer.

kubectl describe ChaosEngine -n kube-system   dsp-config-center-47

Events:
  Type     Reason                         Age    From            Message
  ----     ------                         ----   ----            -------
  Normal   ChaosEngineInitialized         2m55s  chaos-operator  Identifying app under test & launching dsp-config-center-47-runner
  Warning  ChaosResourcesOperationFailed  2m55s  chaos-operator  (app indentification) Unable to filter app by specified info
I, [2023-12-04 09:44:48 UTC #9775]  INFO -- cnf-testsuite: install litmus offline mode
I, [2023-12-04 09:44:48 UTC #9775]  INFO -- cnf-testsuite: KubectlClient::Apply.file command: kubectl apply -f /tmp/manifests/litmus-operator-v2.14.0.yaml
D, [2023-12-04 09:44:49 UTC #9775] DEBUG -- cnf-testsuite: KubectlClient::Apply.file output: namespace/litmus created
serviceaccount/litmus created
clusterrole.rbac.authorization.k8s.io/litmus created
clusterrolebinding.rbac.authorization.k8s.io/litmus created
deployment.apps/chaos-operator-ce created
customresourcedefinition.apiextensions.k8s.io/chaosengines.litmuschaos.io created
customresourcedefinition.apiextensions.k8s.io/chaosexperiments.litmuschaos.io created
customresourcedefinition.apiextensions.k8s.io/chaosresults.litmuschaos.io created

I, [2023-12-04 09:44:49 UTC #9775]  INFO -- cnf-testsuite: KubectlClient::Apply.file command: kubectl apply -f /tmp/manifests/chaos_crds.yaml
D, [2023-12-04 09:44:50 UTC #9775] DEBUG -- cnf-testsuite: KubectlClient::Apply.file output: customresourcedefinition.apiextensions.k8s.io/chaosengines.litmuschaos.io configured
customresourcedefinition.apiextensions.k8s.io/chaosexperiments.litmuschaos.io configured
customresourcedefinition.apiextensions.k8s.io/chaosresults.litmuschaos.io unchanged

I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: task_runner args: #<Sam::Args:0x7f74c70d8960 @arr=["verbose"], @named_args={"offline" => "true", "wait_count" => "300"}>
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: cnf_config_list
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: find: find cnfs/* -name "cnf-testsuite.yml"
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: find response: ["cnfs/dsp-config-center/cnf-testsuite.yml"]
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: ensure_cnf_installed?  true
D, [2023-12-04 09:44:50 UTC #9775] DEBUG -- cnf-testsuite-verbose: args = #<Sam::Args:0x7f74c70d8960 @arr=["verbose"], @named_args={"offline" => "true", "wait_count" => "300"}>
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: check_cnf_config args: #<Sam::Args:0x7f74c70d8960 @arr=["verbose"], @named_args={"offline" => "true", "wait_count" => "300"}>
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: check_cnf_config cnf:
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: cnf_config_list
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: find: find cnfs/* -name "cnf-testsuite.yml"
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: find response: ["cnfs/dsp-config-center/cnf-testsuite.yml"]
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: CNF configs found: 1
D, [2023-12-04 09:44:50 UTC #9775] DEBUG -- cnf-testsuite: single_task_runner args: #<Sam::Args:0x7f74c70d8020 @arr=["verbose"], @named_args={"offline" => "true", "wait_count" => "300", "cnf-config" => "cnfs/dsp-config-center/cnf-testsuite.yml"}>
D, [2023-12-04 09:44:50 UTC #9775] DEBUG -- cnf-testsuite: parse_config_yml config_yml_path: cnfs/dsp-config-center/cnf-testsuite.yml
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: airgapped: false
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: generate_tar_mode: false
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: ensure_cnf_testsuite_yml_path
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: generate_and_set_release_name
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: generate_and_set_release_name config_yml_path: cnfs/dsp-config-center/cnf-testsuite.yml
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: airgapped mode: false
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: generate_tar_mode: false
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: ensure_cnf_testsuite_yml_path
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: ensure_cnf_testsuite_yml_dir
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: parsed_config_file: cnfs/dsp-config-center/cnf-testsuite.yml
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: src_helm_directory:
D, [2023-12-04 09:44:50 UTC #9775] DEBUG -- cnf-testsuite: predefined_release_name: dsp-config-center
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: parsed_config_file: cnfs/dsp-config-center/cnf-testsuite.yml
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: cnf_installation_method
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: cnf_installation_method config: #<Totem::Config:0x7f74c7028c80>
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: cnf_installation_method config: cnfs/dsp-config-center/cnf-testsuite.yml
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: directory_parameter_split :
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: directory_parameter_split :
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: directory :  parameters:
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: release_name: dsp-config-center
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: helm_directory:
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: manifest_directory: chart
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: Change manifest_directory relative path into full path
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: full_helm_directory:  exists? false
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: full_manifest_directory: /cnf-testsuite/chart exists? true
D, [2023-12-04 09:44:50 UTC #9775] DEBUG -- cnf-testsuite: install_type: helm_chart not found in cnfs/dsp-config-center/cnf-testsuite.yml
D, [2023-12-04 09:44:50 UTC #9775] DEBUG -- cnf-testsuite: install_type: helm_directory not found in cnfs/dsp-config-center/cnf-testsuite.yml
D, [2023-12-04 09:44:50 UTC #9775] DEBUG -- cnf-testsuite: install type count install_type: manifest_directory
D, [2023-12-04 09:44:50 UTC #9775] DEBUG -- cnf-testsuite: installation_type_count: 1
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: manifest_directory not empty, using: /cnf-testsuite/chart
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: cnf_destination_dir config_file: cnfs/dsp-config-center/cnf-testsuite.yml
D, [2023-12-04 09:44:50 UTC #9775] DEBUG -- cnf-testsuite: cnf_destination_dir parsed_config_file config: #<Totem::Config:0x7f74c7028be0>
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: release_name: dsp-config-center
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: cnf destination dir: /cnf-testsuite/cnfs/dsp-config-center
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: ensure_cnf_testsuite_yml_dir
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: USING EXPORTED CHART PATH
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite-disk_fill: Starting test
D, [2023-12-04 09:44:50 UTC #9775] DEBUG -- cnf-testsuite-disk_fill: cnf_config: #<CNFManager::Config:0x7f74c70dfd80>
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: install_parameters
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: directory_parameter_split : chart
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: directory : chart parameters:
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: directory_parameters :
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: namespace_from_parameters:
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: namespace_from_parameters namespace:
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: cnf_resource_ymls
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: cnf_installation_method config : CNFManager::Config
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: config_cnf_config: {destination_cnf_dir: "/cnf-testsuite/cnfs/dsp-config-center", source_cnf_file: "cnfs/dsp-config-center/cnf-testsuite.yml", source_cnf_dir: "cnfs/dsp-config-center/", yml_file_path: "cnfs/dsp-config-center/", install_method: {ManifestDirectory, "/cnf-testsuite/chart"}, manifest_directory: "chart", helm_directory: "", source_helm_directory: "", helm_chart_path: "/cnf-testsuite/cnfs/dsp-config-center/exported_chart", manifest_file_path: "/cnf-testsuite/cnfs/dsp-config-center/temp_template.yml", release_name: "dsp-config-center", service_name: "", helm_repository: {name: "", repo_url: ""}, helm_chart: "", helm_values: "", helm_install_namespace: "kube-system", rolling_update_tag: "", container_names: [{"name" => "", "rolling_update_test_tag" => "", "rolling_downgrade_test_tag" => "", "rolling_version_change_test_tag" => "", "rollback_from_tag" => ""}], white_list_container_names: ["dsp-config-center"], docker_insecure_registries: [], core_label: "", ric_label: "", fiveG_core: {amf_pod_name: "", mmc: "", mnc: "", sst: "", sd: "", tac: "", protectionScheme: "", publicKey: "", publicKeyId: "", routingIndicator: "", enabled: "", count: "", initialMSISDN: "", key: "", op: "", opType: "", type: "", apn: "", emergency: ""}, image_registry_fqdns: {}}
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: parsed_config_file: cnfs/dsp-config-center/cnf-testsuite.yml
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: cnf_installation_method
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: cnf_installation_method config: #<Totem::Config:0x7f74c7028d20>
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: cnf_installation_method config: cnfs/dsp-config-center/cnf-testsuite.yml
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: directory_parameter_split :
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: directory_parameter_split :
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: directory :  parameters:
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: release_name: dsp-config-center
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: helm_directory:
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: manifest_directory: chart
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: Change manifest_directory relative path into full path
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: full_helm_directory:  exists? false
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: full_manifest_directory: /cnf-testsuite/chart exists? true
D, [2023-12-04 09:44:50 UTC #9775] DEBUG -- cnf-testsuite: install_type: helm_chart not found in cnfs/dsp-config-center/cnf-testsuite.yml
D, [2023-12-04 09:44:50 UTC #9775] DEBUG -- cnf-testsuite: install_type: helm_directory not found in cnfs/dsp-config-center/cnf-testsuite.yml
D, [2023-12-04 09:44:50 UTC #9775] DEBUG -- cnf-testsuite: install type count install_type: manifest_directory
D, [2023-12-04 09:44:50 UTC #9775] DEBUG -- cnf-testsuite: installation_type_count: 1
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: manifest_directory not empty, using: /cnf-testsuite/chart
D, [2023-12-04 09:44:50 UTC #9775] DEBUG -- cnf-testsuite: install_method: {ManifestDirectory, "/cnf-testsuite/chart"}
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: manifest_file_list
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: manifest_directory: /cnf-testsuite/cnfs/dsp-config-center/chart
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: find /cnf-testsuite/cnfs/dsp-config-center/chart/ -name "*.yml" -o -name "*.yaml"
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: find response: ["/cnf-testsuite/cnfs/dsp-config-center/chart/manifest.yaml"]
I, [2023-12-04 09:44:50 UTC #9775]  INFO -- cnf-testsuite: parse_manifest_as_ymls template_file_name: /cnf-testsuite/cnfs/dsp-config-center/chart/manifest.yaml
...
...
...
D, [2023-12-04 09:44:53 UTC #9775] DEBUG -- cnf-testsuite: resource_labels: {"app" => "dsp-config-center", "appgroup" => "dsp-config-center-dsp-c
onfig-center_dsp-config-center", "avoid-root-stack-name" => "dsp-config-center-dsp-config-center", "stack-name" => "dsp-config-center-dsp-config-
center"}
I, [2023-12-04 09:44:53 UTC #9775]  INFO -- cnf-testsuite-dsp-config-center-47:spec_labels: Spec labels for chaos template. Key: app; Value: dsp-
config-center
I, [2023-12-04 09:44:53 UTC #9775]  INFO -- cnf-testsuite: KubectlClient::Apply.file command: kubectl apply -f /cnf-testsuite/cnfs/dsp-config-cen
ter/disk-fill-chaosengine.yml
D, [2023-12-04 09:44:55 UTC #9775] DEBUG -- cnf-testsuite: KubectlClient::Apply.file output: chaosengine.litmuschaos.io/dsp-config-center-47 crea
ted

I, [2023-12-04 09:44:55 UTC #9775]  INFO -- cnf-testsuite-wait_for_test: Checking experiment status kubectl get chaosengine.litmuschaos.io dsp-co
nfig-center-47 -n kube-system -o jsonpath='{.status.engineStatus}'
I, [2023-12-04 09:44:57 UTC #9775]  INFO -- cnf-testsuite-wait_for_test: Checking experiment status  kubectl get chaosengine.litmuschaos.io dsp-c
onfig-center-47  -n kube-system -o jsonpath='{.status.experiments[0].status}'
I, [2023-12-04 09:44:57 UTC #9775]  INFO -- cnf-testsuite-wait_for_test: status_code: 0
I, [2023-12-04 09:44:57 UTC #9775]  INFO -- cnf-testsuite-wait_for_test: Checking experiment status  kubectl get chaosengine.litmuschaos.io dsp-config-center-47  -n kube-system -o jsonpath='{.status.experiments[0].status}'
I, [2023-12-04 09:44:57 UTC #9775]  INFO -- cnf-testsuite: disk-fill experiment status:
I, [2023-12-04 09:44:57 UTC #9775]  INFO -- cnf-testsuite: experimentStatus
I, [2023-12-04 09:44:57 UTC #9775]  INFO -- cnf-testsuite: dsp-config-center-47: wait_for_test failed.
I, [2023-12-04 09:44:59 UTC #9775]  INFO -- cnf-testsuite-wait_for_test: Checking experiment status  kubectl get chaosengine.litmuschaos.io dsp-config-center-47  -n kube-system -o jsonpath='{.status.experiments[0].status}'
I, [2023-12-04 09:44:59 UTC #9775]  INFO -- cnf-testsuite-wait_for_test: status_code: 0
I, [2023-12-04 09:44:59 UTC #9775]  INFO -- cnf-testsuite-wait_for_test: Checking experiment status  kubectl get chaosengine.litmuschaos.io dsp-config-center-47  -n kube-system -o jsonpath='{.status.experiments[0].status}'
I, [2023-12-04 09:44:59 UTC #9775]  INFO -- cnf-testsuite: disk-fill experiment status:
I, [2023-12-04 09:44:59 UTC #9775]  INFO -- cnf-testsuite: dsp-config-center-47: wait_for_test failed.
I, [2023-12-04 09:45:01 UTC #9775]  INFO -- cnf-testsuite-wait_for_test: Checking experiment status  kubectl get chaosengine.litmuschaos.io dsp-c
onfig-center-47  -n kube-system -o jsonpath='{.status.experiments[0].status}'
I, [2023-12-04 09:45:01 UTC #9775]  INFO -- cnf-testsuite-wait_for_test: status_code: 0
I, [2023-12-04 09:45:01 UTC #9775]  INFO -- cnf-testsuite-wait_for_test: Checking experiment status  kubectl get chaosengine.litmuschaos.io dsp-c
onfig-center-47  -n kube-system -o jsonpath='{.status.experiments[0].status}'
I, [2023-12-04 09:45:01 UTC #9775]  INFO -- cnf-testsuite: disk-fill experiment status:
I, [2023-12-04 09:45:01 UTC #9775]  INFO -- cnf-testsuite: experimentStatus
I, [2023-12-04 09:45:01 UTC #9775]  INFO -- cnf-testsuite: dsp-config-center-47: wait_for_test failed.
I, [2023-12-04 09:45:03 UTC #9775]  INFO -- cnf-testsuite-wait_for_test: Checking experiment status  kubectl get chaosengine.litmuschaos.io dsp-c
onfig-center-47  -n kube-system -o jsonpath='{.status.experiments[0].status}'
I, [2023-12-04 09:45:03 UTC #9775]  INFO -- cnf-testsuite-wait_for_test: status_code: 0
I, [2023-12-04 09:45:03 UTC #9775]  INFO -- cnf-testsuite-wait_for_test: Checking experiment status  kubectl get chaosengine.litmuschaos.io dsp-c
onfig-center-47  -n kube-system -o jsonpath='{.status.experiments[0].status}'
I, [2023-12-04 09:45:03 UTC #9775]  INFO -- cnf-testsuite: disk-fill experiment status:
I, [2023-12-04 09:45:03 UTC #9775]  INFO -- cnf-testsuite: experimentStatus
I, [2023-12-04 09:45:03 UTC #9775]  INFO -- cnf-testsuite: dsp-config-center-47: wait_for_test failed.
kubectl get chaosengine.litmuschaos.io dsp-config-center-47  -n kube-system -oyaml

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"litmuschaos.io/v1alpha1","kind":"ChaosEngine","metadata":{"annotations":{},"name":"dsp-config-center-47","namespace":"kube-system"},"spec":{"annotationCheck":"true","appinfo":{"appkind":"deployment","applabel":"app=dsp-config-center","appns":"kube-system"},"auxiliaryAppInfo":"","chaosServiceAccount":"disk-fill-sa","engineState":"active","experiments":[{"name":"disk-fill","spec":{"components":{"env":[{"name":"EPHEMERAL_STORAGE_MEBIBYTES","value":"500"},{"name":"TARGET_CONTAINER","value":""},{"name":"FILL_PERCENTAGE","value":""},{"name":"CONTAINER_PATH","value":"/var/lib/containerd/io.containerd.grpc.v1.cri/containers/"}]}}}],"jobCleanUpPolicy":"delete"}}
  creationTimestamp: "2023-12-04T09:44:55Z"
  generation: 4
  name: dsp-config-center-47
  namespace: kube-system
  resourceVersion: "20231217"
  uid: cdb478e4-7800-4e64-8de4-490447061e6b
spec:
  annotationCheck: "true"
  appinfo:
    appkind: deployment
    applabel: app=dsp-config-center
    appns: kube-system
  chaosServiceAccount: disk-fill-sa
  components:
    runner:
      resources: {}
  engineState: stop
  experiments:
  - name: disk-fill
    spec:
      components:
        env:
        - name: EPHEMERAL_STORAGE_MEBIBYTES
          value: "500"
        - name: TARGET_CONTAINER
        - name: FILL_PERCENTAGE
        - name: CONTAINER_PATH
          value: /var/lib/containerd/io.containerd.grpc.v1.cri/containers/
        resources: {}
        statusCheckTimeouts: {}
  jobCleanUpPolicy: delete
status:
  engineStatus: stopped
  experiments: null

less cnfs/dsp-config-center/disk-fill-chaosengine.yml

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: dsp-config-center-47
  namespace: kube-system
spec:
  annotationCheck: 'true'
  engineState: 'active'
  auxiliaryAppInfo: ''
  appinfo:
    appns: 'kube-system'
    applabel: 'app=dsp-config-center'
    appkind: 'deployment'
  chaosServiceAccount: disk-fill-sa
  jobCleanUpPolicy: 'delete'
  experiments:
    - name: disk-fill
      spec:
        components:
          env:
            # specify the fill percentage according to the disk pressure required
            - name: EPHEMERAL_STORAGE_MEBIBYTES
              value: '500'

            - name: TARGET_CONTAINER
              value: ''
            - name: FILL_PERCENTAGE
              value: ''
            - name: CONTAINER_PATH
              value: '/var/lib/containerd/io.containerd.grpc.v1.cri/containers/'
kubectl get sts -l app=dsp-config-center -n kube-system
NAME                READY   AGE
dsp-config-center   2/2     36m

kubectl get deployment -l app=dsp-config-center -n kube-system
No resources found in kube-system namespace.

Is it due to applind="deployment"?

lixuna commented 2 months ago

@agentpoyo @HashNuke is this issue still happening with the latest release?