search

litmuschaos / litmus

Litmus helps SREs and developers practice chaos engineering in a Cloud-native way. Chaos experiments are published at the ChaosHub (https://hub.litmuschaos.io). Community notes is at https://hackmd.io/a4Zu_sH4TZGeih-xCimi3Q
https://litmuschaos.io
Apache License 2.0
4.41k stars 692 forks source link

experiment pod io stress fails on persistent storage #3782

Open alelerac opened 2 years ago

alelerac commented 2 years ago

What happened: Litmus experiment "pod io stress" fails on persistent volumes.

What you expected to happen: Litmus experiment "pod io stress" successful on persistent volumes.

Where can this issue be corrected? (optional)

How to reproduce it (as minimally and precisely as possible): 1.) deploy zookeeper statefulset kubectl apply -f https://k8s.io/examples/application/zookeeper/zookeeper.yaml -n zookeeper 2.) set value of VOLUME_MOUNT_PATH to "/var/lib/zookeeper" in chaosengine 3) run experiment

Anything else we need to know?: workflow.yaml and helper pod logs are attached pod-io-stress.zip

neelanjan00 commented 2 years ago

Hi @alelerac, thanks for raising this issue. We are currently working on making the error message more productive, rather than just outputting exit status 1. We'll need some more information to debug this:

  1. Can you confirm that the path /var/lib/zookeeper is valid?
  2. Can you confirm that /var/lib/zookeeper is read and write permissive?
alelerac commented 2 years ago

Hi @neelanjan00, yes I can confirm that path is valid and permission for read and write are set. I noticed that the problem is also valid for ephemeral storage. As soon as we set a path for "VOLUME_MOUNT_PATH " the experiment fails.

neelanjan00 commented 2 years ago

@ispeakc0de can you PTAL?