search

litmuschaos / litmus

Litmus helps SREs and developers practice chaos engineering in a Cloud-native way. Chaos experiments are published at the ChaosHub (https://hub.litmuschaos.io). Community notes is at https://hackmd.io/a4Zu_sH4TZGeih-xCimi3Q
https://litmuschaos.io
Apache License 2.0
4.43k stars 694 forks source link

Logs are lost if we dont disable the "revert chaos" feature #3155

Open Vr00mm opened 3 years ago

Vr00mm commented 3 years ago

What happened: Experiments logs are lost if we enable "revert-chaos" feature

What you expected to happen: Store logs somewhere in mongo or in the chaosresult cr to get some persistance on

How to reproduce it (as minimally and precisely as possible): Launch a chaos workflow with revert-chaos

Anything else we need to know?: Failed to get argo pod logs Chaos logs unavailable

ksatchit commented 3 years ago

The default policy is being changed to - disable it by default to retain the chaos resources (the ChaosEngine, to be precise) so experiment pods are kept alive to view logs. Also the name given to this toggle is being updated ti make it more intuitive/meaningful cc: @amityt

Having said that, keeping it enabled removes the chaos pods after the exp completion, & with it, logs. It may be useful for cases where the setup has logging infra -like EFK existing already.

Vr00mm commented 3 years ago

CR are stacking fast... And the agent namespace pod list is less visible The target is at least 1k tests/day I prefere clean it up.

We already have everything for centralized logs. But there is a feature in litmus.

I already have everything for metrics .... You see where I want to go ? :)