nirs
commented
7 months ago Example errors that are impossible to debug without collecting data:
- If this is an issue of slow image pull, we can see a Pulled event in
kubectl events -A - If this is an image pull error, description of the pod will show the issue
- If this is an issue with the operator pod the pod log can explain the issue.
- If the cluster was overloaded by some process, top output will show this
With what we have now, we can only blindly increase the timeout which may slow down the retry that can recover from this issue.
drenv.commands.Error: Command failed:
command: ('addons/rook-operator/start', 'dr1')
exitcode: 1
error:
Traceback (most recent call last):
File "/home/nsoffer/ramen/test/addons/rook-operator/start", line 55, in <module>
wait(cluster)
File "/home/nsoffer/ramen/test/addons/rook-operator/start", line 28, in wait
kubectl.rollout(
File "/home/nsoffer/ramen/test/drenv/kubectl.py", line 134, in rollout
_watch("rollout", *args, context=context, log=log)
File "/home/nsoffer/ramen/test/drenv/kubectl.py", line 157, in _watch
for line in commands.watch(*cmd, input=input):
File "/home/nsoffer/ramen/test/drenv/commands.py", line 155, in watch
raise Error(args, error, exitcode=p.returncode)
drenv.commands.Error: Command failed:
command: ('kubectl', 'rollout', '--context', 'dr1', 'status', 'deploy/rook-ceph-operator', '--namespace=rook-ceph', '--timeout=300s')
exitcode: 1
error:
error: timed out waiting for the condition
When starting an env fails in CI environment, we don't have good way to debug the issue. We need to collect data from the failed cluster to allow understanding the failure later. The data can be published for few days as build artifact.
Use cases
CI unattended build
When unattended build fails, we want to delete the environment quickly and use it to run the next job. Without collecting data from the failed system we cannot analyze the failure. We can try to reproduce the issue with a local test environment but this will not help with random errors.
Debugging a system
When debugging a system we can inspect the system manually, but this is very hard and time consuming. It is much easier to gather all the data and use grep on local files.
Creating a snapshot of the system
When debugging an issue, you may want top take a snapshot of the system before an operation, perform the operation, and compare the state of the system to the state before the operation. This can be done manually for few resources using kubectl, but in the time to copy few resources manually, you can gather all resources from the entire cluster.
Getting help from people in different time zone
When you an issue with a system, you can wait few hours for help when someone wakes up on the other side of the planet, or gather everything for the cluster and recreated your environment.
Helping upstream users
On OpenShift users can use oc adm must-gather. It seems that there is no similar tool for upstream Kubernetes.
Data to collect
Resources
It hard to tell which resources are needed to debug an issue, and many resources are well hidden (no way to discover them without knowing about the kind). We will collect all resources from the entire system.
non-namespaced resources
Logs
It is hard to tell which logs are needed to debug an issue. We will collect all logs from all pods in the system.
Nodes
Commands
Not sure we can run sos on minikube nodes, and it creates huge reports and very slow, but we can use some commands it uses to collect basic data about a failed system.
General info about the system
Can run via
minikube sshorkubectl debug.top -o %CPUfrom all nodes (see #1282)top -o %MEMfrom all nodes (see #1282)minikube
submariner
We can use subctl commands to get info about the health of the cluster.
Submariner includes also a gather command
subctl gather all, but using it will probably collect the same info we already collect.kubevirt
Maybe use https://github.com/kubevirt/must-gather?
rook
No gather tool.
Thread in rook slack: https://rook-io.slack.com/archives/C46Q5UC05/p1711399331728259
We can open an issue to add this to rook-ceph plugin.
We can use rbd and ceph commands via rook-ceph-tools pod to get info about the health of the system:
ocm
No gather tool.
Output format
For long term we want to be compatible with
oc adm must-gather, so if we create tools for analyzing gathered data, we can use the same tool for upstream and downstream.We don't know if
oc adm must-gatherworks on upstream (e.g. minikube) and if it is quick enough for development purposes.It is not clear how
oc adm must-gathercan be used to collect custom data. It looks like we need to run the tool several times with different images.We will start with a simple solution and check integration or using
oc adm must-gatherlater.Testing
I think testing the tool as part of e2e will be the most useful test.