kubevirt / kubevirtci

Contains cluster definitions and client tools to quickly spin up and destroy ephemeral and scalable k8s and ocp clusters for testing
Apache License 2.0
82 stars 119 forks source link

REQ: Add post mortem logs in case cluster-up failed #1329

Open oshoval opened 15 hours ago

oshoval commented 15 hours ago

Is your feature request related to a problem? Please describe: Upon Kubevirtci cluster-up failures we don't have logs, such as kubectl pods, logs, describe etc

example https://prow.ci.kubevirt.io/view/gs/kubevirt-prow/pr-logs/pull/kubevirt_kubevirt/13282/pull-kubevirt-e2e-k8s-1.30-ipv6-sig-network/1859255594477686784

16:41:01: Waiting on istio operator to become ready failed after maximum retries: Istio operator failed to move to Healthy status after max retries
16:41:02: /home/prow/go/src/github.com/kubevirt/kubevirt/kubevirtci/cluster-up/up.sh: line 34: pop_var_context: head of shell_variables not a function context
make: *** [Makefile:163: cluster-up] Error 1

Describe the solution you'd like: Would be good to have collected artifacts on CI in case the cluster-up itself failed.

We need for example the following commands: kubectl get pods -A for each pod, we need please the kubectl describe, and the logs (for each of its containers) Or whatever crucial info that we find beside that on the fly.

Describe alternatives you've considered:

Additional context: Note that we do have some logs collecting as far as i remember, need to adapt / see if something miss. (they are mostly about the docker BM containers, not the pods)

oshoval commented 15 hours ago

/cc @brianmcarey