search

vmware-tanzu / crash-diagnostics

Crash-Diagnostics (Crashd) is a tool to help investigate, analyze, and troubleshoot unresponsive or crashed Kubernetes clusters.
Other
182 stars 43 forks source link

crashd overwrites the same resources from different API versions #182

Open ncdc opened 4 years ago

ncdc commented 4 years ago

For the following script

kc="{0}/.kube/config".format(os.home)
kube_capture(what="objects", kinds=["clusters"], kube_config=kube_config(path=kc))

With the following cluster:

$ k get clusters -A
NAMESPACE   NAME    PHASE
default     test1   Provisioning

Run this:

$ /crashd run --debug andy.crsh
time="2020-10-02T15:27:07-04:00" level=debug msg="creating working directory /tmp/crashd"
time="2020-10-02T15:27:07-04:00" level=debug msg="kube_capture(what=objects)"
time="2020-10-02T15:27:07-04:00" level=debug msg="Search filters groups:[]; kinds:[clusters]; namespaces:[]; versions:[]; names:[]; labels:[] containers:[]"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching in 28 groups"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group core"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion v1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group apiregistration.k8s.io"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion apiregistration.k8s.io/v1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion apiregistration.k8s.io/v1beta1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group extensions"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion extensions/v1beta1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group apps"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion apps/v1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group events.k8s.io"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion events.k8s.io/v1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion events.k8s.io/v1beta1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group authentication.k8s.io"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion authentication.k8s.io/v1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion authentication.k8s.io/v1beta1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group authorization.k8s.io"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion authorization.k8s.io/v1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion authorization.k8s.io/v1beta1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group autoscaling"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion autoscaling/v1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion autoscaling/v2beta1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion autoscaling/v2beta2"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group batch"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion batch/v1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion batch/v1beta1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group certificates.k8s.io"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion certificates.k8s.io/v1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion certificates.k8s.io/v1beta1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group networking.k8s.io"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion networking.k8s.io/v1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion networking.k8s.io/v1beta1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group policy"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion policy/v1beta1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group rbac.authorization.k8s.io"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion rbac.authorization.k8s.io/v1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion rbac.authorization.k8s.io/v1beta1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group storage.k8s.io"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion storage.k8s.io/v1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion storage.k8s.io/v1beta1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group admissionregistration.k8s.io"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion admissionregistration.k8s.io/v1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion admissionregistration.k8s.io/v1beta1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group apiextensions.k8s.io"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion apiextensions.k8s.io/v1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion apiextensions.k8s.io/v1beta1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group scheduling.k8s.io"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion scheduling.k8s.io/v1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion scheduling.k8s.io/v1beta1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group coordination.k8s.io"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion coordination.k8s.io/v1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion coordination.k8s.io/v1beta1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group node.k8s.io"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion node.k8s.io/v1beta1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group discovery.k8s.io"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion discovery.k8s.io/v1beta1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group acme.cert-manager.io"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion acme.cert-manager.io/v1beta1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion acme.cert-manager.io/v1alpha3"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion acme.cert-manager.io/v1alpha2"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group bootstrap.cluster.x-k8s.io"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion bootstrap.cluster.x-k8s.io/v1alpha3"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion bootstrap.cluster.x-k8s.io/v1alpha2"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group cert-manager.io"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion cert-manager.io/v1beta1"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion cert-manager.io/v1alpha3"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion cert-manager.io/v1alpha2"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in Group cluster.x-k8s.io"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching resources in GroupVersion cluster.x-k8s.io/v1alpha3"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching for GroupResource schema.GroupVersionResource{Group:\"cluster.x-k8s.io\", Version:\"v1alpha3\", Resource:\"clusters\"}"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching for clusters in namespace capi-system [GroupRes: cluster.x-k8s.io/v1alpha3, Resource=clusters]"
time="2020-10-02T15:27:07-04:00" level=debug msg="Found 0 clusters in namespace [capi-system]"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching for clusters in namespace capi-webhook-system [GroupRes: cluster.x-k8s.io/v1alpha3, Resource=clusters]"
time="2020-10-02T15:27:07-04:00" level=debug msg="Found 0 clusters in namespace [capi-webhook-system]"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching for clusters in namespace cert-manager [GroupRes: cluster.x-k8s.io/v1alpha3, Resource=clusters]"
time="2020-10-02T15:27:07-04:00" level=debug msg="Found 0 clusters in namespace [cert-manager]"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching for clusters in namespace default [GroupRes: cluster.x-k8s.io/v1alpha3, Resource=clusters]"
time="2020-10-02T15:27:07-04:00" level=debug msg="Found 1 clusters in namespace [default]"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching for clusters in namespace kube-node-lease [GroupRes: cluster.x-k8s.io/v1alpha3, Resource=clusters]"
time="2020-10-02T15:27:07-04:00" level=debug msg="Found 0 clusters in namespace [kube-node-lease]"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching for clusters in namespace kube-public [GroupRes: cluster.x-k8s.io/v1alpha3, Resource=clusters]"
time="2020-10-02T15:27:07-04:00" level=debug msg="Found 0 clusters in namespace [kube-public]"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching for clusters in namespace kube-system [GroupRes: cluster.x-k8s.io/v1alpha3, Resource=clusters]"
time="2020-10-02T15:27:07-04:00" level=debug msg="Found 0 clusters in namespace [kube-system]"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching for clusters in namespace local-path-storage [GroupRes: cluster.x-k8s.io/v1alpha3, Resource=clusters]"
time="2020-10-02T15:27:07-04:00" level=debug msg="Found 0 clusters in namespace [local-path-storage]"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching for GroupResource schema.GroupVersionResource{Group:\"cluster.x-k8s.io\", Version:\"v1alpha3\", Resource:\"clusters/status\"}"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching for clusters/status in namespace capi-system [GroupRes: cluster.x-k8s.io/v1alpha3, Resource=clusters/status]"
time="2020-10-02T15:27:07-04:00" level=debug msg="WARN: K8s.Search failed to get clusters/status in capi-system [GroupRes: cluster.x-k8s.io/v1alpha3][labels: ]: clusters.cluster.x-k8s.io \"status\" not found"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching for clusters/status in namespace capi-webhook-system [GroupRes: cluster.x-k8s.io/v1alpha3, Resource=clusters/status]"
time="2020-10-02T15:27:07-04:00" level=debug msg="WARN: K8s.Search failed to get clusters/status in capi-webhook-system [GroupRes: cluster.x-k8s.io/v1alpha3][labels: ]: clusters.cluster.x-k8s.io \"status\" not found"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching for clusters/status in namespace cert-manager [GroupRes: cluster.x-k8s.io/v1alpha3, Resource=clusters/status]"
time="2020-10-02T15:27:07-04:00" level=debug msg="WARN: K8s.Search failed to get clusters/status in cert-manager [GroupRes: cluster.x-k8s.io/v1alpha3][labels: ]: clusters.cluster.x-k8s.io \"status\" not found"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching for clusters/status in namespace default [GroupRes: cluster.x-k8s.io/v1alpha3, Resource=clusters/status]"
time="2020-10-02T15:27:07-04:00" level=debug msg="WARN: K8s.Search failed to get clusters/status in default [GroupRes: cluster.x-k8s.io/v1alpha3][labels: ]: clusters.cluster.x-k8s.io \"status\" not found"
time="2020-10-02T15:27:07-04:00" level=debug msg="Searching for clusters/status in namespace kube-node-lease [GroupRes: cluster.x-k8s.io/v1alpha3, Resource=clusters/status]"
time="2020-10-02T15:27:08-04:00" level=debug msg="WARN: K8s.Search failed to get clusters/status in kube-node-lease [GroupRes: cluster.x-k8s.io/v1alpha3][labels: ]: clusters.cluster.x-k8s.io \"status\" not found"
time="2020-10-02T15:27:08-04:00" level=debug msg="Searching for clusters/status in namespace kube-public [GroupRes: cluster.x-k8s.io/v1alpha3, Resource=clusters/status]"
time="2020-10-02T15:27:08-04:00" level=debug msg="WARN: K8s.Search failed to get clusters/status in kube-public [GroupRes: cluster.x-k8s.io/v1alpha3][labels: ]: clusters.cluster.x-k8s.io \"status\" not found"
time="2020-10-02T15:27:08-04:00" level=debug msg="Searching for clusters/status in namespace kube-system [GroupRes: cluster.x-k8s.io/v1alpha3, Resource=clusters/status]"
time="2020-10-02T15:27:08-04:00" level=debug msg="WARN: K8s.Search failed to get clusters/status in kube-system [GroupRes: cluster.x-k8s.io/v1alpha3][labels: ]: clusters.cluster.x-k8s.io \"status\" not found"
time="2020-10-02T15:27:08-04:00" level=debug msg="Searching for clusters/status in namespace local-path-storage [GroupRes: cluster.x-k8s.io/v1alpha3, Resource=clusters/status]"
time="2020-10-02T15:27:08-04:00" level=debug msg="WARN: K8s.Search failed to get clusters/status in local-path-storage [GroupRes: cluster.x-k8s.io/v1alpha3][labels: ]: clusters.cluster.x-k8s.io \"status\" not found"
time="2020-10-02T15:27:08-04:00" level=debug msg="Searching resources in GroupVersion cluster.x-k8s.io/v1alpha2"
time="2020-10-02T15:27:08-04:00" level=debug msg="Searching for GroupResource schema.GroupVersionResource{Group:\"cluster.x-k8s.io\", Version:\"v1alpha2\", Resource:\"clusters\"}"
time="2020-10-02T15:27:08-04:00" level=debug msg="Searching for clusters in namespace capi-system [GroupRes: cluster.x-k8s.io/v1alpha2, Resource=clusters]"
time="2020-10-02T15:27:08-04:00" level=debug msg="Found 0 clusters in namespace [capi-system]"
time="2020-10-02T15:27:08-04:00" level=debug msg="Searching for clusters in namespace capi-webhook-system [GroupRes: cluster.x-k8s.io/v1alpha2, Resource=clusters]"
time="2020-10-02T15:27:09-04:00" level=debug msg="Found 0 clusters in namespace [capi-webhook-system]"
time="2020-10-02T15:27:09-04:00" level=debug msg="Searching for clusters in namespace cert-manager [GroupRes: cluster.x-k8s.io/v1alpha2, Resource=clusters]"
time="2020-10-02T15:27:09-04:00" level=debug msg="Found 0 clusters in namespace [cert-manager]"
time="2020-10-02T15:27:09-04:00" level=debug msg="Searching for clusters in namespace default [GroupRes: cluster.x-k8s.io/v1alpha2, Resource=clusters]"
time="2020-10-02T15:27:09-04:00" level=debug msg="Found 1 clusters in namespace [default]"
time="2020-10-02T15:27:09-04:00" level=debug msg="Searching for clusters in namespace kube-node-lease [GroupRes: cluster.x-k8s.io/v1alpha2, Resource=clusters]"
time="2020-10-02T15:27:09-04:00" level=debug msg="Found 0 clusters in namespace [kube-node-lease]"
time="2020-10-02T15:27:09-04:00" level=debug msg="Searching for clusters in namespace kube-public [GroupRes: cluster.x-k8s.io/v1alpha2, Resource=clusters]"
time="2020-10-02T15:27:09-04:00" level=debug msg="Found 0 clusters in namespace [kube-public]"
time="2020-10-02T15:27:09-04:00" level=debug msg="Searching for clusters in namespace kube-system [GroupRes: cluster.x-k8s.io/v1alpha2, Resource=clusters]"
time="2020-10-02T15:27:10-04:00" level=debug msg="Found 0 clusters in namespace [kube-system]"
time="2020-10-02T15:27:10-04:00" level=debug msg="Searching for clusters in namespace local-path-storage [GroupRes: cluster.x-k8s.io/v1alpha2, Resource=clusters]"
time="2020-10-02T15:27:10-04:00" level=debug msg="Found 0 clusters in namespace [local-path-storage]"
time="2020-10-02T15:27:10-04:00" level=debug msg="Searching for GroupResource schema.GroupVersionResource{Group:\"cluster.x-k8s.io\", Version:\"v1alpha2\", Resource:\"clusters/status\"}"
time="2020-10-02T15:27:10-04:00" level=debug msg="Searching for clusters/status in namespace capi-system [GroupRes: cluster.x-k8s.io/v1alpha2, Resource=clusters/status]"
time="2020-10-02T15:27:10-04:00" level=debug msg="WARN: K8s.Search failed to get clusters/status in capi-system [GroupRes: cluster.x-k8s.io/v1alpha2][labels: ]: clusters.cluster.x-k8s.io \"status\" not found"
time="2020-10-02T15:27:10-04:00" level=debug msg="Searching for clusters/status in namespace capi-webhook-system [GroupRes: cluster.x-k8s.io/v1alpha2, Resource=clusters/status]"
time="2020-10-02T15:27:10-04:00" level=debug msg="WARN: K8s.Search failed to get clusters/status in capi-webhook-system [GroupRes: cluster.x-k8s.io/v1alpha2][labels: ]: clusters.cluster.x-k8s.io \"status\" not found"
time="2020-10-02T15:27:10-04:00" level=debug msg="Searching for clusters/status in namespace cert-manager [GroupRes: cluster.x-k8s.io/v1alpha2, Resource=clusters/status]"
time="2020-10-02T15:27:10-04:00" level=debug msg="WARN: K8s.Search failed to get clusters/status in cert-manager [GroupRes: cluster.x-k8s.io/v1alpha2][labels: ]: clusters.cluster.x-k8s.io \"status\" not found"
time="2020-10-02T15:27:10-04:00" level=debug msg="Searching for clusters/status in namespace default [GroupRes: cluster.x-k8s.io/v1alpha2, Resource=clusters/status]"
time="2020-10-02T15:27:11-04:00" level=debug msg="WARN: K8s.Search failed to get clusters/status in default [GroupRes: cluster.x-k8s.io/v1alpha2][labels: ]: clusters.cluster.x-k8s.io \"status\" not found"
time="2020-10-02T15:27:11-04:00" level=debug msg="Searching for clusters/status in namespace kube-node-lease [GroupRes: cluster.x-k8s.io/v1alpha2, Resource=clusters/status]"
time="2020-10-02T15:27:11-04:00" level=debug msg="WARN: K8s.Search failed to get clusters/status in kube-node-lease [GroupRes: cluster.x-k8s.io/v1alpha2][labels: ]: clusters.cluster.x-k8s.io \"status\" not found"
time="2020-10-02T15:27:11-04:00" level=debug msg="Searching for clusters/status in namespace kube-public [GroupRes: cluster.x-k8s.io/v1alpha2, Resource=clusters/status]"
time="2020-10-02T15:27:11-04:00" level=debug msg="WARN: K8s.Search failed to get clusters/status in kube-public [GroupRes: cluster.x-k8s.io/v1alpha2][labels: ]: clusters.cluster.x-k8s.io \"status\" not found"
time="2020-10-02T15:27:11-04:00" level=debug msg="Searching for clusters/status in namespace kube-system [GroupRes: cluster.x-k8s.io/v1alpha2, Resource=clusters/status]"
time="2020-10-02T15:27:11-04:00" level=debug msg="WARN: K8s.Search failed to get clusters/status in kube-system [GroupRes: cluster.x-k8s.io/v1alpha2][labels: ]: clusters.cluster.x-k8s.io \"status\" not found"
time="2020-10-02T15:27:11-04:00" level=debug msg="Searching for clusters/status in namespace local-path-storage [GroupRes: cluster.x-k8s.io/v1alpha2, Resource=clusters/status]"
time="2020-10-02T15:27:11-04:00" level=debug msg="WARN: K8s.Search failed to get clusters/status in local-path-storage [GroupRes: cluster.x-k8s.io/v1alpha2][labels: ]: clusters.cluster.x-k8s.io \"status\" not found"
time="2020-10-02T15:27:11-04:00" level=debug msg="Searching resources in Group infrastructure.cluster.x-k8s.io"
time="2020-10-02T15:27:11-04:00" level=debug msg="Searching resources in GroupVersion infrastructure.cluster.x-k8s.io/v1alpha3"
time="2020-10-02T15:27:11-04:00" level=debug msg="Searching resources in GroupVersion infrastructure.cluster.x-k8s.io/v1alpha2"
time="2020-10-02T15:27:11-04:00" level=debug msg="Searching resources in Group addons.cluster.x-k8s.io"
time="2020-10-02T15:27:11-04:00" level=debug msg="Searching resources in GroupVersion addons.cluster.x-k8s.io/v1alpha3"
time="2020-10-02T15:27:11-04:00" level=debug msg="Searching resources in Group controlplane.cluster.x-k8s.io"
time="2020-10-02T15:27:11-04:00" level=debug msg="Searching resources in GroupVersion controlplane.cluster.x-k8s.io/v1alpha3"
time="2020-10-02T15:27:11-04:00" level=debug msg="Searching resources in Group exp.cluster.x-k8s.io"
time="2020-10-02T15:27:11-04:00" level=debug msg="Searching resources in GroupVersion exp.cluster.x-k8s.io/v1alpha3"
time="2020-10-02T15:27:11-04:00" level=debug msg="kube_capture(): saving clusters search results to: /tmp/crashd/kubecapture/capi-system/clusters.json"
time="2020-10-02T15:27:11-04:00" level=debug msg="kube_capture(): saving clusters search results to: /tmp/crashd/kubecapture/capi-webhook-system/clusters.json"
time="2020-10-02T15:27:11-04:00" level=debug msg="kube_capture(): saving clusters search results to: /tmp/crashd/kubecapture/cert-manager/clusters.json"
time="2020-10-02T15:27:11-04:00" level=debug msg="kube_capture(): saving clusters search results to: /tmp/crashd/kubecapture/default/clusters.json"
time="2020-10-02T15:27:11-04:00" level=debug msg="kube_capture(): saving clusters search results to: /tmp/crashd/kubecapture/kube-node-lease/clusters.json"
time="2020-10-02T15:27:11-04:00" level=debug msg="kube_capture(): saving clusters search results to: /tmp/crashd/kubecapture/kube-public/clusters.json"
time="2020-10-02T15:27:11-04:00" level=debug msg="kube_capture(): saving clusters search results to: /tmp/crashd/kubecapture/kube-system/clusters.json"
time="2020-10-02T15:27:11-04:00" level=debug msg="kube_capture(): saving clusters search results to: /tmp/crashd/kubecapture/local-path-storage/clusters.json"
time="2020-10-02T15:27:11-04:00" level=debug msg="kube_capture(): saving clusters search results to: /tmp/crashd/kubecapture/capi-system/clusters.json"
time="2020-10-02T15:27:11-04:00" level=debug msg="kube_capture(): saving clusters search results to: /tmp/crashd/kubecapture/capi-webhook-system/clusters.json"
time="2020-10-02T15:27:11-04:00" level=debug msg="kube_capture(): saving clusters search results to: /tmp/crashd/kubecapture/cert-manager/clusters.json"
time="2020-10-02T15:27:11-04:00" level=debug msg="kube_capture(): saving clusters search results to: /tmp/crashd/kubecapture/default/clusters.json"
time="2020-10-02T15:27:11-04:00" level=debug msg="kube_capture(): saving clusters search results to: /tmp/crashd/kubecapture/kube-node-lease/clusters.json"
time="2020-10-02T15:27:11-04:00" level=debug msg="kube_capture(): saving clusters search results to: /tmp/crashd/kubecapture/kube-public/clusters.json"
time="2020-10-02T15:27:11-04:00" level=debug msg="kube_capture(): saving clusters search results to: /tmp/crashd/kubecapture/kube-system/clusters.json"
time="2020-10-02T15:27:11-04:00" level=debug msg="kube_capture(): saving clusters search results to: /tmp/crashd/kubecapture/local-path-storage/clusters.json"

Look at clusters.json:

{
    "apiVersion": "cluster.x-k8s.io/v1alpha2",
    "items": [
        {
            "apiVersion": "cluster.x-k8s.io/v1alpha2",
            "kind": "Cluster",
            "metadata": {
                "annotations": {
                    "cluster.x-k8s.io/conversion-data": "{\"apiVersion\":\"cluster.x-k8s.io/v1alpha3\",\"kind\":\"Cluster\",\"spec\":{\"clusterNetwork\":{\"pods\":{\"cidrBlocks\":[\"192.168.0.0/16\"]}},\"controlPlaneEndpoint\":{\"host\":\"\",\"port\":0},\"infrastructureRef\":{\"apiVersion\":\"infrastructure.cluster.x-k8s.io/v1alpha3\",\"kind\":\"AWSCluster\",\"name\":\"test1\",\"namespace\":\"default\"}},\"status\":{\"conditions\":[{\"lastTransitionTime\":\"2020-10-02T19:23:23Z\",\"reason\":\"WaitingForInfrastructure\",\"severity\":\"Info\",\"status\":\"False\",\"type\":\"Ready\"},{\"lastTransitionTime\":\"2020-10-02T19:23:23Z\",\"reason\":\"WaitingForInfrastructure\",\"severity\":\"Info\",\"status\":\"False\",\"type\":\"InfrastructureReady\"}],\"controlPlaneInitialized\":false,\"infrastructureReady\":false,\"observedGeneration\":2,\"phase\":\"Provisioning\"}}",
                    "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"cluster.x-k8s.io/v1alpha3\",\"kind\":\"Cluster\",\"metadata\":{\"annotations\":{},\"name\":\"test1\",\"namespace\":\"default\"},\"spec\":{\"clusterNetwork\":{\"pods\":{\"cidrBlocks\":[\"192.168.0.0/16\"]}},\"infrastructureRef\":{\"apiVersion\":\"infrastructure.cluster.x-k8s.io/v1alpha3\",\"kind\":\"AWSCluster\",\"name\":\"test1\"}}}\n"
                },
                "creationTimestamp": "2020-10-02T19:19:59Z",
                "finalizers": [
                    "cluster.cluster.x-k8s.io"
                ],
                "generation": 2,
                "managedFields": [
                    {
                        "apiVersion": "cluster.x-k8s.io/v1alpha3",
                        "fieldsType": "FieldsV1",
                        "fieldsV1": {
                            "f:metadata": {
                                "f:annotations": {
                                    ".": {},
                                    "f:kubectl.kubernetes.io/last-applied-configuration": {}
                                }
                            },
                            "f:spec": {
                                ".": {},
                                "f:clusterNetwork": {
                                    ".": {},
                                    "f:pods": {
                                        ".": {},
                                        "f:cidrBlocks": {}
                                    }
                                },
                                "f:infrastructureRef": {
                                    ".": {},
                                    "f:apiVersion": {},
                                    "f:kind": {},
                                    "f:name": {}
                                }
                            }
                        },
                        "manager": "kubectl-client-side-apply",
                        "operation": "Update",
                        "time": "2020-10-02T19:19:59Z"
                    },
                    {
                        "apiVersion": "cluster.x-k8s.io/v1alpha3",
                        "fieldsType": "FieldsV1",
                        "fieldsV1": {
                            "f:metadata": {
                                "f:finalizers": {
                                    ".": {},
                                    "v:\"cluster.cluster.x-k8s.io\"": {}
                                }
                            },
                            "f:status": {
                                ".": {},
                                "f:conditions": {},
                                "f:observedGeneration": {},
                                "f:phase": {}
                            }
                        },
                        "manager": "manager",
                        "operation": "Update",
                        "time": "2020-10-02T19:23:23Z"
                    }
                ],
                "name": "test1",
                "namespace": "default",
                "resourceVersion": "2770",
                "selfLink": "/apis/cluster.x-k8s.io/v1alpha2/namespaces/default/clusters/test1",
                "uid": "677518fa-79c5-4742-862a-eadf12b3ff37"
            },
            "spec": {
                "clusterNetwork": {
                    "pods": {
                        "cidrBlocks": [
                            "192.168.0.0/16"
                        ]
                    }
                },
                "infrastructureRef": {
                    "apiVersion": "infrastructure.cluster.x-k8s.io/v1alpha3",
                    "kind": "AWSCluster",
                    "name": "test1",
                    "namespace": "default"
                }
            },
            "status": {
                "controlPlaneInitialized": false,
                "infrastructureReady": false,
                "phase": "Provisioning"
            }
        }
    ],
    "kind": "ClusterList",
    "metadata": {
        "continue": "",
        "resourceVersion": "3733",
        "selfLink": "/apis/cluster.x-k8s.io/v1alpha2/namespaces/default/clusters"
    }
}

Note that it's the v1alpha2 API version. It appears that crashd is finding both the v1alpha2 and v1alpha3 API versions of my Cluster, processing them both, and writing the results to the same file, with the last one in winning:

DEBU[0004] kube_capture(): saving clusters search results to: /tmp/crashd/kubecapture/default/clusters.json
[snip]
DEBU[0004] kube_capture(): saving clusters search results to: /tmp/crashd/kubecapture/default/clusters.json

If you use a RESTMapper and assume that when capturing objects, you always concatenate groups and kinds (which should really be resources) together, you can do something like this to get the server preferred version without having to loop through discovery data:

type Client struct {
  // other fields
  mapper meta.RESTMapper
}

func New(kubeconfig string) (*Client, error) {
  // other init
  res, err := restmapper.GetAPIGroupResources(disco)
  if err != nil {
    return nil, err
  }
  mapper := restmapper.NewDiscoveryRESTMapper(res)
  // store mapper in struct
}

func (k8sc *Client) _search(groups, kinds, namespaces, versions, names, labels, containers string) ([]SearchResult, error) {
    // prep code

    // (please ignore the quick hacks with variable names)
    groupsA := strings.Split(groups, ",")
    resourcesA := strings.Split(kinds, ",")
    for _, group := range groupsA {
        group = strings.TrimSpace(group)
        for _, resource := range resourcesA {
            resource = strings.TrimSpace(resource)
            gvk, err := k8sc.mapper.KindFor(schema.GroupVersionResource{Group: group, Resource: resource})
            if err != nil {
                // probably should collect errors, continue, and return an aggregate at the end
                return nil, err
            }
            mapping, err := k8sc.mapper.RESTMapping(gvk.GroupKind())
            if err != nil {
                // probably should collect errors, continue, and return an aggregate at the end
                return nil, err
            }

            // retrieve API objects based on GroupVersionResource and
            // filter by namespaces and names
            listOptions := metav1.ListOptions{
                LabelSelector: labels,
            }

            gvr := mapping.Resource

            // gather found resources
            var results []SearchResult
            if mapping.Scope.Name() == meta.RESTScopeNameNamespace {
                for _, ns := range splitParamList(namespaces) {
                    logrus.Debugf("Searching for %s in namespace %s [GroupRes: %v]", resource, ns, gvr)
                    list, err := k8sc.Client.Resource(gvr).Namespace(ns).List(listOptions)
                    if err != nil {
                        logrus.Debugf(
                            "WARN: K8s.Search failed to get %s in %s [GroupRes: %s][labels: %v]: %s",
                            resource, ns, gvr.GroupVersion(), listOptions.LabelSelector, err,
                        )
                        continue
                    }
                    logrus.Debugf("Found %d %s in namespace [%s]", len(list.Items), resource, ns)
                    result := SearchResult{
                        ListKind:             list.GetKind(),
                        ResourceName:         resource,
                        ResourceKind:         mapping.GroupVersionKind.GroupKind().Kind,
                        Namespaced:           true,
                        Namespace:            ns,
                        GroupVersionResource: gvr,
                        List:                 list,
                    }
                    results = append(results, result)
                }
            } else {
                logrus.Debugf("Searching for resource %s (non-namespaced)", resource)
                list, err := k8sc.Client.Resource(gvr).List(listOptions)
                if err != nil {
                    logrus.Debugf(
                        "WARN: K8s.Search failed to get %s: [GroupRes: %s] [labels: %v]: %s",
                        resource, gvr.GroupVersion(), listOptions.LabelSelector, err,
                    )
                    continue
                }
                logrus.Debugf("Found %d %s (non-namespaced)", len(list.Items), resource)
                result := SearchResult{
                    ListKind:             list.GetKind(),
                    ResourceName:         resource,
                    ResourceKind:         mapping.GroupVersionKind.GroupKind().Kind,
                    Namespaced:           false,
                    GroupVersionResource: gvr,
                    List:                 list,
                }
                results = append(results, result)
            }

            // apply name filters
            for _, result := range results {
                filteredResult := result
                if len(containers) > 0 && result.ListKind == "PodList" {
                    filteredResult = filterPodsByContainers(result, containers)
                    logrus.Debugf("Found %d %s with container filter [%s]", len(filteredResult.List.Items), filteredResult.ResourceName, containers)
                }
                if len(names) > 0 {
                    filteredResult = filterByNames(result, names)
                    logrus.Debugf("Found %d %s with name filter [%s]", len(filteredResult.List.Items), filteredResult.ResourceName, names)
                }
                finalResults = append(finalResults, filteredResult)
            }
        }
    }

    return finalResults, nil
}
ncdc commented 4 years ago

With my hacky changes from above (and limiting to just the default namespace to cut down on noise):

DEBU[0000] creating working directory /tmp/crashd
DEBU[0000] kube_capture(what=objects)
DEBU[0000] Search filters groups:[]; kinds:[clusters]; namespaces:[default]; versions:[]; names:[]; labels:[] containers:[]
DEBU[0000] Searching for clusters in namespace default [GroupRes: cluster.x-k8s.io/v1alpha3, Resource=clusters]
DEBU[0000] Found 1 clusters in namespace [default]
DEBU[0000] kube_capture(): saving clusters search results to: /tmp/crashd/kubecapture/default/clusters.json
vladimirvivien commented 4 years ago

@ncdc thanks for the superb detailed issue.

mcwumbly commented 3 years ago

Just chiming in to note taht @gwang550 and I just ran into this problem when exploring how to provide guidance on how to troubleshoot issues bootstrapping a management cluster due to connectivity issues to the infrastructure APIs.

In some scenarios, we were looking at the vspherecluster objects on the cluster and seeing these conditions:

            "status": {
                "conditions": [
                    {
                        "lastTransitionTime": "2021-07-06T20:12:44Z",
                        "status": "True",
                        "type": "Ready"
                    },
                    {
                        "lastTransitionTime": "2021-07-06T20:12:44Z",
                        "status": "True",
                        "type": "LoadBalancerAvailable"
                    },
                    {
                        "lastTransitionTime": "2021-07-06T20:12:44Z",
                        "message": "Secret \"tkg-mgmt-vc\" not found",
                        "reason": "VCenterUnreachable",
                        "severity": "Error",
                        "status": "False",
                        "type": "VCenterAvailable"
                    }
                ],
                "ready": true
            }

But when we grabbed the resource with crashd, we saw this instead:

            "status": {
                "apiEndpoints": [
                    {
                        "host": "2620:124:6020:c00a:0:a:cccc:eebb",
                        "port": 6443
                    }
                ],
                "ready": true
            }

We were confused about why the crashd output was missing the conditions, but in looking at the output more closely, we saw that the apiVersion of the resource was older (v1alpha2 instead of v1alpha3).

vladimirvivien commented 3 years ago

@mcwumbly will move this higher in priority. Thanks for pointing this out.

vladimirvivien commented 3 years ago

I @mcwumbly #207 implemented some changes that should fix your issue.