search

krkn-chaos / cerberus

Guardian of Kubernetes clusters. Tool to monitor clusters health and signal/alert on failures.
Apache License 2.0
92 stars 42 forks source link

Using limit doesn't always get all namespaces #165

Closed paigerube14 closed 2 years ago

paigerube14 commented 2 years ago

Using the limit parameter in list_namespaces cuts off the list of returned namespaces to the limit size and never gets the rest of the namespace on the cluster. Sometimes I have 100's of namespaces with kube-burner tests and the system namespaces that are defined in the config are not able to be found because they are not in the first response

Need to poll if the response from list_namespaces has ret.metadata._continue != None and use that _continue string to poll for the next set of namespaces

ret = cli.list_namespace(pretty=True, limit=request_chunk_size, _continue=continue_string)

(venv3) prubenda@prubenda1-mac cerberus % python3 start_cerberus.py
               _                         
  ___ ___ _ __| |__   ___ _ __ _   _ ___ 
 / __/ _ \ '__| '_ \ / _ \ '__| | | / __|
| (_|  __/ |  | |_) |  __/ |  | |_| \__ \
 \___\___|_|  |_.__/ \___|_|   \__,_|___/

2022-05-05 15:37:39,528 [INFO] Starting ceberus
2022-05-05 15:37:39,537 [INFO] Initializing client to talk to the Kubernetes cluster
2022-05-05 15:37:41,331 [INFO] check namespaces errorThere exists no namespaces matching: {'openshift-monitoring', 'openshift-ovn-kubernetes'}
(venv3) prubenda@prubenda1-mac cerberus % python3 start_cerberus.py
               _                         
  ___ ___ _ __| |__   ___ _ __ _   _ ___ 
 / __/ _ \ '__| '_ \ / _ \ '__| | | / __|
| (_|  __/ |  | |_) |  __/ |  | |_| \__ \
 \___\___|_|  |_.__/ \___|_|   \__,_|___/

2022-05-05 15:39:21,201 [INFO] Starting ceberus
2022-05-05 15:39:21,216 [INFO] Initializing client to talk to the Kubernetes cluster
2022-05-05 15:39:22,115 [INFO] valid namespaces  ['c51f5890-058f-4012-aee0-b0b6b0b4aef0-1', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-10', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-100', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-101', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-102', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-103', 
'c51f5890-058f-4012-aee0-b0b6b0b4aef0-104', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-105', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-106', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-107', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-108', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-109', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-11', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-110', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-111', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-112', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-113', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-114', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-115', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-116', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-117', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-118', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-119', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-12', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-120', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-121', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-122', 
'c51f5890-058f-4012-aee0-b0b6b0b4aef0-123', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-124', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-125', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-126', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-127', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-128', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-129', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-13', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-130', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-131', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-132', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-133', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-134', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-135', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-136', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-137', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-138', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-139', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-14', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-140', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-141', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-142', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-143',
 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-144', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-145', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-146', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-147', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-148', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-149', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-15', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-150', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-151', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-152', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-153', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-154', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-155', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-156', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-157', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-158', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-159', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-16', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-160', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-161', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-162', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-163', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-164', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-165', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-166',
 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-167', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-168', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-169', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-17', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-170', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-171', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-172', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-173', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-174', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-175', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-176', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-177', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-178', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-179', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-18', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-180', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-181', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-182', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-183', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-184', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-185', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-186', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-187', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-188', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-189', 
'c51f5890-058f-4012-aee0-b0b6b0b4aef0-19', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-190', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-191', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-192', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-193', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-194', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-195', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-196', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-197', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-198', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-199', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-2', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-20', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-200', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-21', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-22', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-23', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-24', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-25', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-26', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-27', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-28', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-29', 
'c51f5890-058f-4012-aee0-b0b6b0b4aef0-3', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-30', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-31', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-32', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-33', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-34', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-35', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-36', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-37', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-38', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-39', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-4', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-40', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-41', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-42', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-43', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-44', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-45', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-46', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-47', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-48', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-49', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-5', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-50', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-51', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-52', 
'c51f5890-058f-4012-aee0-b0b6b0b4aef0-53', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-54', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-55', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-56', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-57', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-58', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-59', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-6', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-60', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-61', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-62', 
'c51f5890-058f-4012-aee0-b0b6b0b4aef0-63', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-64', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-65', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-66', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-67', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-68', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-69', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-7', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-70', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-71', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-72', 
'c51f5890-058f-4012-aee0-b0b6b0b4aef0-73', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-74', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-75', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-76', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-77', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-78', 
'c51f5890-058f-4012-aee0-b0b6b0b4aef0-79', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-8', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-80', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-81', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-82', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-83', 
'c51f5890-058f-4012-aee0-b0b6b0b4aef0-84', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-85', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-86', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-87', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-88', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-89', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-9', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-90', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-91', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-92',
 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-93', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-94', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-95', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-96', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-97', 
'c51f5890-058f-4012-aee0-b0b6b0b4aef0-98', 'c51f5890-058f-4012-aee0-b0b6b0b4aef0-99', 'default', 'kube-node-lease', 'kube-public', 'kube-system', 'openshift', 'openshift-apiserver', 
'openshift-apiserver-operator', 'openshift-authentication', 'openshift-authentication-operator', 'openshift-cloud-controller-manager', 'openshift-cloud-controller-manager-operator', 'openshift-cloud-credential-operator', 'openshift-cloud-network-config-controller', 'openshift-cluster-csi-drivers', 'openshift-cluster-machine-approver', 'openshift-cluster-node-tuning-operator', 'openshift-cluster-samples-operator', 'openshift-cluster-storage-operator', 'openshift-cluster-version', 
'openshift-config', 'openshift-config-managed', 'openshift-config-operator', 'openshift-console', 'openshift-console-operator', 'openshift-console-user-settings', 'openshift-controller-manager', 'openshift-controller-manager-operator', 'openshift-dns', 'openshift-dns-operator', 'openshift-etcd', 'openshift-etcd-operator', 'openshift-host-network', 'openshift-image-registry', 'openshift-infra', 'openshift-ingress', 'openshift-ingress-canary', 'openshift-ingress-operator', 
'openshift-insights', 'openshift-kni-infra', 'openshift-kube-apiserver', 'openshift-kube-apiserver-operator', 'openshift-kube-controller-manager', 'openshift-kube-controller-manager-operator', 'openshift-kube-scheduler', 'openshift-kube-scheduler-operator', 'openshift-kube-storage-version-migrator', 'openshift-kube-storage-version-migrator-operator', 'openshift-kubevirt-infra', 'openshift-machine-api', 'openshift-machine-config-operator']
**2022-05-05 15:39:22,126 [INFO] check namespaces errorThere exists no namespaces matching: {'openshift-monitoring', 'openshift-ovn-kubernetes'}**
paigerube14 commented 2 years ago

This issue should be closed with the continue PR that was merged above