search

fabric8io / fluent-plugin-kubernetes_metadata_filter

Enrich your fluentd events with Kubernetes metadata
Apache License 2.0
351 stars 168 forks source link

kubernetes_metadata_filter gets stuck trying to read metadata for a deleted kubernetes pod #292

Closed alex-vmw closed 1 year ago

alex-vmw commented 3 years ago

Describe the bug kubernetes_metadata_filter gets stuck in a loop trying to read metadata for a deleted kubernetes pod and stops sending logs.

To Reproduce Have fluentd collect metadata of kubernetes pods while constantly creating and deleting kubernetes pods.

Expected behavior kubernetes_metadata_filter should NOT get gets stuck in a loop trying to read metadata for a deleted kubernetes pod.

Your Environment Fluentd: 1.12.3. kubernetes_metadata_filter: 2.6.0 Operating system: Ubuntu 20.04.1 LTS Kernel version: 5.4.0-62-generic

Fluentd Configuration

<ROOT>
  <system>
    log_level info
    rpc_endpoint "127.0.0.1:24444"
  </system>
  <source>
    @type systemd
    @id in_systemd_docker
    path "/var/log/journal"
    tag "systemd.unit"
    read_from_head false
    <storage>
      @type "local"
      persistent true
      path "/var/log/-kube-fluentd-operator--fluentd-journald-cursor.json"
    </storage>
    <entry>
      field_map {"_SYSTEMD_UNIT":"unit","MESSAGE":"log","_PID":["pid"],"_PRIORITY":"priority","_COMM":"cmd","_HOSTNAME":"hostname"}
      field_map_strict true
      fields_lowercase true
    </entry>
  </source>
  <match systemd.unit>
    @type rewrite_tag_filter
    <rule>
      key "unit"
      pattern ^(.+)$
      tag "systemd.$1"
    </rule>
  </match>
  <filter systemd.kubelet.service>
    @type parser
    reserve_data true
    key_name "log"
    emit_invalid_record_to_error false
    <parse>
      @type "kubernetes"
      time_format "%m%d %H:%M:%S.%N"
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
    </parse>
  </filter>
  <source>
    @type tail
    @id in_tail_container_logs
    path "/var/log/containers/*.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-containers.log.pos"
    tag "kubernetes.*"
    read_from_head true
    <parse>
      @type "multiline"
      format1 /^(?<partials>([^\n]+ (stdout|stderr) P [^\n]+\n)*)/
      format2 /(?<time>[^\n]+) (?<stream>stdout|stderr) F (?<log>[^\n]*)/
      format3 /|(?<json>{.*})/
      time_format "%Y-%m-%dT%H:%M:%S.%N%:z"
      unmatched_lines
    </parse>
  </source>
  <filter kubernetes.**>
    @type record_transformer
    @id filter_crio_container_logs
    enable_ruby true
    remove_keys partials
    <record>
      log ${record["partials"]&.gsub(/.+ (stdout|stderr) P (.+)\n/, '\\2')}${record["log"]}
    </record>
  </filter>
  <filter kubernetes.**>
    @type parser
    @id filter_docker_container_logs
    key_name "json"
    remove_key_name_field true
    reserve_data true
    emit_invalid_record_to_error false
    <parse>
      @type "json"
      time_format "%Y-%m-%dT%H:%M:%S.%NZ"
      time_type string
    </parse>
  </filter>
  <source>
    @type tail
    @id in_tail_startupscript
    path "/var/log/startupscript.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-startupscript.log.pos"
    tag "startupscript"
    <parse>
      @type "syslog"
      unmatched_lines
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_docker
    path "/var/log/docker.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-docker.log.pos"
    tag "docker"
    <parse>
      @type "regexp"
      expression /^time="(?<time>[^)]*)" level=(?<severity>[^ ]*) msg="(?<message>[^"]*)"( err="(?<error>[^"]*)")?( statusCode=($<status_code>\d+))?/
      unmatched_lines
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_etcd
    path "/var/log/etcd.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-etcd.log.pos"
    tag "k8s.etcd"
    <parse>
      @type "none"
      unmatched_lines
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kubelet
    multiline_flush_interval 5s
    path "/var/log/kubelet.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-kubelet.log.pos"
    tag "k8s.kubelet"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kube_proxy
    multiline_flush_interval 5s
    path "/var/log/kube-proxy.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-kube-proxy.log.pos"
    tag "k8s.kube-proxy"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kube_apiserver
    multiline_flush_interval 5s
    path "/var/log/kube-apiserver.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-kube-apiserver.log.pos"
    tag "k8s.kube-apiserver"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kube_controller_manager
    multiline_flush_interval 5s
    path "/var/log/kube-controller-manager.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-kube-controller-manager.log.pos"
    tag "k8s.kube-controller-manager"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kube_scheduler
    multiline_flush_interval 5s
    path "/var/log/kube-scheduler.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-kube-scheduler.log.pos"
    tag "k8s.kube-scheduler"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_rescheduler
    multiline_flush_interval 5s
    path "/var/log/rescheduler.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-rescheduler.log.pos"
    tag "k8s.rescheduler"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_glbc
    multiline_flush_interval 5s
    path "/var/log/glbc.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-glbc.log.pos"
    tag "k8s.glbc"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_cluster_autoscaler
    multiline_flush_interval 5s
    path "/var/log/cluster-autoscaler.log"
    pos_file "/var/log/-kube-fluentd-operator--fluentd-cluster-autoscaler.log.pos"
    tag "k8s.cluster-autoscaler"
    <parse>
      @type "kubernetes"
      unmatched_lines
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
      time_format "%m%d %H:%M:%S.%N"
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_kube_apiserver_audit
    multiline_flush_interval 5s
    path "/var/log/kubernetes/kube-apiserver-audit.log"
    pos_file "/var/log/-kube-fluentd-operator--kube-apiserver-audit.log.pos"
    tag "k8s.kube-apiserver-audit"
    <parse>
      @type "multiline"
      format_firstline "/^\\S+\\s+AUDIT:/"
      format1 /^(?<time>\S+) AUDIT:(?: (?:id="(?<id>(?:[^"\\]|\\.)*)"|ip="(?<ip>(?:[^"\\]|\\.)*)"|method="(?<method>(?:[^"\\]|\\.)*)"|user="(?<user>(?:[^"\\]|\\.)*)"|groups="(?<groups>(?:[^"\\]|\\.)*)"|as="(?<as>(?:[^"\\]|\\.)*)"|asgroups="(?<asgroups>(?:[^"\\]|\\.)*)"|namespace="(?<namespace>(?:[^"\\]|\\.)*)"|uri="(?<uri>(?:[^"\\]|\\.)*)"|response="(?<response>(?:[^"\\]|\\.)*)"|\w+="(?:[^"\\]|\\.)*"))*/
      time_format "%Y-%m-%dT%T.%L%Z"
      unmatched_lines
    </parse>
  </source>
  <filter kubernetes.**>
    @type kubernetes_metadata
    @id filter_kube_metadata
  </filter>
  <filter kubernetes.**>
    @type kubernetes_metadata
  </filter>
  <filter kubernetes.**>
    @type record_transformer
    enable_ruby true
    <record>
      kubernetes_namespace_container_name ${record["kubernetes"]["namespace_name"]}.${record["kubernetes"]["pod_name"]}.${record["kubernetes"]["container_name"]}
      container_info ${record["docker"]["container_id"]}-${record["stream"]}
    </record>
  </filter>
  <match kubernetes.**>
    @type rewrite_tag_filter
    <rule>
      key "kubernetes_namespace_container_name"
      pattern ^(.+)$
      tag "kube.$1"
    </rule>
  </match>
  <filter kube.*.*.*>
    @type record_modifier
    remove_keys "dummy_"
    <record>
      dummy_ ${record["kubernetes"]&.delete("master_url"); record["kubernetes"]&.delete("namespace_id"); if record["kubernetes"]&.has_key?("labels"); record["kubernetes"]["labels"].delete("pod-template-generation"); record["kubernetes"]["labels"].delete("controller-revision-hash");  record["kubernetes"]["labels"].delete("pod-template-hash"); end; nil}
    </record>
  </filter>
  <filter kube.*.*.*>
    @type record_transformer
    remove_keys kubernetes_namespace_container_name
  </filter>
  <filter kube.kube-system.**>
    @type parser
    reserve_data true
    key_name "log"
    emit_invalid_record_to_error false
    <parse>
      @type "kubernetes"
      time_format "%m%d %H:%M:%S.%N"
      expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
    </parse>
  </filter>
  <source>
    @type prometheus
  </source>
  <source>
    @type prometheus_monitor
  </source>
  <source>
    @type prometheus_output_monitor
  </source>
  <match systemd.** kube.kube-system.**>
    @type copy
    <store>
      @id X
      @type "vmware_loginsight"
      host "X.X.X.X"
      include_tag_key true
      log_text_keys ["log","msg","message"]
      port 9000
      scheme "http"
      serializer "json"
      ssl_verify false
      tag_key "tag"
    </store>
  </match>
  <match **>
    @type null
  </match>
</ROOT>

Error Log

  1. Notice that this enourmous amount of errors is covering just 2 seconds of time!
  2. The below errors just basically continues forever in a loop.

2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:55 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api 2021-06-04 19:16:56 +0000 [debug]: #0 Exception 'HTTP status code 404, pods "cat-api-5c5c564d99-tq6ng" not found for GET https://172.21.0.1/api/v1/namespaces/ara-cat-beta/pods/cat-api-5c5c564d99-tq6ng' encountered fetching pod metadata from Kubernetes API v1 endpoint https://172.21.0.1:443/api

jcantrill commented 3 years ago

The breakdown in the logic looks to be here https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter/blob/master/lib/fluent/plugin/kubernetes_metadata_cache_strategy.rb#L45-L56 where we find NS meta and not podmeta. It's a deleted pod so we will NEVER find meta so this should likely update the batch_cache_miss to short circuit the logic.

When you say "stuck" do you mean it takes time to process entries or it actually blocks entries all-to-gether?

alex-vmw commented 3 years ago

When you say "stuck" do you mean it takes time to process entries or it actually blocks entries all-to-gether?

When I say "stuck", I mean all we see in the fluentd logs for hours and hours is error message posted, so we assumed nothing else was being processed.

P.S. Can you also explain why it would be doing 100+ API calls per second?

jcantrill commented 3 years ago

When I say "stuck", I mean all we see in the fluentd logs for hours and hours is error message posted, so we assumed nothing else was being processed.

Note the error message is "debug" and if you trace this to the code it actually returns an empty hash so lower your loglevel and the message goes away.

P.S. Can you also explain why it would be doing 100+ API calls per second?

Messages are run through the filter in batches. The filter includes a caching mechanism and attempts to retrieve meta from the cache or hits the API server when its not found. There is logic to avoid the behavior you are seeing which is to skip trying to make an API call if you are never going to find anything; no info will ever be found for a deleted pod. In this case, it looks like the issue is we find the namespace meta but not the pod meta, but this scenario is never recognized as a "miss" so it continues to try and find the pod metadata when evaluating subsequent records. This scenario will play out for every deleted pod which:

alex-vmw commented 3 years ago

In this case, it looks like the issue is we find the namespace meta but not the pod meta, but this scenario is never recognized as a "miss" so it continues to try and find the pod metadata when evaluating subsequent records. This scenario will play out for every deleted pod which:

  • produces log messages
  • the kublet has not GC'd the container log files before fluent is able to read them

Sorry, are you saying that this is intended behavior or are you saying that this is in fact an issue and filter needs protection from the scenario you described above?

jcantrill commented 3 years ago

In this case, it looks like the issue is we find the namespace meta but not the pod meta, but this scenario is never recognized as a "miss" so it continues to try and find the pod metadata when evaluating subsequent records. This scenario will play out for every deleted pod which:

  • produces log messages
  • the kublet has not GC'd the container log files before fluent is able to read them

Sorry, are you saying that this is intended behavior or are you saying that this is in fact an issue and filter needs protection from the scenario you described above?

I believe it is in fact an issue but I will need to write a unit test to confirm. Regardless, I do not believe messages are "stuck" (I encourage you to see if messages are delivered) but the filter is unnecessarily hitting the API server.

alex-vmw commented 3 years ago

Regardless, I do not believe messages are "stuck" (I encourage you to see if messages are delivered) but the filter is unnecessarily hitting the API server.

In fact, we started investigating this issue because we stopped receiving logs. However, I can't 100% be sure that metadata filter was the issue of not receiving logs as we have seen way too many in_tail plugin issues lately that also result in logs not getting processed.

jcantrill commented 3 years ago

In fact, we started investigating this issue because we stopped receiving logs. However, I can't 100% be sure that metadata filter was the issue of not receiving logs as we have seen way too many in_tail plugin issues lately that also result in logs not getting processed.

Our experience with fluend in a containerized environment for OpenShift is this: when you begin to see a high container density, and higher log volumes, fluentd is unable to recognize the file rotations that occur from the container runtime. Whole sets of files are being rotated out from under fluent and it never sees the logs. This is because fluentd is simply not efficient enough in reading the large sets of container files and is CPU bound. You can mitigate this loss by moving to a more performant collector like fluentbit or vector which is C and Rust based. The throughput is an order of magnitude faster.

alex-vmw commented 3 years ago

@jcantrill Sorry to bug you, do you have any possible ETA for a fix?

tingeltangelthomas commented 2 years ago

Hi @jcantrill , are there any news about that topic?

jcantrill commented 2 years ago

I have not done anything to address this issue. PRs are welcome.

jcantrill commented 1 year ago

Closing this issue due to age but please reopen with any PRs or if the behavior still exists