search

jaegertracing / jaeger-operator

Jaeger Operator for Kubernetes simplifies deploying and running Jaeger on Kubernetes.
https://www.jaegertracing.io/docs/latest/operator/
Apache License 2.0
1.03k stars 345 forks source link

Unusual info message on es cluster self-provision deployment #465

Closed jkandasa closed 5 years ago

jkandasa commented 5 years ago

I installed jaeger services with self-provision ES cluster. I see an unusual info message on the OpenShift console, I do not see any functional impact.

elasticsearch-cdm-lc57w4ft-1 is paused.This will stop any new rollouts or triggers from running until resumed. Resume Rollouts

image

Version details:

OpenShift Master: v3.11.98 Kubernetes Master: v1.11.0+d4cacc0 OpenShift Web Console: v3.11.98

CR File:

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: jaegerqe
spec:
  ingress:
    security: none
  strategy: production
  collector:
    replicas: 1
    image: jaegertracing/jaeger-collector:1.12
    resources:
      requests:
        memory: "512Mi"
        cpu: "1"
      limits:
        memory: "512Mi"
        cpu: "1"
    options:
      log-level: info
      metrics-backend: prometheus
      collector:
        num-workers: 1
        queue-size: 20000
      es:
        bulk:
          size: 524288
          workers: 1
          flush-interval: 200ms
        tags-as-fields:
          all: false
  query:
    replicas: 1
    image: jaegertracing/jaeger-query:1.12
    resources:
      requests:
        cpu: "500m"
        memory: "512Mi"
      limits:
        cpu: "500m"
        memory: "512Mi"
    options:
      log-level: info
      metrics-backend: prometheus
      query:
        port: 16686
      es:
        timeout: 10s
  agent:
    strategy: sidecar
    image: jaegertracing/jaeger-agent:1.12
    resources:
      requests:
        cpu: "200m"
        memory: "128Mi"
      limits:
        cpu: "200m"
        memory: "128Mi"
    options:
      log-level: info
      metrics-backend: prometheus
      processor:
        jaeger-compact:
          server-queue-size: 10000
          workers: 10
  storage:
    type: elasticsearch
    esIndexCleaner:
      enabled: false
    dependencies:
      enabled: false
    elasticsearch:
      image: quay.io/openshift/origin-logging-elasticsearch5:latest
      nodeCount: 3
      resources:

oc describe

$ oc describe pod elasticsearch-cdm-lc57w4ft-1
Name:               elasticsearch-cdm-lc57w4ft-1-54d98cbbc7-k8ljp
Namespace:          jaeger-pipeline
Priority:           0
PriorityClassName:  <none>
Node:               private.redhat.com/10.16.23.52
Start Time:         Thu, 06 Jun 2019 19:47:54 +0530
Labels:             cluster-name=elasticsearch
                    component=elasticsearch
                    es-node-client=true
                    es-node-data=true
                    es-node-master=true
                    node-name=elasticsearch-cdm-lc57w4ft-1
                    pod-template-hash=1085476673
                    tuned.openshift.io/elasticsearch=true
Annotations:        openshift.io/scc=restricted
Status:             Running
IP:                 10.129.1.125
Controlled By:      ReplicaSet/elasticsearch-cdm-lc57w4ft-1-54d98cbbc7
Containers:
  elasticsearch:
    Container ID:   docker://1fc32ccfaba0b572e15a11bf1a05be468a27697bb00bb86b947b4a76019d6364
    Image:          quay.io/openshift/origin-logging-elasticsearch5:latest
    Image ID:       docker-pullable://quay.io/openshift/origin-logging-elasticsearch5@sha256:d8d850b1d6c5047449c8014b4039d9f4ab5b891618bd968d52b86f3f07ef75e2
    Ports:          9300/TCP, 9200/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Running
      Started:      Thu, 06 Jun 2019 19:47:58 +0530
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     4
      memory:  4Gi
    Requests:
      cpu:      100m
      memory:   1Gi
    Readiness:  exec [/usr/share/elasticsearch/probe/readiness.sh] delay=10s timeout=30s period=5s #success=1 #failure=3
    Environment:
      DC_NAME:                  elasticsearch-cdm-lc57w4ft-1
      NAMESPACE:                jaeger-pipeline (v1:metadata.namespace)
      KUBERNETES_TRUST_CERT:    true
      SERVICE_DNS:              elasticsearch-cluster
      CLUSTER_NAME:             elasticsearch
      INSTANCE_RAM:             4Gi
      HEAP_DUMP_LOCATION:       /elasticsearch/persistent/heapdump.hprof
      RECOVER_AFTER_TIME:       5m
      READINESS_PROBE_TIMEOUT:  30
      POD_LABEL:                cluster=elasticsearch
      IS_MASTER:                true
      HAS_DATA:                 true
    Mounts:
      /elasticsearch/persistent from elasticsearch-storage (rw)
      /etc/openshift/elasticsearch/secret from certificates (rw)
      /usr/share/java/elasticsearch/config from elasticsearch-config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from elasticsearch-token-kklc8 (ro)
  proxy:
    Container ID:  docker://957bf699e68e5405bcd5ab3e87412a4d0835d028d8dd1f0296f239bbbf1a8918
    Image:         quay.io/openshift/origin-oauth-proxy:v4.0.0
    Image ID:      docker-pullable://quay.io/openshift/origin-oauth-proxy@sha256:f73bfe880c1caaf4a0a03cb6ffdb58baab2170e12ebafab26ea8e6abba66b3f4
    Port:          60000/TCP
    Host Port:     0/TCP
    Args:
      --https-address=:60000
      --provider=openshift
      --upstream=https://127.0.0.1:9200
      --tls-cert=/etc/proxy/secrets/tls.crt
      --tls-key=/etc/proxy/secrets/tls.key
      --upstream-ca=/etc/proxy/elasticsearch/admin-ca
      --openshift-service-account=elasticsearch
      -openshift-sar={"resource": "namespaces", "verb": "get"}
      -openshift-delegate-urls={"/": {"resource": "namespaces", "verb": "get"}}
      --pass-user-bearer-token
      --cookie-secret=xWM5BnXDDsroK0IvKv/RFw==
    State:          Running
      Started:      Thu, 06 Jun 2019 19:47:59 +0530
    Ready:          True
    Restart Count:  0
    Limits:
      memory:  64Mi
    Requests:
      cpu:        100m
      memory:     64Mi
    Environment:  <none>
    Mounts:
      /etc/proxy/elasticsearch from certificates (rw)
      /etc/proxy/secrets from elasticsearch-metrics (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from elasticsearch-token-kklc8 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  elasticsearch-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      elasticsearch
    Optional:  false
  elasticsearch-storage:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:  
  certificates:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  elasticsearch
    Optional:    false
  elasticsearch-metrics:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  elasticsearch-metrics
    Optional:    false
  elasticsearch-token-kklc8:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  elasticsearch-token-kklc8
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  node-role.kubernetes.io/compute=true
Tolerations:     node.kubernetes.io/disk-pressure:NoSchedule
                 node.kubernetes.io/memory-pressure:NoSchedule
Events:
  Type     Reason       Age               From                                       Message
  ----     ------       ----              ----                                       -------
  Normal   Scheduled    1m                default-scheduler                          Successfully assigned jaeger-pipeline/elasticsearch-cdm-lc57w4ft-1-54d98cbbc7-k8ljp to private.redhat.com
  Warning  FailedMount  1m                kubelet, private.redhat.com  MountVolume.SetUp failed for volume "elasticsearch-metrics" : secrets "elasticsearch-metrics" not found
  Normal   Pulled       1m                kubelet, private.redhat.com  Container image "quay.io/openshift/origin-logging-elasticsearch5:latest" already present on machine
  Normal   Created      1m                kubelet, private.redhat.com  Created container
  Normal   Started      1m                kubelet, private.redhat.com  Started container
  Normal   Pulled       1m                kubelet, private.redhat.com  Container image "quay.io/openshift/origin-oauth-proxy:v4.0.0" already present on machine
  Normal   Created      1m                kubelet, private.redhat.com  Created container
  Normal   Started      1m                kubelet, private.redhat.com  Started container
  Warning  Unhealthy    1m (x3 over 1m)   kubelet, private.redhat.com  Readiness probe failed: Elasticsearch node is not ready to accept HTTP requests yet [response code: 000]
  Warning  Unhealthy    1m (x2 over 1m)   kubelet, private.redhat.com  Readiness probe failed: Elasticsearch node is not ready to accept HTTP requests yet [response code: 503]
  Warning  Unhealthy    56s (x5 over 1m)  kubelet, private.redhat.com  Readiness probe failed: cat: /opt/app-root/src/init_failures: No such file or directory
pavolloffay commented 5 years ago

@jkandasa does this still happen? Can we close this ticket?

jkandasa commented 5 years ago

@pavolloffay Yes, Still I see this message,

image

oc describe:
$ oc describe pod elasticsearch-cdm-jkandasajaegerqe-1-699595fdf8-rslgf
Name:               elasticsearch-cdm-jkandasajaegerqe-1-699595fdf8-rslgf
Namespace:          jkandasa
Priority:           0
PriorityClassName:  <none>
Node:               ocp4-jqe2-7rbfh-worker-t76g7/192.168.0.15
Start Time:         Thu, 14 Nov 2019 17:01:50 +0530
Labels:             cluster-name=elasticsearch
                    component=elasticsearch
                    es-node-client=true
                    es-node-data=true
                    es-node-master=true
                    node-name=elasticsearch-cdm-jkandasajaegerqe-1
                    pod-template-hash=699595fdf8
                    tuned.openshift.io/elasticsearch=true
Annotations:        k8s.v1.cni.cncf.io/networks-status:
                      [{
                          "name": "openshift-sdn",
                          "interface": "eth0",
                          "ips": [
                              "10.128.2.122"
                          ],
                          "default": true,
                          "dns": {}
                      }]
                    openshift.io/scc: restricted
Status:             Running
IP:                 10.128.2.122
Controlled By:      ReplicaSet/elasticsearch-cdm-jkandasajaegerqe-1-699595fdf8
Containers:
  elasticsearch:
    Container ID:   cri-o://a5d1cfddf47c1831c28b26c36b9d94caf1e294d049aa933bd128435d73b17451
    Image:          registry.redhat.io/openshift4/ose-logging-elasticsearch5@sha256:a671fe2339f38dc58795e45de4a4310d12efb8ff476b729086badb4900b9c7ba
    Image ID:       registry.redhat.io/openshift4/ose-logging-elasticsearch5@sha256:a671fe2339f38dc58795e45de4a4310d12efb8ff476b729086badb4900b9c7ba
    Ports:          9300/TCP, 9200/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Running
      Started:      Thu, 14 Nov 2019 17:01:59 +0530
    Ready:          True
    Restart Count:  0
    Limits:
      memory:  1Gi
    Requests:
      cpu:      500m
      memory:   1Gi
    Readiness:  exec [/usr/share/elasticsearch/probe/readiness.sh] delay=10s timeout=30s period=5s #success=1 #failure=3
    Environment:
      DC_NAME:                  elasticsearch-cdm-jkandasajaegerqe-1
      NAMESPACE:                jkandasa (v1:metadata.namespace)
      KUBERNETES_TRUST_CERT:    true
      SERVICE_DNS:              elasticsearch-cluster
      CLUSTER_NAME:             elasticsearch
      INSTANCE_RAM:             1Gi
      HEAP_DUMP_LOCATION:       /elasticsearch/persistent/heapdump.hprof
      RECOVER_AFTER_TIME:       5m
      READINESS_PROBE_TIMEOUT:  30
      POD_LABEL:                cluster=elasticsearch
      IS_MASTER:                true
      HAS_DATA:                 true
    Mounts:
      /elasticsearch/persistent from elasticsearch-storage (rw)
      /etc/openshift/elasticsearch/secret from certificates (rw)
      /usr/share/java/elasticsearch/config from elasticsearch-config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from elasticsearch-token-jxr86 (ro)
  proxy:
    Container ID:  cri-o://f378a40d4f73231e714c5ad051f36e6e202ca3c4b43b562632b0f25a7e856b74
    Image:         registry.redhat.io/openshift4/ose-oauth-proxy@sha256:737b9f043fb85469441af421d0549146648fbfd04b0ef990dbcc7531c23bf1c9
    Image ID:      registry.redhat.io/openshift4/ose-oauth-proxy@sha256:737b9f043fb85469441af421d0549146648fbfd04b0ef990dbcc7531c23bf1c9
    Port:          60000/TCP
    Host Port:     0/TCP
    Args:
      --https-address=:60000
      --provider=openshift
      --upstream=https://127.0.0.1:9200
      --tls-cert=/etc/proxy/secrets/tls.crt
      --tls-key=/etc/proxy/secrets/tls.key
      --upstream-ca=/etc/proxy/elasticsearch/admin-ca
      --openshift-service-account=elasticsearch
      -openshift-sar={"resource": "namespaces", "verb": "get"}
      -openshift-delegate-urls={"/": {"resource": "namespaces", "verb": "get"}}
      --pass-user-bearer-token
      --cookie-secret=uGYP9IjOJ3bbSkaPu6jg4A==
    State:          Running
      Started:      Thu, 14 Nov 2019 17:01:59 +0530
    Ready:          True
    Restart Count:  0
    Limits:
      memory:  64Mi
    Requests:
      cpu:        100m
      memory:     64Mi
    Environment:  <none>
    Mounts:
      /etc/proxy/elasticsearch from certificates (rw)
      /etc/proxy/secrets from elasticsearch-metrics (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from elasticsearch-token-jxr86 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  elasticsearch-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      elasticsearch
    Optional:  false
  elasticsearch-storage:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  certificates:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  elasticsearch
    Optional:    false
  elasticsearch-metrics:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  elasticsearch-metrics
    Optional:    false
  elasticsearch-token-jxr86:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  elasticsearch-token-jxr86
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  kubernetes.io/os=linux
Tolerations:     node.kubernetes.io/disk-pressure:NoSchedule
                 node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                   From                                   Message
  ----     ------     ----                  ----                                   -------
  Normal   Scheduled  3m29s                 default-scheduler                      Successfully assigned jkandasa/elasticsearch-cdm-jkandasajaegerqe-1-699595fdf8-rslgf to ocp4-jqe2-7rbfh-worker-t76g7
  Normal   Pulled     3m20s                 kubelet, ocp4-jqe2-7rbfh-worker-t76g7  Container image "registry.redhat.io/openshift4/ose-logging-elasticsearch5@sha256:a671fe2339f38dc58795e45de4a4310d12efb8ff476b729086badb4900b9c7ba" already present on machine
  Normal   Created    3m20s                 kubelet, ocp4-jqe2-7rbfh-worker-t76g7  Created container elasticsearch
  Normal   Started    3m20s                 kubelet, ocp4-jqe2-7rbfh-worker-t76g7  Started container elasticsearch
  Normal   Pulled     3m20s                 kubelet, ocp4-jqe2-7rbfh-worker-t76g7  Container image "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:737b9f043fb85469441af421d0549146648fbfd04b0ef990dbcc7531c23bf1c9" already present on machine
  Normal   Created    3m20s                 kubelet, ocp4-jqe2-7rbfh-worker-t76g7  Created container proxy
  Normal   Started    3m20s                 kubelet, ocp4-jqe2-7rbfh-worker-t76g7  Started container proxy
  Warning  Unhealthy  2m59s (x3 over 3m9s)  kubelet, ocp4-jqe2-7rbfh-worker-t76g7  Readiness probe failed: cat: /opt/app-root/src/init_failures: No such file or directory
pavolloffay commented 5 years ago

I don't think this is related to Jaeger.

@ewolinetz is this message also prevent in ES deployment created for cluster logging?

pavolloffay commented 5 years ago

Copying discussion from slack

Eric Wolinetz  1 hour ago
@jcantril @ploffay the operator pauses the deployment so that if someone were to make an update it doesn't automatically roll that change out... its a way that we can control rollouts and make sure the cluster is ready to restart first

Pavol Loffay  1 hour ago
So is this message correct? Is the ES cluster in a correct state?

Eric Wolinetz  44 minutes ago
yes, thats intended -- its simply saying that that deployment is paused
https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#pausing-and-resuming-a-deployment (edit