search

jaegertracing / jaeger-operator

Jaeger Operator for Kubernetes simplifies deploying and running Jaeger on Kubernetes.
https://www.jaegertracing.io/docs/latest/operator/
Apache License 2.0
1.02k stars 344 forks source link

jaeger-operator fails to deploy jaeger services - redunancyPolicy issue #497

Closed jkandasa closed 5 years ago

jkandasa commented 5 years ago

Error:

time="2019-07-03T11:21:56Z" level=error msg="failed to apply the changes" error="Elasticsearch.logging.openshift.io \"elasticsearch\" is invalid: []: Invalid value: map[string]interface {}{\"metadata\":map[string]interface {}{\"uid\":\"c4d2fdeb-9d84-11e9-a135-fa163ee1b294\", \"name\":\"elasticsearch\", \"namespace\":\"jkandasa\", \"creationTimestamp\":\"2019-07-03T11:21:57Z\", \"labels\":map[string]interface {}{\"app\":\"jaeger\", \"app.kubernetes.io/component\":\"elasticsearch\", \"app.kubernetes.io/instance\":\"simple-prod\", \"app.kubernetes.io/name\":\"elasticsearch\", \"app.kubernetes.io/part-of\":\"jaeger\"}, \"ownerReferences\":[]interface {}{map[string]interface {}{\"apiVersion\":\"jaegertracing.io/v1\", \"kind\":\"Jaeger\", \"name\":\"simple-prod\", \"uid\":\"c43494f3-9d84-11e9-a135-fa163ee1b294\", \"controller\":true}}, \"generation\":1}, \"spec\":map[string]interface {}{\"managementState\":\"Managed\", \"redundancyPolicy\":\"\", \"nodes\":[]interface {}{map[string]interface {}{\"roles\":[]interface {}{\"client\", \"data\", \"master\"}, \"nodeCount\":1, \"resources\":map[string]interface {}{}, \"storage\":map[string]interface {}{}}}, \"nodeSpec\":map[string]interface {}{\"resources\":map[string]interface {}{}}}, \"status\":map[string]interface {}{\"nodes\":interface {}(nil), \"clusterHealth\":\"\", \"shardAllocationEnabled\":\"\", \"pods\":interface {}(nil), \"conditions\":interface {}(nil)}, \"kind\":\"Elasticsearch\", \"apiVersion\":\"logging.openshift.io/v1\"}: validation failure list:\nspec.redundancyPolicy in body should be one of [FullRedundancy MultipleRedundancy SingleRedundancy ZeroRedundancy]" execution="2019-07-03 11:21:55.966963378 +0000 UTC" instance=simple-prod namespace=jkandasa

jaeger Operator log: jaeger-operator-b4bcfb56d-z5k2v-jaeger-operator.log

Versions:

jaeger CR file: https://github.com/jaegertracing/jaeger-operator/blob/v1.13.0/deploy/examples/simple-prod-deploy-es.yaml

Elastic search operator deployed from OperatorHub:

apiVersion: operators.coreos.com/v1alpha1
kind: ClusterServiceVersion
metadata:
  annotations:
    olm.skipRange: '>=4.1.0 <4.1.2'
    certified: 'false'
    olm.targetNamespaces: ''
    support: 'AOS Cluster Logging, Jaeger'
    alm-examples: |-
      [
          {
              "apiVersion": "logging.openshift.io/v1",
              "kind": "Elasticsearch",
              "metadata": {
                "name": "elasticsearch"
              },
              "spec": {
                "managementState": "Managed",
                "nodeSpec": {
                  "image": "registry.redhat.io/openshift4/ose-logging-elasticsearch5:v4.1.0-201905191700",
                  "resources": {
                    "limits": {
                      "memory": "1Gi"
                    },
                    "requests": {
                      "memory": "512Mi"
                    }
                  }
                },
                "nodes": [
                  {
                      "nodeCount": 1,
                      "roles": ["client","data","master"],
                      "redundancyPolicy": "SingleRedundancy"
                  }
                ]
              }
          }
      ]
    capabilities: Seamless Upgrades
    olm.operatorNamespace: openshift-operators
    containerImage: >-
      registry.redhat.io/openshift4/ose-elasticsearch-operator:v4.1.2-201906121519
    createdAt: '2019-02-20 08:00:00'
    categories: 'OpenShift Optional, Logging & Tracing'
    description: >-
      The Elasticsearch Operator for OKD provides a means for configuring and
      managing an Elasticsearch cluster for tracing and cluster logging.

      ## Prerequisites and Requirements

      ### Elasticsearch Operator Namespace

      The Elasticsearch Operator must be deployed to the global operator group
      namespace

      ### Memory Considerations

      Elasticsearch is a memory intensive application.  The initial

      set of OKD nodes may not be large enough to support the Elasticsearch
      cluster.  Additional OKD nodes must be added

      to the OKD cluster if you desire to run with the recommended(or better)
      memory. Each ES node can operate with a

      lower memory setting though this is not recommended for production
      deployments.
    olm.operatorGroup: global-operators
  selfLink: >-
    /apis/operators.coreos.com/v1alpha1/namespaces/openshift-operators/clusterserviceversions/elasticsearch-operator.v4.1.2
  resourceVersion: '1765346'
  name: elasticsearch-operator.v4.1.2
  uid: d92c93b5-9d7d-11e9-a43e-fa163e78af6f
  creationTimestamp: '2019-07-03T10:32:25Z'
  generation: 1
  namespace: openshift-operators
  labels:
    olm.api.e43efcaa45c9f8d0: provided
spec:
  customresourcedefinitions:
    owned:
      - description: An Elasticsearch cluster instance
        displayName: Elasticsearch
        kind: Elasticsearch
        name: elasticsearches.logging.openshift.io
        resources:
          - kind: Deployment
            name: ''
            version: v1
          - kind: StatefulSet
            name: ''
            version: v1
          - kind: ReplicaSet
            name: ''
            version: v1
          - kind: Pod
            name: ''
            version: v1
          - kind: ConfigMap
            name: ''
            version: v1
          - kind: Service
            name: ''
            version: v1
          - kind: Route
            name: ''
            version: v1
        specDescriptors:
          - description: The name of the serviceaccount used by the Elasticsearch pods
            displayName: Service Account
            path: serviceAccountName
            x-descriptors:
              - 'urn:alm:descriptor:io.kubernetes:ServiceAccount'
          - description: The name of the configmap used by the Elasticsearch pods
            displayName: Config Map
            path: configMapName
            x-descriptors:
              - 'urn:alm:descriptor:io.kubernetes:ConfigMap'
          - description: The name of the secret used by the Elasticsearch pods
            displayName: Secret
            path: secretName
            x-descriptors:
              - 'urn:alm:descriptor:io.kubernetes:Secret'
          - description: >-
              Limits describes the minimum/maximum amount of compute resources
              required/allowed
            displayName: Resource Requirements
            path: nodeSpec.resources
            x-descriptors:
              - 'urn:alm:descriptor:com.tectonic.ui:resourceRequirements'
        statusDescriptors:
          - description: The current health of Elasticsearch Cluster
            displayName: Elasticsearch Cluster Health
            path: clusterHealth
            x-descriptors:
              - 'urn:alm:descriptor:io.kubernetes.phase'
          - description: The status for each of the Elasticsearch pods with the Client role
            displayName: Elasticsearch Client Status
            path: pods.client
            x-descriptors:
              - 'urn:alm:descriptor:com.tectonic.ui:podStatuses'
          - description: The status for each of the Elasticsearch pods with the Data role
            displayName: Elasticsearch Data Status
            path: pods.data
            x-descriptors:
              - 'urn:alm:descriptor:com.tectonic.ui:podStatuses'
          - description: The status for each of the Elasticsearch pods with the Master role
            displayName: Elasticsearch Master Status
            path: pods.master
            x-descriptors:
              - 'urn:alm:descriptor:com.tectonic.ui:podStatuses'
        version: v1
  apiservicedefinitions: {}
  keywords:
    - elasticsearch
    - jaeger
  displayName: Elasticsearch Operator
  provider:
    name: 'Red Hat, Inc'
  installModes:
    - supported: true
      type: OwnNamespace
    - supported: false
      type: SingleNamespace
    - supported: false
      type: MultiNamespace
    - supported: true
      type: AllNamespaces
  version: 4.1.2
  links:
    - name: Elastic
      url: 'https://www.elastic.co/'
    - name: Elasticsearch Operator
      url: 'https://github.com/openshift/elasticsearch-operator'
  install:
    spec:
      clusterPermissions:
        - rules:
            - apiGroups:
                - logging.openshift.io
              resources:
                - '*'
              verbs:
                - '*'
            - apiGroups:
                - ''
              resources:
                - pods
                - pods/exec
                - services
                - endpoints
                - persistentvolumeclaims
                - events
                - configmaps
                - secrets
                - serviceaccounts
              verbs:
                - '*'
            - apiGroups:
                - apps
              resources:
                - deployments
                - daemonsets
                - replicasets
                - statefulsets
              verbs:
                - '*'
            - apiGroups:
                - monitoring.coreos.com
              resources:
                - prometheusrules
                - servicemonitors
              verbs:
                - '*'
            - apiGroups:
                - rbac.authorization.k8s.io
              resources:
                - clusterroles
                - clusterrolebindings
              verbs:
                - '*'
            - nonResourceURLs:
                - /metrics
              verbs:
                - get
            - apiGroups:
                - authentication.k8s.io
              resources:
                - tokenreviews
                - subjectaccessreviews
              verbs:
                - create
            - apiGroups:
                - authorization.k8s.io
              resources:
                - subjectaccessreviews
              verbs:
                - create
          serviceAccountName: elasticsearch-operator
      deployments:
        - name: elasticsearch-operator
          spec:
            replicas: 1
            selector:
              matchLabels:
                name: elasticsearch-operator
            template:
              metadata:
                labels:
                  name: elasticsearch-operator
              spec:
                containers:
                  - command:
                      - elasticsearch-operator
                    env:
                      - name: WATCH_NAMESPACE
                        valueFrom:
                          fieldRef:
                            fieldPath: 'metadata.annotations[''olm.targetNamespaces'']'
                      - name: OPERATOR_NAME
                        value: elasticsearch-operator
                      - name: PROXY_IMAGE
                        value: >-
                          registry.redhat.io/openshift4/ose-oauth-proxy:v4.1.0-201905191700
                      - name: ELASTICSEARCH_IMAGE
                        value: >-
                          registry.redhat.io/openshift4/ose-logging-elasticsearch5:v4.1.0-201905191700
                    image: >-
                      registry.redhat.io/openshift4/ose-elasticsearch-operator:v4.1.2-201906121519
                    imagePullPolicy: IfNotPresent
                    name: elasticsearch-operator
                    ports:
                      - containerPort: 60000
                        name: metrics
                serviceAccountName: elasticsearch-operator
    strategy: deployment
  maintainers:
    - email: aos-logging@redhat.com
      name: 'Red Hat, AOS Logging'
  description: >
    The Elasticsearch Operator for OKD provides a means for configuring and
    managing an Elasticsearch cluster for use in tracing and cluster logging.

    This operator only supports OKD Cluster Logging and Jaeger.  It is tightly
    coupled to each and is not currently capable of

    being used as a general purpose manager of Elasticsearch clusters running on
    OKD.

    It is recommended this operator be deployed to the **openshift-operators**
    namespace to properly support the Cluster Logging and Jaeger use cases.

    Once installed, the operator provides the following features:

    * **Create/Destroy**: Deploy an Elasticsearch cluster to the same namespace
    in which the Elasticsearch custom resource is created.
status:
  reason: InstallSucceeded
  message: install strategy completed with no errors
  lastUpdateTime: '2019-07-03T10:32:44Z'
  requirementStatus:
    - group: operators.coreos.com
      kind: ClusterServiceVersion
      message: CSV missing minimum kube version specification
      name: elasticsearch-operator.v4.1.2
      status: NotPresent
      version: v1alpha1
    - group: apiextensions.k8s.io
      kind: CustomResourceDefinition
      message: CRD is present and Established condition is true
      name: elasticsearches.logging.openshift.io
      status: Present
      uuid: d92f8e7b-9d7d-11e9-a43e-fa163e78af6f
      version: v1beta1
    - dependents:
        - group: rbac.authorization.k8s.io
          kind: PolicyRule
          message: >-
            cluster
            rule:{"verbs":["*"],"apiGroups":["logging.openshift.io"],"resources":["*"]}
          status: Satisfied
          version: v1beta1
        - group: rbac.authorization.k8s.io
          kind: PolicyRule
          message: >-
            cluster
            rule:{"verbs":["*"],"apiGroups":[""],"resources":["pods","pods/exec","services","endpoints","persistentvolumeclaims","events","configmaps","secrets","serviceaccounts"]}
          status: Satisfied
          version: v1beta1
        - group: rbac.authorization.k8s.io
          kind: PolicyRule
          message: >-
            cluster
            rule:{"verbs":["*"],"apiGroups":["apps"],"resources":["deployments","daemonsets","replicasets","statefulsets"]}
          status: Satisfied
          version: v1beta1
        - group: rbac.authorization.k8s.io
          kind: PolicyRule
          message: >-
            cluster
            rule:{"verbs":["*"],"apiGroups":["monitoring.coreos.com"],"resources":["prometheusrules","servicemonitors"]}
          status: Satisfied
          version: v1beta1
        - group: rbac.authorization.k8s.io
          kind: PolicyRule
          message: >-
            cluster
            rule:{"verbs":["*"],"apiGroups":["rbac.authorization.k8s.io"],"resources":["clusterroles","clusterrolebindings"]}
          status: Satisfied
          version: v1beta1
        - group: rbac.authorization.k8s.io
          kind: PolicyRule
          message: 'cluster rule:{"verbs":["get"],"nonResourceURLs":["/metrics"]}'
          status: Satisfied
          version: v1beta1
        - group: rbac.authorization.k8s.io
          kind: PolicyRule
          message: >-
            cluster
            rule:{"verbs":["create"],"apiGroups":["authentication.k8s.io"],"resources":["tokenreviews","subjectaccessreviews"]}
          status: Satisfied
          version: v1beta1
        - group: rbac.authorization.k8s.io
          kind: PolicyRule
          message: >-
            cluster
            rule:{"verbs":["create"],"apiGroups":["authorization.k8s.io"],"resources":["subjectaccessreviews"]}
          status: Satisfied
          version: v1beta1
      group: ''
      kind: ServiceAccount
      message: ''
      name: elasticsearch-operator
      status: Present
      version: v1
  certsLastUpdated: null
  lastTransitionTime: '2019-07-03T10:32:44Z'
  conditions:
    - lastTransitionTime: '2019-07-03T10:32:26Z'
      lastUpdateTime: '2019-07-03T10:32:26Z'
      message: requirements not yet checked
      phase: Pending
      reason: RequirementsUnknown
    - lastTransitionTime: '2019-07-03T10:32:26Z'
      lastUpdateTime: '2019-07-03T10:32:26Z'
      message: 'all requirements found, attempting install'
      phase: InstallReady
      reason: AllRequirementsMet
    - lastTransitionTime: '2019-07-03T10:32:26Z'
      lastUpdateTime: '2019-07-03T10:32:26Z'
      message: waiting for install components to report healthy
      phase: Installing
      reason: InstallSucceeded
    - lastTransitionTime: '2019-07-03T10:32:26Z'
      lastUpdateTime: '2019-07-03T10:32:27Z'
      message: >
        installing: Waiting: waiting for deployment elasticsearch-operator to
        become ready: Waiting for rollout to finish: 0 of 1 updated replicas are
        available...
      phase: Installing
      reason: InstallWaiting
    - lastTransitionTime: '2019-07-03T10:32:44Z'
      lastUpdateTime: '2019-07-03T10:32:44Z'
      message: install strategy completed with no errors
      phase: Succeeded
      reason: InstallSucceeded
  phase: Succeeded
  certsRotateAt: null
jkandasa commented 5 years ago

When we add redundancyPolicy: ZeroRedundancy ES cluster comes up. Available Options: [FullRedundancy MultipleRedundancy SingleRedundancy ZeroRedundancy]

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: simple-prod
spec:
  strategy: production
  storage:
    type: elasticsearch
    elasticsearch:
      nodeCount: 1
      redundancyPolicy: ZeroRedundancy
      resources: