search

jaegertracing / jaeger

CNCF Jaeger, a Distributed Tracing Platform
https://www.jaegertracing.io/
Apache License 2.0
20.52k stars 2.44k forks source link

jaeger collector cannot TLS connect to ES v.6 with ES self signed certificate #1504

Closed yossishm closed 5 years ago

yossishm commented 5 years ago

Requirement - what kind of business use case are you trying to solve?

Trying to generate on same cluster (minikube) ES with self signed TLS deployed on ES.

Problem - what in Jaeger blocks you from solving the requirement?

Can't connect to ES cluster from jaeger collector: "level":"info","ts":1556354044.524639,"caller":"healthcheck/handler.go:99","msg":"Health Check server started","http-port":14269,"status":"unavailable"} {"level":"fatal","ts":1556354049.5972986,"caller":"collector/main.go:104","msg":"Failed to init storage factory","error":"failed to create primary Elasticsearch client: health check timeout: Head https://jaeger-elasticsearch-master:9200: x509: certificate signed by unknown authority: no Elasticsearch node available","errorVerbose":"no Elasticsearch node available\ngithub.com/jaegertracing/jaeger/vendor/gopkg.in/olivere/elastic%2ev5.init\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/gopkg.in/olivere/elastic.v5/client.go:88\ngithub.com/jaegertracing/jaeger/pkg/es.init\n\t:1\ngithub.com/jaegertracing/jaeger/plugin/storage/es.init\n\t:1\ngithub.com/jaegertracing/jaeger/plugin/storage.init\n\t:1\ngithub.com/jaegertracing/jaeger/cmd/env.init\n\t:1\nmain.init\n\t:1\nruntime.main\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/runtime/proc.go:189\nruntime.goexit\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/runtime/asm_amd64.s:1333\nhealth check timeout: Head https://jaeger-elasticsearch-master:9200: x509: certificate signed by unknown authority\ngithub.com/jaegertracing/jaeger/vendor/gopkg.in/olivere/elastic%2ev5.(Client).startupHealthcheck\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/gopkg.in/olivere/elastic.v5/client.go:1116\ngithub.com/jaegertracing/jaeger/vendor/gopkg.in/olivere/elastic%2ev5.NewClient\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/gopkg.in/olivere/elastic.v5/client.go:244\ngithub.com/jaegertracing/jaeger/pkg/es/config.(Configuration).NewClient\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/pkg/es/config/config.go:97\ngithub.com/jaegertracing/jaeger/plugin/storage/es.(Factory).Initialize\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/plugin/storage/es/factory.go:80\ngithub.com/jaegertracing/jaeger/plugin/storage.(Factory).Initialize\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/plugin/storage/factory.go:90\nmain.main.func1\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/collector/main.go:103\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(Command).execute\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:762\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(Command).ExecuteC\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:852\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(Command).Execute\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:800\nmain.main\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/collector/main.go:204\nruntime.main\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/runtime/proc.go:201\nruntime.goexit\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/runtime/asm_amd64.s:1333\nfailed to create primary Elasticsearch client\ngithub.com/jaegertracing/jaeger/plugin/storage/es.(Factory).Initialize\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/plugin/storage/es/factory.go:82\ngithub.com/jaegertracing/jaeger/plugin/storage.(Factory).Initialize\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/plugin/storage/factory.go:90\nmain.main.func1\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/collector/main.go:103\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(Command).execute\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:762\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(Command).ExecuteC\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:852\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(Command).Execute\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:800\nmain.main\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/collector/main.go:204\nruntime.main\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/runtime/proc.go:201\nruntime.goexit\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/runtime/asm_amd64.s:1333","stacktrace":"main.main.func1\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/collector/main.go:104\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(Command).execute\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:762\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(Command).ExecuteC\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:852\ngithub.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra.(*Command).Execute\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:800\nmain.main\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/collector/main.go:204\nruntime.main\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/runtime/proc.go:201"}

Proposal - what do you suggest to solve the problem or improve the existing situation?

Have a config option to ignore self signed certs?

Any open questions to address

yossishm commented 5 years ago

I configured the certificate on Elastic with Alternative SN of the domain. (--dns) https://www.elastic.co/guide/en/elasticsearch/reference/6.6/configuring-tls.html#node-certificates

yurishkuro commented 5 years ago

Recommendation:

yossishm commented 5 years ago
  1. Steps
    • generating certificate using elastic utility (v.6.6.2)
cd elasticsearch-6.6.2/
# elastic installed on mac

bin/elasticsearch-certutil cert --ca-dn cn=jaeger-elasticsearch-master --name "jaeger-elasticsearch-master" --dns "jaeger-elasticsearch-master"

kubectl create secret generic elastic-credentials  --from-literal=password=changeme --from-literal=username=elastic
kubectl create secret generic elastic-license --from-file=license.json
kubectl create secret generic elastic-certificates --from-file=elastic-certificates.p12

---
# myrel-jaeger-elastic-jubqc-test
apiVersion: v1
kind: Pod
metadata:
  name: "myrel-jaeger-elastic-jubqc-test"
  annotations:
    "helm.sh/hook": test-success
spec:
  containers:
  - name: "myrel-jaeger-elastic-yshpg-test"
    image: "docker.elastic.co/elasticsearch/elasticsearch:6.6.2"
    command:
      - "sh"
      - "-c"
      - |
        #!/usr/bin/env bash -e
        curl -XGET --fail 'jaeger-elasticsearch-master:9200/_cluster/health?wait_for_status=green&timeout=1s'
  restartPolicy: Never
MANIFEST:

---
# Source: elasticsearch/templates/poddisruptionbudget.yaml
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: "jaeger-elasticsearch-master-pdb"
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: "jaeger-elasticsearch-master"
---
# Source: elasticsearch/templates/service.yaml
kind: Service
apiVersion: v1
metadata:
  name: jaeger-elasticsearch-master
spec:
  selector:
    heritage: "Tiller"
    release: "myrel-jaeger-elastic"
    chart: "elasticsearch-6.5.0"
    app: "jaeger-elasticsearch-master"
  ports:
  - name: http
    protocol: TCP
    port: 9200
  - name: transport
    protocol: TCP
    port: 9300
---
# Source: elasticsearch/templates/service.yaml
kind: Service
apiVersion: v1
metadata:
  name: jaeger-elasticsearch-master-headless
  labels:
    heritage: "Tiller"
    release: "myrel-jaeger-elastic"
    chart: "elasticsearch-6.5.0"
    app: "jaeger-elasticsearch-master"
  annotations:
    # Create endpoints also if the related pod isn't ready
    service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
spec:
  clusterIP: None # This is needed for statefulset hostnames like elasticsearch-0 to resolve
  selector:
    app: "jaeger-elasticsearch-master"
  ports:
  - name: http
    port: 9200
---
# Source: elasticsearch/templates/statefulset.yaml
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: jaeger-elasticsearch-master
  labels:
    heritage: "Tiller"
    release: "myrel-jaeger-elastic"
    chart: "elasticsearch-6.5.0"
    app: "jaeger-elasticsearch-master"
spec:
  serviceName: jaeger-elasticsearch-master-headless
  selector:
    matchLabels:
      app: "jaeger-elasticsearch-master"
  replicas: 1
  podManagementPolicy: Parallel
  updateStrategy:
    type: RollingUpdate
  volumeClaimTemplates:
  - metadata:
      name: jaeger-elasticsearch-master
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi
      storageClassName: standard

  template:
    metadata:
      name: "jaeger-elasticsearch-master"
      labels:
        heritage: "Tiller"
        release: "myrel-jaeger-elastic"
        chart: "elasticsearch-6.5.0"
        app: "jaeger-elasticsearch-master"
    spec:
      securityContext:
        fsGroup: 1000
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 1
            podAffinityTerm:
              topologyKey: kubernetes.io/hostname
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - "jaeger-elasticsearch-master"
      terminationGracePeriodSeconds: 120
      volumes:
        - name: elastic-certificates
          secret:
            secretName: elastic-certificates
        - name: elastic-license
          secret:
            secretName: elastic-license
      initContainers:
      - name: configure-sysctl
        securityContext:
          runAsUser: 0
          privileged: true
        image: "docker.elastic.co/elasticsearch/elasticsearch:6.6.2"
        command: ["sysctl", "-w", "vm.max_map_count=262144"]
      containers:
      - name: "elasticsearch"
        image: "docker.elastic.co/elasticsearch/elasticsearch:6.6.2"
        imagePullPolicy: "IfNotPresent"
        readinessProbe:
          failureThreshold: 3
          initialDelaySecond#s: 10
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 3
          timeoutSeconds: 5

          exec:
            command: 
              - sh
              - -c
              - |
                #!/usr/bin/env bash -e
                # If the node is starting up wait for the cluster to be green
                # Once it has started only check that the node itself is responding
                START_FILE=/tmp/.es_start_file

                http () {
                    local path="${1}"
                    if [ -n "${ELASTIC_USERNAME}" ] && [ -n "${ELASTIC_PASSWORD}" ]; then
                      BASIC_AUTH="-u ${ELASTIC_USERNAME}:${ELASTIC_PASSWORD}"
                    else
                      BASIC_AUTH=''
                    fi
                    curl -XGET -s -k --fail ${BASIC_AUTH} https://127.0.0.1:9200${path}
                }

                if [ -f "${START_FILE}" ]; then
                    echo 'Elasticsearch is already running, lets check the node is healthy'
                    http "/"
                else
                    echo 'Waiting for elasticsearch cluster to become green'
                    if http "/_cluster/health?wait_for_status=green&timeout=1s" ; then
                        touch ${START_FILE}
                        exit 0
                    else
                        echo 'Cluster is not yet green'
                        exit 1
                    fi
                fi
        ports:
        - name: http
          containerPort: 9200
        - name: transport
          containerPort: 9300
        resources:
          limits:
            cpu: 1000m
            memory: 1Gi
          requests:
            cpu: 100m
            memory: 1Gi

        env:
          - name: node.name
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: discovery.zen.ping.unicast.hosts
            value: "jaeger-elasticsearch-master-headless"
          - name: cluster.name
            value: "jaeger-elasticsearch"
          - name: discovery.zen.minimum_master_nodes 
            value: "1"
          - name: network.host
            value: "0.0.0.0"
          - name: ES_JAVA_OPTS
            value: "-Xmx512m -Xms512m"
          - name: node.master
            value: "true"
          - name: node.data
            value: "true"
          - name: node.ingest
            value: "true"
          - name: xpack.security.enabled
            value: "true"
          - name: xpack.security.transport.ssl.enabled
            value: "false"
          - name: xpack.security.http.ssl.enabled
            value: "true"
          - name: xpack.security.http.ssl.truststore.path
            value: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
          - name: xpack.security.http.ssl.keystore.path
            value: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
          - name: ELASTIC_PASSWORD
            valueFrom:
              secretKeyRef:
                key: password
                name: elastic-credentials
          - name: ELASTIC_USERNAME
            valueFrom:
              secretKeyRef:
                key: username
                name: elastic-credentials

        volumeMounts:
          - name: "jaeger-elasticsearch-master"
            mountPath: /usr/share/elasticsearch/data
          - name: elastic-certificates
            mountPath: /usr/share/elasticsearch/config/certs
          - name: elastic-license
            mountPath: /usr/share/elasticsearch/config/license
---
# Source: elasticsearch/templates/ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: jaeger-elasticsearch-master
  labels:
    app: elasticsearch
    release: myrel-jaeger-elastic
    heritage: Tiller
spec:
  rules:
    - host: chart-example.local
      http:
        paths:
          - path: /
            backend:
              serviceName: jaeger-elasticsearch-master
              servicePort: 9200

Source: jaeger/templates/agent-svc.yaml

apiVersion: v1 kind: Service metadata: name: myrel-jaeger-agent labels: app.kubernetes.io/name: jaeger helm.sh/chart: jaeger-0.10.0 app.kubernetes.io/instance: myrel-jaeger app.kubernetes.io/managed-by: Tiller app.kubernetes.io/component: agent spec: ports:

  1. which "other means" you excpect - the error is quite clear: "x509: certificate signed by unknown authority: "
yossishm commented 5 years ago

I solved this issue by assigning cerificate using k8s CA and auto mounting to client. thanks.

yurishkuro commented 5 years ago

šŸŽ‰

madeinindiadot commented 4 years ago

I solved this issue by assigning cerificate using k8s CA and auto mounting to client. thanks.

@yossishm can you explain in detail how you did this