search

jaegertracing / jaeger-operator

Jaeger Operator for Kubernetes simplifies deploying and running Jaeger on Kubernetes.
https://www.jaegertracing.io/docs/latest/operator/
Apache License 2.0
1.03k stars 344 forks source link

Can we deploy agent as a daemonset in production strategy. #1284

Open rajeshdevops87 opened 4 years ago

rajeshdevops87 commented 4 years ago

I am able to deploy agent as a daemoset using all-in-one strategy but not as production strategy.

BTW, I am using https://github.com/jaegertracing/jaeger-operator/blob/master/examples/agent-as-daemonset.yaml for all-in-one

For prod I am using https://github.com/jaegertracing/jaeger-operator/blob/master/examples/simple-prod.yaml

jpkrohling commented 4 years ago

Please, provide the CR you are using, or a minimal CR that is able to reproduce the problem on minikube.

rajeshdevops87 commented 4 years ago

No major difference just adding agent info in the cr as below

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: simple-prod
spec:
  strategy: production
  storage:
    type: elasticsearch
    options:
      es:
        # Note: This assumes elasticsearch is running in the "default" namespace.
        server-urls: http://elasticsearch.default.svc:9200
  agent:
    strategy: DaemonSet 
    annotations:
       scheduler.alpha.kubernetes.io/critical-pod: ""
jpkrohling commented 4 years ago

I tried your CR here, and it does work for me:

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: simple-prod
spec:
  strategy: production
  storage:
    type: elasticsearch
    options:
      es:
        # Note: This assumes elasticsearch is running in the "default" namespace.
        server-urls: http://elasticsearch.default.svc:9200
  agent:
    strategy: DaemonSet 
    annotations:
       scheduler.alpha.kubernetes.io/critical-pod: ""

Here are my debug logs, which you can compare with yours and see what's going on:

INFO[0000] Running the operator locally; watching namespace "" 
INFO[0000] Versions                                      arch=amd64 identity=default.jaeger-operator jaeger=1.20.0 jaeger-operator=v1.20.0-22-g8b07b3fc operator-sdk=v0.18.2 os=linux version=go1.14.9
INFO[0000] Auto-detected the platform                    platform=kubernetes
INFO[0000] Auto-detected ingress api                     ingress-api=networking
INFO[0000] Automatically adjusted the 'es-provision' flag  es-provision=no
INFO[0000] Automatically adjusted the 'kafka-provision' flag  kafka-provision=no
INFO[0001] Install prometheus-operator in your cluster to create ServiceMonitor objects  error="no ServiceMonitor registered with the API"
DEBU[0001] Not running on OpenShift, so won't configure OAuthProxy imagestream. 
WARN[0002] failed to setup the Jaeger exporter           error="write udp 127.0.0.1:40889->127.0.0.1:6831: write: connection refused"
DEBU[0015] Reconciling Jaeger                            execution="2020-10-30 09:49:45.87548594 +0000 UTC" instance=simple-prod namespace=default
DEBU[0015] configured this operator as the owner of the CR  execution="2020-10-30 09:49:45.87548594 +0000 UTC" instance=simple-prod namespace=default operator-identity=default.jaeger-operator
DEBU[0015] Reconciling Jaeger                            execution="2020-10-30 09:49:45.89559606 +0000 UTC" instance=simple-prod namespace=default
DEBU[0015] Strategy chosen                               instance=simple-prod namespace=default strategy=production
DEBU[0015] Assembling the Sampling configmap             instance=simple-prod namespace=default
DEBU[0015] assembling a collector deployment             instance=simple-prod namespace=default
DEBU[0015] Assembling a query deployment                 instance=simple-prod namespace=default
DEBU[0015] injecting sidecar                             deployment=simple-prod-query instance=simple-prod namespace=default
DEBU[0015] creating service account                      account=simple-prod instance=simple-prod namespace=default
DEBU[0015] creating config maps                          configMap=simple-prod-sampling-configuration instance=simple-prod namespace=default
DEBU[0015] creating cronjob                              cronjob=simple-prod-spark-dependencies instance=simple-prod namespace=default
DEBU[0015] creating cronjob                              cronjob=simple-prod-es-index-cleaner instance=simple-prod namespace=default
DEBU[0015] creating service                              instance=simple-prod namespace=default service=simple-prod-query
DEBU[0015] creating service                              instance=simple-prod namespace=default service=simple-prod-collector-headless
DEBU[0015] creating service                              instance=simple-prod namespace=default service=simple-prod-collector
DEBU[0015] creating deployment                           deployment=simple-prod-collector instance=simple-prod namespace=default
DEBU[0015] creating deployment                           deployment=simple-prod-query instance=simple-prod namespace=default
DEBU[0015] Deployment has stabilized                     desired=0 name=simple-prod-collector namespace=default ready=0
DEBU[0015] Deployment doesn't exist yet.                 name=simple-prod-query namespace=default
WARN[0016] failed to setup the Jaeger exporter           error="write udp 127.0.0.1:40889->127.0.0.1:6831: write: connection refused"
DEBU[0016] Waiting for deployment to stabilize           desired=1 name=simple-prod-query namespace=default ready=0
DEBU[0038] Deployment has stabilized                     desired=1 name=simple-prod-query namespace=default ready=1
DEBU[0038] creating ingress                              ingress=simple-prod-query instance=simple-prod namespace=default
DEBU[0038] creating hpa                                  hpa=simple-prod-collector instance=simple-prod namespace=default
DEBU[0038] creating daemonset                            daemonset=simple-prod-agent-daemonset instance=simple-prod namespace=default
DEBU[0038] Reconciling Jaeger completed                  execution="2020-10-30 09:49:45.89559606 +0000 UTC" instance=simple-prod namespace=default
DEBU[0038] Reconciling Jaeger                            execution="2020-10-30 09:50:09.103082854 +0000 UTC" instance=simple-prod namespace=default
DEBU[0038] Strategy chosen                               instance=simple-prod namespace=default strategy=production
DEBU[0038] Assembling the Sampling configmap             instance=simple-prod namespace=default
DEBU[0038] assembling a collector deployment             instance=simple-prod namespace=default
DEBU[0038] Assembling a query deployment                 instance=simple-prod namespace=default
DEBU[0038] injecting sidecar                             deployment=simple-prod-query instance=simple-prod namespace=default
WARN[0038] failed to setup the Jaeger exporter           error="write udp 127.0.0.1:40889->127.0.0.1:6831: write: connection refused"
DEBU[0038] updating service account                      account=simple-prod instance=simple-prod namespace=default
DEBU[0038] updating config maps                          configMap=simple-prod-sampling-configuration instance=simple-prod namespace=default
DEBU[0038] updating cronjob                              cronjob=simple-prod-spark-dependencies instance=simple-prod namespace=default
DEBU[0038] updating cronjob                              cronjob=simple-prod-es-index-cleaner instance=simple-prod namespace=default
DEBU[0038] updating service                              instance=simple-prod namespace=default service=simple-prod-collector-headless
DEBU[0038] updating service                              instance=simple-prod namespace=default service=simple-prod-collector
DEBU[0038] updating service                              instance=simple-prod namespace=default service=simple-prod-query
DEBU[0038] updating deployment                           deployment=simple-prod-collector instance=simple-prod namespace=default
DEBU[0038] updating deployment                           deployment=simple-prod-query instance=simple-prod namespace=default
DEBU[0038] Deployment has stabilized                     desired=1 name=simple-prod-collector namespace=default ready=1
DEBU[0038] Deployment has stabilized                     desired=1 name=simple-prod-query namespace=default ready=1
DEBU[0038] updating ingress                              ingress=simple-prod-query instance=simple-prod namespace=default
DEBU[0038] updating hpa                                  hpa=simple-prod-collector instance=simple-prod namespace=default
DEBU[0038] updating daemonset                            daemonset=simple-prod-agent-daemonset instance=simple-prod namespace=default
DEBU[0038] Reconciling Jaeger completed                  execution="2020-10-30 09:50:09.103082854 +0000 UTC" instance=simple-prod namespace=default
DEBU[0038] Reconciling Jaeger                            execution="2020-10-30 09:50:09.159176178 +0000 UTC" instance=simple-prod namespace=default
DEBU[0038] Strategy chosen                               instance=simple-prod namespace=default strategy=production
DEBU[0038] Assembling the Sampling configmap             instance=simple-prod namespace=default
DEBU[0038] assembling a collector deployment             instance=simple-prod namespace=default
DEBU[0038] Assembling a query deployment                 instance=simple-prod namespace=default
DEBU[0038] injecting sidecar                             deployment=simple-prod-query instance=simple-prod namespace=default
WARN[0038] failed to setup the Jaeger exporter           error="write udp 127.0.0.1:40889->127.0.0.1:6831: write: connection refused"
DEBU[0038] updating service account                      account=simple-prod instance=simple-prod namespace=default
DEBU[0038] updating config maps                          configMap=simple-prod-sampling-configuration instance=simple-prod namespace=default
DEBU[0038] updating cronjob                              cronjob=simple-prod-spark-dependencies instance=simple-prod namespace=default
DEBU[0038] updating cronjob                              cronjob=simple-prod-es-index-cleaner instance=simple-prod namespace=default
DEBU[0038] updating service                              instance=simple-prod namespace=default service=simple-prod-collector-headless
DEBU[0038] updating service                              instance=simple-prod namespace=default service=simple-prod-collector
DEBU[0038] updating service                              instance=simple-prod namespace=default service=simple-prod-query
DEBU[0039] updating deployment                           deployment=simple-prod-collector instance=simple-prod namespace=default
DEBU[0039] updating deployment                           deployment=simple-prod-query instance=simple-prod namespace=default
DEBU[0039] Deployment has stabilized                     desired=1 name=simple-prod-collector namespace=default ready=1
DEBU[0039] Deployment has stabilized                     desired=1 name=simple-prod-query namespace=default ready=1
DEBU[0039] updating ingress                              ingress=simple-prod-query instance=simple-prod namespace=default
DEBU[0039] updating hpa                                  hpa=simple-prod-collector instance=simple-prod namespace=default
DEBU[0039] updating daemonset                            daemonset=simple-prod-agent-daemonset instance=simple-prod namespace=default
DEBU[0039] Reconciling Jaeger completed                  execution="2020-10-30 09:50:09.159176178 +0000 UTC" instance=simple-prod namespace=default
WARN[0040] failed to setup the Jaeger exporter           error="write udp 127.0.0.1:40889->127.0.0.1:6831: write: connection refused"
tux-o-matic commented 3 years ago

BTW: If this is in an OCP cluster (@jpkrohling's test is vanilla K8s), Red Hat doesn't support DaemonSet Jaeger strategy, only side car injection. Was told that it was to enforce multi-tenancy so a single agent on a Node wouldn't handle data from various Pod/Namespace.

stale[bot] commented 3 years ago

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

jpkrohling commented 3 years ago

If this is indeed an OpenShift cluster, you might need to enable one or two options in your cluster to allow the pod to bind to a node port. It should be part of the docs, but let me know if you aren't able to find it. In any case, I'd suggest opening a support case, as Red Hat Support is properly staffed to answer those questions, even when engineering is on PTO ;-)