Cloudevent traces not available on jaeger for PingSource

[Shashankft9]

Describe the bug Even though the PingSource is sending cloudevents to a sink every minute, I cant see the traces on jager UI.

Expected behavior The traces should come up with cloudevent tags.

To Reproduce PingSource CR:

apiVersion: sources.knative.dev/v1beta2
kind: PingSource
metadata:
  name: test-ping-source
  namespace: serverless
spec:
  schedule: "*/1 * * * *"
  contentType: "application/json"
  data: '{"message": "Hello world!"}'
  sink:
    ref:
      apiVersion: serving.knative.dev/v1
      kind: Service
      name: eventdisplay-go

Edited config-tracing cm:

  backend: zipkin
  enable: "true"
  sample-rate: "1"
  zipkin-endpoint: http://istio-jaeger-collector.istio-system.svc.cluster.local:9411/api/v2/spans

Knative release version v0.19.0

Additional context I can see APIServerSource traces on jaeger properly, a screen cap just so we are sure that zipking endpoint is not at fault here:

And the logs of pingsource adapter (of just moments ago):

{"level":"debug","ts":"2021-01-19T14:12:00.347Z","caller":"mtping/runner.go:129","msg":"sending cloudevent id: 9cf7aa15-bd6f-45f0-8ea0-bc50b12559a4, source: /apis/v1/namespaces/serverless/pingsources/test-ping-source, target: http://eventdisplay-go.serverless.svc.cluster.local","commit":"d2440ed","logging.googleapis.com/labels":{},"logging.googleapis.com/sourceLocation":{"file":"knative.dev/eventing/pkg/adapter/mtping/runner.go","line":"129","function":"knative.dev/eventing/pkg/adapter/mtping.(*cronJobsRunner).cronTick.func1"}}
{"level":"debug","ts":"2021-01-19T14:12:00.351Z","caller":"mtping/runner.go:136","msg":"Finished sending cloudevent id: 9cf7aa15-bd6f-45f0-8ea0-bc50b12559a4","commit":"d2440ed","logging.googleapis.com/labels":{},"logging.googleapis.com/sourceLocation":{"file":"knative.dev/eventing/pkg/adapter/mtping/runner.go","line":"136","function":"knative.dev/eventing/pkg/adapter/mtping.(*cronJobsRunner).cronTick.func1"}}

[lionelvillard]

Pinging @slinkydeveloper and @pierDipi as they recently worked on enabling tracing for other components.

[slinkydeveloper]

@lionelvillard mtping watches and properly configures the config-tracing CM?

[lionelvillard]

@Shashankft9 do you see this error in the PingSource adapter logs: Error setting up trace publishing ?

[Shashankft9]

I just checked, I do get this at the startup:

{"level":"warn","ts":"2021-01-19T14:11:31.574Z","caller":"v2/config.go:185","msg":"Tracing configuration is invalid, using the no-op default{error 26 0  empty json tracing config}","commit":"d2440ed","logging.googleapis.com/labels":{},"logging.googleapis.com/sourceLocation":{"file":"knative.dev/eventing/pkg/adapter/v2/config.go","line":"185","function":"knative.dev/eventing/pkg/adapter/v2.(*EnvConfig).SetupTracing"}}

[lionelvillard]

That's explain why you don't get traces but that does not explain why it works for APIServerSource. Can you do:

kubectl get -n knative-eventing deployments.apps pingsource-mt-adapter -oyaml 

and share the result?

[Shashankft9]

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "27"
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"eventing.knative.dev/release":"devel"},"name":"pingsource-mt-adapter","namespace":"knative-eventing"},"spec":{"replicas":0,"selector":{"matchLabels":{"eventing.knative.dev/source":"ping-source-controller","sources.knative.dev/role":"adapter"}},"template":{"metadata":{"labels":{"eventing.knative.dev/release":"devel","eventing.knative.dev/source":"ping-source-controller","sources.knative.dev/role":"adapter"}},"spec":{"containers":[{"env":[{"name":"SYSTEM_NAMESPACE","value":"","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"K_METRICS_CONFIG","value":""},{"name":"K_LOGGING_CONFIG","value":""},{"name":"K_LEADER_ELECTION_CONFIG","value":""},{"name":"K_NO_SHUTDOWN_AFTER","value":""},{"name":"K_SINK_TIMEOUT","value":"-1"},{"name":"POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}}],"image":"docker.io/shashankft/mtping-26ac4e6a49a610ffaac30cc0a8ab693a@sha256:3bd890f1bc05eea17ebae520d48a95122f1c9017259f99cc1d79fd000c705c93","name":"dispatcher","ports":[{"containerPort":9090,"name":"metrics","protocol":"TCP"}],"resources":{"limits":{"cpu":"1000m","memory":"2048Mi"},"requests":{"cpu":"125m","memory":"64Mi"}}}],"serviceAccountName":"pingsource-mt-adapter"}}}}
  creationTimestamp: "2021-01-04T07:38:15Z"
  generation: 28
  labels:
    eventing.knative.dev/release: devel
  name: pingsource-mt-adapter
  namespace: knative-eventing
  resourceVersion: "33580175"
  selfLink: /apis/apps/v1/namespaces/knative-eventing/deployments/pingsource-mt-adapter
  uid: 94b44820-1312-474f-8eb7-973cf67e3a29
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      eventing.knative.dev/source: ping-source-controller
      sources.knative.dev/role: adapter
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        eventing.knative.dev/release: devel
        eventing.knative.dev/source: ping-source-controller
        sources.knative.dev/role: adapter
    spec:
      containers:
      - env:
        - name: SYSTEM_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: K_METRICS_CONFIG
          value: '{"Domain":"knative.dev/eventing","Component":"pingsource","PrometheusPort":0,"PrometheusHost":"","ConfigMap":{}}'
        - name: K_LOGGING_CONFIG
          value: '{"zap-logger-config":"{\n  \"level\": \"debug\",\n  \"development\":
            false,\n  \"outputPaths\": [\"stdout\"],\n  \"errorOutputPaths\": [\"stderr\"],\n  \"encoding\":
            \"json\",\n  \"encoderConfig\": {\n    \"timeKey\": \"ts\",\n    \"levelKey\":
            \"level\",\n    \"nameKey\": \"logger\",\n    \"callerKey\": \"caller\",\n    \"messageKey\":
            \"msg\",\n    \"stacktraceKey\": \"stacktrace\",\n    \"lineEnding\":
            \"\",\n    \"levelEncoder\": \"\",\n    \"timeEncoder\": \"iso8601\",\n    \"durationEncoder\":
            \"\",\n    \"callerEncoder\": \"\"\n  }\n}\n"}'
        - name: K_LEADER_ELECTION_CONFIG
          value: '{"Component":"pingsource-mt-adapter","Buckets":1,"LeaseDuration":15000000000,"RenewDeadline":10000000000,"RetryPeriod":2000000000,"Identity":""}'
        - name: K_NO_SHUTDOWN_AFTER
          value: "55"
        - name: K_SINK_TIMEOUT
          value: "-1"
        image: docker.io/shashankft/mtping-26ac4e6a49a610ffaac30cc0a8ab693a@sha256:3bd890f1bc05eea17ebae520d48a95122f1c9017259f99cc1d79fd000c705c93
        imagePullPolicy: IfNotPresent
        name: dispatcher
        ports:
        - containerPort: 9090
          name: metrics
          protocol: TCP
        resources:
          limits:
            cpu: "1"
            memory: 2Gi
          requests:
            cpu: 125m
            memory: 64Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: pingsource-mt-adapter
      serviceAccountName: pingsource-mt-adapter
      terminationGracePeriodSeconds: 30
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2021-01-04T07:38:15Z"
    lastUpdateTime: "2021-01-19T14:11:32Z"
    message: ReplicaSet "pingsource-mt-adapter-58478687f4" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  - lastTransitionTime: "2021-01-21T10:09:00Z"
    lastUpdateTime: "2021-01-21T10:09:00Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 28
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

[Shashankft9]

hmm i am using the image after doing some code change here: https://github.com/knative/eventing/blob/master/pkg/adapter/v2/config.go#L187 I changed e.Component to "" to make APIServerSource traces to work. context: https://github.com/knative/eventing/issues/1889#issuecomment-762033421

[lionelvillard]

thanks @Shashankft9 I see what's wrong, K_TRACING_CONFIG is missing. The controller does not set it.

[lionelvillard]

@cr22rc Do you have cycles to look at this one?