Opentelemetry collector for Knative works well in "metrics" namespace but not in others. How could we deploy it in another namespace ?

[AlexandreDelforgeAdeo]

Expected Behavior

I would like to deploy opentelemetry collector objects in another namespace than "metrics" one

Actual Behavior

When I am doing that, no metrics are exposed

And I have some errors in knative activator pod

Steps to Reproduce the Problem

1. Create a namespace "kmetrics"
2. Apply collector.yaml

   apiVersion: v1
   kind: ConfigMap
   metadata:
   name: otel-collector-config
   namespace: kmetrics
   data:
   collector.yaml: |
   receivers:
     opencensus:
       endpoint: "0.0.0.0:55678"
   exporters:
     logging:
     prometheus:
       endpoint: "0.0.0.0:8889"
   extensions:
       health_check:
       pprof:
       zpages:
   service:
     extensions: [health_check, pprof, zpages]
     pipelines:
       metrics:
         receivers: [opencensus]
         processors: []
         exporters: [prometheus]
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
   name: otel-collector
   namespace: kmetrics
   labels:
   app: otel-collector
   spec:
   selector:
   matchLabels:
     app: otel-collector
   replicas: 1  # This can be increased for a larger system.
   template:
   metadata:
     labels:
       app: otel-collector
   spec:
     containers:
     - name: collector
       args:
       - --config=/conf/collector.yaml
       image: otel/opentelemetry-collector:latest
       resources:
         requests:  # Note: these are suitable for a small instance, but may need to be increased for a large instance.
           memory: 100Mi
           cpu: 50m
       ports:
       - name: otel
         containerPort: 55678
       - name: prom-export
         containerPort: 8889
       - name: zpages  # A /debug page
         containerPort: 55679
       volumeMounts:
         - mountPath: /conf
           name: config
     volumes:
     - name: config
       configMap:
         name: otel-collector-config
         items:
           - key: collector.yaml
             path: collector.yaml
   ---
   apiVersion: v1
   kind: Service
   metadata:
   name: otel-collector
   namespace: kmetrics
   spec:
   selector:
   app: "otel-collector"
   ports:
   - port: 55678
   name: otel
   ---
   apiVersion: v1
   kind: Service
   metadata:
   name: otel-export
   namespace: kmetrics
   labels:
   app: otel-export
   spec:
   selector:
   app: otel-collector
   ports:
   - port: 8889
   name: prom-export

3. Apply the patch on cm config-observability

   kubectl patch --namespace knative-serving configmap/config-observability \
   --type merge \
   --patch '{"data":{"metrics.backend-destination":"opencensus","metrics.request-metrics-backend-destination":"opencensus","metrics.opencensus-address":"otel-collector.metrics:55678"}}'

4. Create a ksvc hello-world

   apiVersion: serving.knative.dev/v1
   kind: Service
   metadata:
   name: hello
   namespace: ktest
   spec:
   template:
   spec:
     containers:
       - image: gcr.io/knative-samples/helloworld-go
         ports:
           - containerPort: 8080
         env:
           - name: TARGET
             value: "World"

5. kubectl port-forward --namespace kmetrics deployment/otel-collector 8889

Additional Info

Install information:

• Platform (GKE, IKS, AKS, etc.): GKE
• Knative Version: 1.6

[abrennan89]

@skonto can you provide some help here to figure out if this is possible and whether there's anything we can add to docs?

@AlexandreDelforgeAdeo FYI I believe that questions like this should be first directed to the @knative/serving-observability-reviewers @knative/serving-wg-leads groups before docs, to see if it's possible to work out the issue and provide details before requesting docs.

[abrennan89]

Not clear if there is any actual docs requirement here. Please direct questions about this to the Knative Slack instance.

[dprotaso]

You need to update metrics.opencensus-address to point to the place where you deployed your otel collector.

otel-collector.${namespace}:55678