THREESCALE-9539 opentelemetry support

[eguzki]

What

Implements https://issues.redhat.com/browse/THREESCALE-9539

APICast can be instrumented using the opentelemetry client https://github.com/3scale/APIcast/pull/1379

This PR enables opentelemetry instrumentation using APIcast CR

---
apiVersion: apps.3scale.net/v1alpha1
kind: APIcast
metadata:
  name: apicast1
spec:
  ...
  openTelemetry:
    enabled: true
    tracingConfigSecretRef:
      name: otel-config

The new NGINX instrumentation using OpenTelemetry deprecates the opentracing instrumentation. If opentracing instrumentation is used, warning log will be shown.

TODO

• [x] Doc
• [x] Test

Verification steps

• Setup dev environment

  make kind-create-cluster
  make install
  make run

• Deploy Jaeger

  make jaeger-deploy

• Create opentelemetry configuration secret

  k apply -f - <<EOF
  ---
  apiVersion: v1
  kind: Secret
  metadata:
  name: otel-config
  type: Opaque
  stringData:
  config.json: |
  exporter = "otlp"
  processor = "simple"
  [exporters.otlp]
  # Alternatively the OTEL_EXPORTER_OTLP_ENDPOINT environment variable can also be used.
  host = "jaeger"
  port = 4317
  # Optional: enable SSL, for endpoints that support it
  # use_ssl = true
  # Optional: set a filesystem path to a pem file to be used for SSL encryption
  # (when use_ssl = true)
  # ssl_cert_path = "/path/to/cert.pem"
  [processors.batch]
  max_queue_size = 2048
  schedule_delay_millis = 5000
  max_export_batch_size = 512
  [service]
  name = "apicast" # Opentelemetry resource name
  EOF

• Create apicast embedded configuration secret

  k apply -f - <<EOF
  ---
  apiVersion: v1
  kind: Secret
  metadata:
  name: apicast-config
  type: Opaque
  stringData:
  config.json: |
  {
    "services": [
      {
        "backend_version": "1",
        "proxy": {
            "hosts": ["one"],
            "api_backend": "http://httpbin.org",
            "backend": {
                "endpoint": "http://127.0.0.1:8081",
                "host": "backend"
            },
            "proxy_rules": [
                {
                    "http_method": "GET",
                    "pattern": "/",
                    "metric_system_name": "hits",
                    "delta": 1,
                    "parameters": [],
                    "querystring_parameters": {}
                }
            ]
        }
      }
    ]
  }
  EOF

• Deploy APIcast with tracing enabled

  k apply -f - <<EOF
  ---
  apiVersion: apps.3scale.net/v1alpha1
  kind: APIcast
  metadata:
  name: apicast1
  spec:
  resources: {}
  logLevel: debug
  deploymentEnvironment: staging
  configurationLoadMode: lazy
  cacheConfigurationSeconds: 0
  embeddedConfigurationSecretRef:
  name: apicast-config
  openTelemetry:
  enabled: true
  tracingConfigSecretRef:
    name: otel-config
  EOF

• Wait for APIcast to be Ready

  ❯ kubectl wait --for=condition=Ready apicast/apicast1
  apicast.apps.3scale.net/apicast1 condition met

• Send valid request to APIcast

  
  ❯ kubectl port-forward service/apicast-apicast1 18080:8080&

❯ curl -v -H "Host: one" http://127.0.0.1:18080/get?user_key=foo

• Trying 127.0.0.1:18080...

• Connected to 127.0.0.1 (127.0.0.1) port 18080 (#0)

  GET /get?user_key=foo HTTP/1.1 Host: one User-Agent: curl/7.81.0 Accept: /

  Handling connection for 18080

• Mark bundle as not supporting multiuse < HTTP/1.1 200 OK < Server: openresty < Date: Tue, 25 Apr 2023 22:00:09 GMT < Content-Type: application/json < Content-Length: 371 < Connection: keep-alive < Access-Control-Allow-Origin: < Access-Control-Allow-Credentials: true < { "args": { "user_key": "foo" }, "headers": { "Accept": "/*", "Host": "httpbin.org", "Traceparent": "00-6102746649707d528532de80a9f73f05-3c3e64a0a3ed6d1f-01", "User-Agent": "curl/7.81.0", "X-Amzn-Trace-Id": "Root=1-64484d68-31f41b6460a430d16201926c" }, "origin": "147.161.83.37", "url": "http://httpbin.org/get?user_key=foo" }

• Connection #0 to host 127.0.0.1 left intact

  
  Note that upstream echo'ed request headers show `Traceparent` W3C standard tracing header.

• Check Jaeger for traces being sent from the gateway

  
  ❯ kubectl port-forward service/jaeger 16686&  

❯ open http://127.0.0.1:16686

Hit "Find Traces" with `Service` set to `apicast`. There should be at lease one trace.