search

microsoft / azure-container-apps

Roadmap and issues for Azure Container Apps
MIT License
370 stars 29 forks source link

Feature Request: Expose OpenTelemetry Collector configuration #208

Open denniszielke opened 2 years ago

denniszielke commented 2 years ago

Is your feature request related to a problem? Please describe.
I want to use my own observability solution and not azure monitor. To connect I need the ability to configure my own otel processors and exporters.

Describe the solution you'd like.
I would like the ability to configure receivers, processors, (extensions?), the sampling rate and exporters.

Describe alternatives you've considered.
The alternative would be to run my own otel collector as ACA service and manually change all other ACA apps to use them. This does not seem like a scalable good idea.

Additional context.

kendallroden commented 2 years ago

Thanks for this. It is a consideration moving forward definitely

JennyLawrance commented 2 years ago

@denniszielke, have you considered OTEL as a side car option?

denniszielke commented 2 years ago

@JennyLawrance yes that is what we are doing today but on the long run this does not scale (we need to maintain the same config , we need to make sure every dev creates the right config for each microservice and this is not what a PaaS should enforce). Also the performance overhead is significant as long ACA is limited to 4GB per replica (app+sidecar)

edeandrea commented 2 years ago

Adding onto this (& tagging @agoncal as well) - If I wanted to run an OTel collector instance as its own containerapp its currently very difficult to inject the configuration, since ContainerApps does not expose a ConfigMap that can be mounted as a volume.

For example, if I wanted to run the Otel collector on kubernetes, I could just deploy this:

apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-config
  labels:
    app: otel-collector
    role: monitoring
data:
  otel-collector-config.yml: |2
    receivers:
      otlp:
        protocols:
          grpc:

    exporters:
      jaeger:
        endpoint: jaeger:14250
        tls:
          insecure: true

    processors:
      batch:

    extensions:
      health_check:

    service:
      extensions:
        - health_check
      pipelines:
        traces:
          receivers:
            - otlp
          processors:
            - batch
          exporters:
            - jaeger
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
  labels:
    name: otel-collector
    app: otel-collector
    role: monitoring
    app.kubernetes.io/part-of: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      name: otel-collector
  template:
    metadata:
      labels:
        name: otel-collector
        app: otel-collector
        role: monitoring
    spec:
      containers:
        - image: otel/opentelemetry-collector:0.53.0
          name: otel-collector
          args:
            - "--config=/conf/otel-collector-config.yml"
          ports:
            - name: health-check
              containerPort: 13133
            - name: otlp-receiver
              containerPort: 4317
          volumeMounts:
            - name: otel-collector-config
              mountPath: /conf
      volumes:
        - name: otel-collector-config
          configMap:
            name: otel-collector-config

I haven't yet found a good way to do this in Azure ContainerApps. I found https://docs.microsoft.com/en-us/azure/container-apps/storage-mounts?pivots=aca-cli#azure-files but that means I have to maintain a separate yaml file specific to azure containerapps, which contains most of the same fields as a kubernetes yaml. That doesn't seem like a scalable solution, especially if all my CI/CD automation uses the az cli.

alhardy commented 1 year ago

This approach works https://www.honeycomb.io/blog/opentelemetry-collector-azure-container-apps

Depechie commented 1 year ago

hey @alhardy I followed that guide, it works, but the effort to do it is huge and needs manual intervention ( downloading config and editing ). So we would hope the team can better the experience.

I even needed to do 'more' and detailed it here https://blog.depechie.com/posts/2022-10-13-opentelemetry-on-azure-container-apps/

edeandrea commented 1 year ago

Thank you @alhardy ! That still seems way more complicated than it should!

denniszielke commented 1 year ago

Agree.

Depechie commented 1 year ago

I found out that if you do not use the az cli but instead opt to deploy to azure using Bicep the volume mount does work! Meaning I can now deploy OpenTelemetry Collector fully automated from CI/CD without manually needing to download the config file after creation...

Example part of the Bicep for the container app ( almost the same as the kubernetes one from @edeandrea 

resource containerAppOtel 'Microsoft.App/containerApps@2022-03-01' = {
  name: 'ca-${env}-${resourceLocationShort}-otel'
  location: resourceLocationLong
  properties: {
    managedEnvironmentId: environment.id
    configuration: {
      ingress: {
        external: true
        targetPort: 4318
      }
    }
    template: {
      containers: [
        {
          image: 'otel/opentelemetry-collector-contrib:latest'
          name: 'ca-${env}-${resourceLocationShort}-otel'
          args: [
            '--config=/etc/otel-config-test.yml'
          ]
          resources: {
            cpu: json('0.5')
            memory: '1.0Gi'
          }
          volumeMounts: [
            {
              mountPath: '/etc'
              volumeName: 'config'
            }
          ]
        }
      ]
      scale: {
        minReplicas: 1
        maxReplicas: 1
      }
      volumes: [
        {
          name: 'config'
          storageName: environment::azurefilestorage.name
          storageType: 'AzureFile'
        }
      ]
    }
  }
}
dickiebowuk commented 1 year ago

I'm looking for some assistance. I started with the Honeycomb article to build a collector in an Azure Container App and it wasn't starting correctly until I found this post, pointing to the second article and added the args section to point to my mounted config file. The collector is running and listening for otlp http traffic on port 4318 with the ingress set to allow external traffic. However, I cannot get it to receive any data from my .Net Core 7 api. I can get it all working on my laptop with the otel collector contrib running in docker desktop set for http on port 4318.

Here is the config file for the container:

receivers:
  otlp:
    protocols:
      http:

exporters:
  otlphttp/withauth:
    auth:
      authenticator: oauth2client
    logs_endpoint: $CNAO_LOGS_ENDPOINT
    metrics_endpoint: $CNAO_METRICS_ENDPOINT
    traces_endpoint: $CNAO_TRACES_ENDPOINT

  logging:
    loglevel: debug

processors:
  batch:
    send_batch_size: 8192
    timeout: 10s
  attributes/collector_info:
    actions:
      - key: collector.hostname
        value: $HOSTNAME
        action: insert
      - key: azure.container_app.revision
        value: $CONTAINER_APP_REVISION
        action: insert
      - key: azure.container_app.name
        value: $CONTAINER_APP_NAME
        action: insert
      - key: source.blog
        value: "true"
        action: insert
  filter/healthcheck:
    spans:
      exclude:
        match_type: strict
        attributes:
          - Key: http.target
            Value: /health

extensions:
  oauth2client:
    client_id: $CNAO_CLIENT_ID
    client_secret: $CNAO_CLIENT_SECRET
    token_url: $CNAO_TOKEN_URL

service:
  extensions: [oauth2client]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch,filter/healthcheck,attributes/collector_info]
      exporters: [logging, otlphttp/withauth]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/withauth]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/withauth]
  telemetry:
    logs:
      level: "debug"

The log stream on the container shows that it has started successfully giving this output:

2023-10-16T10:41:15.15488  Connecting to the container 'collector'...

2023-10-16T10:41:15.17288  Successfully Connected to container: 'collector' [Revision: 'collector--1', Replica: 'collector--1']
2023-10-13T15:37:36.605516952Z 2023-10-13T15:37:36.605Z info    service/pipelines.go:92 Processor is starting...    {"kind": "processor", "name": "batch", "pipeline": "logs"}
2023-10-13T15:37:36.605528072Z 2023-10-13T15:37:36.605Z info    service/pipelines.go:96 Processor started.  {"kind": "processor", "name": "batch", "pipeline": "logs"}
2023-10-13T15:37:36.605566835Z 2023-10-13T15:37:36.605Z info    service/pipelines.go:92 Processor is starting...    {"kind": "processor", "name": "attributes/collector_info", "pipeline": "traces"}
2023-10-13T15:37:36.605573347Z 2023-10-13T15:37:36.605Z info    service/pipelines.go:96 Processor started.  {"kind": "processor", "name": "attributes/collector_info", "pipeline": "traces"}
2023-10-13T15:37:36.605622865Z 2023-10-13T15:37:36.605Z info    service/pipelines.go:92 Processor is starting...    {"kind": "processor", "name": "filter/healthcheck", "pipeline": "traces"}
2023-10-13T15:37:36.605652090Z 2023-10-13T15:37:36.605Z info    service/pipelines.go:96 Processor started.  {"kind": "processor", "name": "filter/healthcheck", "pipeline": "traces"}
2023-10-13T15:37:36.605657580Z 2023-10-13T15:37:36.605Z info    service/pipelines.go:92 Processor is starting...    {"kind": "processor", "name": "batch", "pipeline": "traces"}
2023-10-13T15:37:36.605698870Z 2023-10-13T15:37:36.605Z info    service/pipelines.go:96 Processor started.  {"kind": "processor", "name": "batch", "pipeline": "traces"}
2023-10-13T15:37:36.605707717Z 2023-10-13T15:37:36.605Z info    service/pipelines.go:92 Processor is starting...    {"kind": "processor", "name": "batch", "pipeline": "metrics"}
2023-10-13T15:37:36.605711765Z 2023-10-13T15:37:36.605Z info    service/pipelines.go:96 Processor started.  {"kind": "processor", "name": "batch", "pipeline": "metrics"}
2023-10-13T15:37:36.605715602Z 2023-10-13T15:37:36.605Z info    service/pipelines.go:100    Starting receivers...
2023-10-13T15:37:36.605736781Z 2023-10-13T15:37:36.605Z info    service/pipelines.go:104    Receiver is starting... {"kind": "receiver", "name": "otlp", "pipeline": "metrics"}
2023-10-13T15:37:36.605758522Z 2023-10-13T15:37:36.605Z warn    internal/warning.go:51  Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks    {"kind": "receiver", "name": "otlp", "pipeline": "metrics", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2023-10-13T15:37:36.605821510Z 2023-10-13T15:37:36.605Z info    otlpreceiver@v0.67.0/otlp.go:90 Starting HTTP server    {"kind": "receiver", "name": "otlp", "pipeline": "metrics", "endpoint": "0.0.0.0:4318"}
2023-10-13T15:37:36.605855915Z 2023-10-13T15:37:36.605Z info    service/pipelines.go:108    Receiver started.   {"kind": "receiver", "name": "otlp", "pipeline": "metrics"}
2023-10-13T15:37:36.605861705Z 2023-10-13T15:37:36.605Z info    service/pipelines.go:104    Receiver is starting... {"kind": "receiver", "name": "otlp", "pipeline": "logs"}
2023-10-13T15:37:36.605888225Z 2023-10-13T15:37:36.605Z info    service/pipelines.go:108    Receiver started.   {"kind": "receiver", "name": "otlp", "pipeline": "logs"}
2023-10-13T15:37:36.605897011Z 2023-10-13T15:37:36.605Z info    service/pipelines.go:104    Receiver is starting... {"kind": "receiver", "name": "otlp", "pipeline": "traces"}
2023-10-13T15:37:36.605920645Z 2023-10-13T15:37:36.605Z info    service/pipelines.go:108    Receiver started.   {"kind": "receiver", "name": "otlp", "pipeline": "traces"}
2023-10-13T15:37:36.605927208Z 2023-10-13T15:37:36.605Z info    service/service.go:105  Everything is ready. Begin running and processing data.

The code I am using for the otlp exporter in my .Net Core 7 project to hit the container app is as follows:

string endPoint = "https://collector.onazure.azurecontainerapps.io:4318/v1/traces";
                            Console.WriteLine($"AppD Endpoint: {endPoint}");
                            builder.AddOtlpExporter(options =>
                            {
                                AppContext.SetSwitch("System.Net.Http.SocketsHttpHandler.Http2UnencryptedSupport", true);

                                options.Protocol = OtlpExportProtocol.HttpProtobuf;
                                options.ExportProcessorType = ExportProcessorType.Batch;
                                options.Endpoint = new Uri(endPoint);
                            })
                            .AddConsoleExporter(options => options.Targets = ConsoleExporterOutputTargets.Console);

I have tried running the project locally on my laptop and also have the api in it's own container with an APIM in place to test the endpoints. I can see in the console logs that the Writeline is output when starting up the container and that telemetry is being sent to the Console through the Console Exporter. However, it is not being received by the collector.

EDIT: I sorted the issue. At present Azure Container Apps only open one port, so you cannot map multiple ports like you can with Docker. Port 80 and 443 are open by default and I thought 4318 had successfully been opened as it says it's available in the Container overview. I used Telnet to confirm which ports were open and found 4318 was not available, so I tried sending the data to port 443 and it worked. My new endpoint is:

https://collector.onazure.azurecontainerapps.io:433/v1/traces