Revamp datadog observability

lahabana commented 2 years ago

Description

The datadog setup page is very incomplete.

There's an example of setting this to make metrics work:

        ad.datadoghq.com/kuma-sidecar.check_names: '["envoy"]'
        ad.datadoghq.com/kuma-sidecar.init_configs: '[{}]'
        ad.datadoghq.com/kuma-sidecar.tags: '{"mesh": "default"}'
        ad.datadoghq.com/kuma-sidecar.instances: '[{"openmetrics_endpoint": "http://%%host%%:5670/metrics"}]'

And having the set skipMTLS=true to have the datadog agent outside the mesh.

Also mention that users should start seeing “envoy.clusters.*” metrics inside the Datadog Metric Explorer once traffic is flowing.

Some extra context from a discussion:

yeah so there are two ways using dd native envoy integration or use regular prom scrape (which might treat as custom metrics and cost some extra $$$, need to double check). For envoy its 1) Enable traffic metrics in mesh (with skipMTLS is the agent is outside of mesh) 2) Tag workloads with the envoy integration, 3) enable enovy integration on the dd UI 4) metrics will popin and start to auto popluate the default envoy dashboard (we have a custom one). For prom scrape, its the usual 1) Enable traffic metrics in mesh 2) Tag workloads with porm scrape metrics (which will also trigger dd agent to scrape unless they have a custom annotation to look at) 3) The metric names are a slight different in prom ones so we need to build a custom dashboard ground. we uses our automation to auto tag namespaces for every teams so they get the metrics for free

Here's rough content to add to the docs in a nice way