otel: make sure everything existing actually WAI

TBD54566975 / ftl

FTL - Towards a 𝝺-calculus for large-scale systems

https://tbd54566975.github.io/ftl/

Apache License 2.0

20 stars 7 forks source link

otel: make sure everything existing actually WAI #2021

Closed deniseli closed 1 month ago

deniseli commented 2 months ago

Code Links:

otelInterceptor: internal/rpc/context.go#L83
otel client: internal/observability/client.go#L36-L75
templates/ftl-controller.yml#L50 - Exporter is how we send data to otel (docs)

Out of scope: this logging interface we provide through the Go runtime SDK that no one currently uses: go-runtime/ftl/observability/metrics.go

Datadog:

go/tbddd - all TBD services
FTL Dashboard
Controller logs
Example of a call

deniseli commented 1 month ago

Also part of this: make sure metrics + logs work as intended. Add a toy one, deploy to the dev instance, make some verb calls in the dev instance, and then check datadog to make sure it all pipes all the way through. That unblocks us adding metrics/logs/traces in general.

deniseli commented 1 month ago

Metrics are looking pretty solid now: https://app.datadoghq.com/metric/explorer?fromUser=false&graph_layout=split&start=1722446967244&end=1722533367244&paused=false#N4Ig7glgJg5gpgFxALlAGwIYE8D2BXJVEADxQEYAaELcqyKBAC1pEbghkcLIF8qo4AMwgA7CAgg4RKUAiwAHOChASAtnADOcAE4RNIKtrgBHPJoQaUAbVBGN8qVoD6gnNtUZCKiOq279VKY6epbINiAiGOrKQdpYZAYgUJ4YThr42gDGSsgg6gi6mZaBZnHKGniqyIIIaAB0GBpYIplOmRho9aoaTgg4bTiq8miIcMAAVDwNPZn4IggAFACUIDwAulSu7niYoeFbqjsYMaXxqxsgGnIjMiDyHYgIOUk4MG27GhCZiWiicL0KZ7pX5QH5-Jz0JjKERuDxoVb8CD2TBYAGKZS-ERKdY8PiXYbiADCUmEMFuyPETigPjgIk+jmsoCk-xg2gw8kYTkU2mUNXqjWarXanTqbg4onOVF+qnEKAATAAWKjpbSEJnaAQ83ICDTfXE8IA

deniseli commented 1 month ago

Marking this as done now that the correctly parented traces are in prod and the new metrics+attrs are continuing to function as expected