flyteorg / flyte

Scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks.
https://flyte.org
Apache License 2.0
5.17k stars 550 forks source link

Add OTLP and sampling to otelutils #5504

Closed andrewwdye closed 3 days ago

andrewwdye commented 6 days ago

Why are the changes needed?

This changes adds support for OLTP and sampling to the otelutils tracer provider abstraction.

Adds support for OLTP. This is the recommended replacement for the deprecated jaeger exporter.

"go.opentelemetry.io/otel/exporters/jaeger" is deprecated: This module is no longer supported. OpenTelemetry dropped support for Jaeger exporter in July 2023. Jaeger officially accepts and recommends using OTLP

OLTP supports grpc and http, which are added as separate exporter types and configs.

Adds initial sampling support to the top level open telemetry config. Defaults to parent sampler always, but also adds a config for TraceIdRatioBased. See these docs for behavior of parent sampler.

How was this patch tested?

Ran local sandbox with local jaeger all-in-one and verified

Flyte config

❯ docker run --rm -e COLLECTOR_OTLP_ENABLED=true -p 16686:16686 -p 4317:4317 -p 4318:4318 jaegertracing/all-in-one:1.52

Jaeger

otel:
  type: otlpgrpc
  sampler:
    parentSampler: traceid
codecov[bot] commented 6 days ago

Codecov Report

Attention: Patch coverage is 45.94595% with 20 lines in your changes missing coverage. Please review.

Project coverage is 60.99%. Comparing base (2334d3f) to head (d504995). Report is 2 commits behind head on master.

Files Patch % Lines
flytestdlib/otelutils/factory.go 39.39% 20 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #5504 +/- ## ========================================== - Coverage 61.01% 60.99% -0.02% ========================================== Files 794 794 Lines 51441 51470 +29 ========================================== + Hits 31385 31395 +10 - Misses 17164 17183 +19 Partials 2892 2892 ``` | [Flag](https://app.codecov.io/gh/flyteorg/flyte/pull/5504/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flyteorg) | Coverage Δ | | |---|---|---| | [unittests-datacatalog](https://app.codecov.io/gh/flyteorg/flyte/pull/5504/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flyteorg) | `69.31% <ø> (ø)` | | | [unittests-flyteadmin](https://app.codecov.io/gh/flyteorg/flyte/pull/5504/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flyteorg) | `58.73% <ø> (ø)` | | | [unittests-flytecopilot](https://app.codecov.io/gh/flyteorg/flyte/pull/5504/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flyteorg) | `17.79% <ø> (ø)` | | | [unittests-flytectl](https://app.codecov.io/gh/flyteorg/flyte/pull/5504/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flyteorg) | `68.03% <ø> (ø)` | | | [unittests-flyteidl](https://app.codecov.io/gh/flyteorg/flyte/pull/5504/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flyteorg) | `79.04% <ø> (ø)` | | | [unittests-flyteplugins](https://app.codecov.io/gh/flyteorg/flyte/pull/5504/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flyteorg) | `61.85% <ø> (ø)` | | | [unittests-flytepropeller](https://app.codecov.io/gh/flyteorg/flyte/pull/5504/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flyteorg) | `57.30% <ø> (ø)` | | | [unittests-flytestdlib](https://app.codecov.io/gh/flyteorg/flyte/pull/5504/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flyteorg) | `65.59% <45.94%> (-0.23%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flyteorg#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.