AWS Distro for OpenTelemetry EKS add-on for kubernetes 1.30?

[rgl]

Describe the question

EKS is now supporting and suggesting upgrades to Kubernetes 1.30, but the AWS Distro for OpenTelemetry EKS add-on is not supported.

Any idea/milestone when it will available?

Steps to reproduce if your question is related to an action

Launch an EKS instance with Kubernetes 1.30, then try to add the add-on, you'll not be able due to:

What did you expect to see?

I expected to use the add-on with EKS 1.30.

[clementlecorre]

No news?

[catomatik]

Has anyone found the reason why the add-on is not supported? Btw, if it was installed, it remains functional with version 1.30, but you cannot manage it as an EKS add-on anymore.

[gavinjohnson-crowdcube]

Just bumped into this one yesterday when trying to install it in a 1.30 cluster.

[sonalita]

I have also hit this issue when trying to upgrade addons to 1.30

[sonalita]

See also https://github.com/aws/containers-roadmap/issues/2365

[vsakaram]

just a quick note, we plan to address this soon and we will keep this ticket updated

[jnicholls]

Thank you @vsakaram !

[zmully]

@vsakaram any update on when 1.30 support will be available?

It's been 75 days since EKS 1.30 was released, and I'm sure everyone watching this ticket certainly didn't expect it to take this long. Your 0.92 release trailed EKS 1.29 by 22 days, which still isn't great, but this is getting out of hand. With the next EKS release (1.31) expected in the next couple months, the ADOT release cadence really needs attention. This has hamstrung our ability to track the most recent EKS releases, and is actively undermining our trust in AWS's ability to provide a robust ecosystem of services and tooling.

[sidick]

@vsakaram any update on when 1.30 support will be available?

It's been 75 days since EKS 1.30 was released, and I'm sure everyone watching this ticket certainly didn't expect it to take this long. Your 0.92 release trailed EKS 1.29 by 22 days, which still isn't great, but this is getting out of hand. With the next EKS release (1.31) expected in the next couple months, the ADOT release cadence really needs attention. This has hamstrung our ability to track the most recent EKS releases, and is actively undermining our trust in AWS's ability to provide a robust ecosystem of services and tooling.

I agree, this is making me reconsider using the EKS addon and moving to self managed helm charts as it seems they're maintained better and at least aren't tied to particular EKS versions

[clementlecorre]

I used an alternative solution to avoid using the Opentelemetry add-on by directly utilizing the CloudWatch Agent (https://docs.aws.amazon.com/eks/latest/userguide/amazon-cloudwatch-observability.html), which also allows pushing traces to X-Trace.

I added the annotation for automatic configuration on my Java applications: https://opentelemetry.io/docs/kubernetes/operator/automatic/

Annotations:
    instrumentation.opentelemetry.io/inject-java: true

This injects the java agent directly as well as the configuration of the OTEL_ environment variables:

    OTEL_AWS_APP_SIGNALS_ENABLED:                    true
    OTEL_AWS_APPLICATION_SIGNALS_ENABLED:            true
    OTEL_TRACES_SAMPLER_ARG:                         endpoint=http://cloudwatch-agent.amazon-cloudwatch:2000
    OTEL_TRACES_SAMPLER:                             xray
    OTEL_EXPORTER_OTLP_PROTOCOL:                     http/protobuf
    OTEL_EXPORTER_OTLP_TRACES_ENDPOINT:              http://cloudwatch-agent.amazon-cloudwatch:4316/v1/traces
    OTEL_AWS_APP_SIGNALS_EXPORTER_ENDPOINT:          http://cloudwatch-agent.amazon-cloudwatch:4316/v1/metrics
    OTEL_AWS_APPLICATION_SIGNALS_EXPORTER_ENDPOINT:  http://cloudwatch-agent.amazon-cloudwatch:4316/v1/metrics
    OTEL_METRICS_EXPORTER:                           none

....

I used an alternative solution while waiting for support for this add-on

[janrito]

@clementlecorre i don't think this works with fargate

[zmully]

@vsakaram I'm going to continue to beat this drum, if this wasn't already ridiculous, your teams continued inability to package and release Addon support for EKS 1.30 is eroding customer trust faster than I think you understand.

It is now almost 3 months since EKS 1.30 was released on May 23, 2024. Your team released v0.40.0 with support for Kubernetes 1.30 on June 12 (https://github.com/aws-observability/aws-otel-collector/commit/ba62ce386c3ce9c457fb456ed71e5e151dfc507c), but you've still not packaged v0.40.0 as an EKS Addon.

Quite frankly I'm a little confused by what the AWS Observability teams priorities are here. You clearly made the time to release support for Kubernetes 1.30 only 20 days after official EKS support for it was released, but to this day, those of us following the ADOT documentation and the recommended Addon method of installation for ADOT are left unable to upgrade our clusters, and are having to spend time finding other methods to install ADOT collectors.

If you all are giving up on the Addon installation method, fine, just say that and provide documentation on how it should be installed for EKS as a non-Addon. It would also be helpful if you could provide how the Addon was packaged, so that those of us trying to install it outside of the Addon have the best chance of replicating it successfully.

[dejwsz]

we cannot upgrade as we need this addon in 1.30, this is absolutely ridiculous

[sidick]

we cannot upgrade as we need this addon in 1.30, this is absolutely ridiculous

Don't worry, 1.31 is due soon...

[gavinjohnson-crowdcube]

It'd be comedy gold if we didn't actually need it. Half the EKS team have probably been pulled onto some pointless AmazonQ feature. 😂

[nikolaianohyn]

Any updates?

[mhausenblas]

ADOT PM here. Apologies for the delays, we ran into some issue with upstream. We're working on the release, ensuring that all tests pass. At the current time, we expect that the support for 1.30 will be rolled out into all supported regions by end of August, latest. I will keep this thread posted on the status.

[sidick]

ADOT PM here. Apologies for the delays, we ran into some issue with upstream. We're working on the release, ensuring that all tests pass. At the current time, we expect that the support for 1.30 will be rolled out into all supported regions by end of August, latest. I will keep this thread posted on the status.

I don't mind delays with good reasons, the problem has mostly been lack of info, sounds promising going forward

[mhausenblas]

PSA: support for EKS 1.30 has landed now (https://aws-otel.github.io/docs/ReleaseBlogs/aws-distro-for-opentelemetry-eks-add-on-v0.102.0). Thank you for your patience and we're reviewing the learnings to improve the process, going forward.