Open modosc opened 6 months ago
I believe we are seeing similar issues using the Datadog Agent as a sidecar on Fargate. The Datadog Agent exits with a 137 status code.
We are also using ruby dd-trace, hopefully that's helpful in narrowing down the issue.
I'm having this issue as well. I'm pinning the container to version datadog/agent:7.50.3
to avoid this for now.
edit: nevermind, we just saw another crash so the suggested fix did not help.
Our experience is that pinning the agent version keeps the ECS task from failing, but we can still see in the logs the issue between the containers:
Internal error during Datadog::Tracing::Transport::HTTP::Client request. Cause: Errno::ECONNREFUSED Failed to open TCP connection to localhost:8126
Not able to use the datadog/agent:latest
version of datadog-agent as sidecar agent in AWS EKS with Fargate node and container terminated immediately with the following error.
s6-overlay-preinit: fatal: unable to chown /var/run/s6: Operation not permitted
Currently using datadog/agent:7.50.3
version to avoid this issue for now.
We have the same problem with "latest" version running in AWS ECS Fargate sidecar container. Needed to use 7.50.3 to stop crashing our applications.
Hi @modosc @JoshuaSchlichting @LukaszBancarz @praveensudharsan @bjclark13
Thanks for creating and commenting this issue. However it seems that different issues are happening. Could you all contact our support with an agent flare and the ecs task definition, so we can better understand the different scenarios that lead to an agent crash with the 7.51.x release. then share with us the support ticket id.
@clamoriniere i've got 1566759 opened currently.
Support case 1582012 for me
It seems it connected with update of s6-overlay to v1.22.1.0
in a docker image.
in /etc/s6/init/init-stage1 there are additional block that fails
if { /bin/s6-overlay-preinit }
Any eta on a fix? Thanks
Agent Environment we're using
public.ecr.aws/datadog/agent:latest
in a sidecar container and deploying toecs
. we also have the following configuration setup via env variables:Describe what happened: this has worked fine for at least 6 months. since ~2024-02-18 we've started seeing these failures when attempting to deploy:
this causes our deploys to fail. re-running usually resolves this, although today these have happened more and more frequently.
here's a full log entry:
Describe what you expected: this shouldn't happen?
Steps to reproduce the issue: see above. we cannot 100% reliably trigger this.
Additional environment details (Operating System, Cloud provider, etc): aws linux also using the lambda log forwarder
is there more debugging we can enable on the
dd
side to understand what's going on?was a change to this docker image pushed out?