search

DataDog / nginx-datadog

Enhance NGINX Observability and Security with Datadog's Module
https://www.datadoghq.com
Apache License 2.0
22 stars 9 forks source link

Getting "Could not parse Remote Configuration response body" on AL2023 #61

Closed gugaiz closed 6 months ago

gugaiz commented 6 months ago

I built the nginx module using the changes on this PR for the image amazonlinux_2023.3.20240219.0. The build finished without issues but when running the module I am getting

2024/02/23 15:04:46 [error] 14490#14493: datadog: Could not parse Remote Configuration response body

I got the image from here because it is not available on Docker hub

Any ideas?

dgoffredo commented 6 months ago

I vaguely recall this happening when the Datadog Agent returns 200 OK with a body that is not JSON, and is instead a plaintext error message about some unexpected gRPC error status received from Datadog's intake.

The log message in the tracer is coming from here. The message would be more helpful if it printed (an excerpt of) the response body.

Since you are building from source, perhaps you can confirm whether the error message is as I suspect. In nginx-datadog, dd-trace-cpp is a git submodule. Point it at a branch that has the following change: the error message here should print the response body, as is done here.

If it's the error that I think it is, then we can find a solution... ah, wait. I think that @dmehala fixed this in https://github.com/DataDog/dd-trace-cpp/pull/95. Perhaps all we need to do is release those changes (in dd-trace-cpp), update the submodule in nginx-datadog, and it will be fixed.

For now, please try building nginx-datadog with the dd-trace-cpp submodule pointed at the current main branch of dd-trace-cpp. See if that fixes the error message.

dgoffredo commented 6 months ago

I plan on cutting a new release today, so you'll just be able to use that.

dgoffredo commented 6 months ago

Try today's new release: https://github.com/DataDog/nginx-datadog/releases/tag/v1.0.5

gugaiz commented 6 months ago

Hi @dgoffredo thanks for your answers. Last week I was off and I didn't have the chance to test it.

I can confirm that the new release fixed the issue. I am using the new module on the latest version of AWS Elastic Beanstalk running amazonlinux:2023.3.20240219.0 using the changes suggested on this PR

dgoffredo commented 6 months ago

I'll take a look at your Amazon Linux PR again today. Pardon the delay, the team is strapped.