Closed youssefNM closed 2 years ago
Thank you @youssefNM for the report. As this does not seem to be a problem with the profiler code itself but seems to be related to the infrastructure, could you open a support ticket https://www.datadoghq.com/support/ ?
We'd probably need more info, such as where it's come from and some timestamps to see how/if it matches on our side.
Thank you!
I had the same issue while running a test in a AWS Cloud9 instance (should be easy for you to reproduce). Is this because the server is unreacheable (port closed ?) ?
Tell me if you need more specific data.
@Anto59290 what do you mean by the server is unreachable? Our servers are reachable, but they might not be from Cloud9 — I wouldn't know about that. Feel free to open a support ticket if you need help setting this up.
Our servers are reachable, but they might not be from Cloud9 — I wouldn't know about that.
That's what I meant, yes.
Feel free to open a support ticket if you need help setting this up.
IMHO, it is a bit of a shame that the documentation does not provide more details about this. Which port should be open, etc. I know that on other Datadog features more details are given on port/network configuration. I am in trial period, with only a few days left, and I know that with the support it takes a while to get a proper answer, so I think I'll just give up on this one ;)
Thanks for your quick answer by the way.
Port 443 to intake.profile.datadoghq.com
or intake.profile.datadoghq.eu
depending on your site.
Port 443 to
intake.profile.datadoghq.com
orintake.profile.datadoghq.eu
depending on your site.
I've sent a ticket about this already, but it seems that the Python setup docs for profiling are missing the mention about DD_SITE
. Took me some time to realise that I forgot to set it to the EU host.
Thanks @kjagiello that was indeed my issue. Which lead to the "Unable to export" error (with a 403 error status, not with a 502 like in the initial issue, my bad). If you simply follow the documentation from https://docs.datadoghq.com/fr/tracing/profiling/?tab=python and your account is configured in EU (that is my case), it will not work. After export DD_SITE=datadoghq.eu
everything went smoothly. The Java documentation does state it by the way:
Note: With dd-java-agent.jar library versions 0.48+, if your organization is on Datadog EU site, add -Ddd.site=datadoghq.eu or set DD_SITE=datadoghq.eu as environment variable.
This is still hitting us, and no useful response from the support team @jd
{"asctime": "2020-05-13 02:01:35,672", "levelname": "ERROR", "name": "ddtrace.profiling.scheduler", "filename": "scheduler.py", "lineno": 42, "dd.trace_id": 0, "dd.span_id": 0, "message": "Unable to export 18026 events: ddtrace.profiling.exporter.http.UploadFailed: Unable to upload: urllib.error.HTTPError: HTTP Error 502: Bad Gateway\n\n", "dd.version": "", "dd.env": "", "dd.service": ""}
It seems to me that Datadog api is rejecting our profiling events maybe due to a rate limit in place (like the one you have with APM traces)
@youssefNM I'll check but I don't think our intake returns a 502 error like that in any case.
Do you have anything between the intake endpoint and your application by any chance? A proxy?
no, no proxy @jd, our application communicates with the intake endpoint directly. I got a response from the support team about this
Having that retry logic can help with this issue.
I had the same problem
1) Configuration was missing in my datadog-agent
DD_PROFILING_ENABLED: "true"
2) Missing name of the service of my datadog-agent for ddtrace-run
DD_PROFILING_ENABLED=true DD_SERVICE=my_dd_service_name ddtrace-run myprogram
Going to close this out due to age, @youssefNM please re-open if the issue is still occurring! Retry logic has been implemented since this issue was opened.
Which version of dd-trace-py are you using?
ddtrace[profile]==0.36.1
Which version of the libraries are you using?
How can we reproduce your problem?
What is the result that you get?
I get this error occasionally, it seems profiling failed to export the events, though this does not apply for all, since i can see new events from my app in profiling page.