search

DataDog / datadogpy

The Datadog Python library
https://datadoghq.com/
Other
609 stars 303 forks source link

Upgrading to DD 0.45.0 breaks StatsD metrics #764

Open joshhoegen opened 1 year ago

joshhoegen commented 1 year ago

Describe the bug After upgrading to 0.45.0 with Python 3.11, our metrics stopped reporting. It seems only the combo of Python 3.10 and datadogpy 0.44.0 fixed our issue. This repo's build from 3 weeks ago also has some failures (I'm not sure if this is related: https://dev.azure.com/datadoghq/datadogpy/_build/results?buildId=131198&view=logs&j=abc39910-4c32-5be0-0b39-6d3f303c7218&t=837b8d2b-2a00-5222-b449-dfcd3b59faed&l=825)

To Reproduce Steps to reproduce the behavior:

  1. Upgrade Python to 3.11
  2. Upgrade Datadog to 0.45.0
  3. In my case I ran a Fast API app with Gunicorn, using uvicorn for workers
  4. No errors, stats aren't sent to DD

Expected behavior Stats are sent to DD and visible in dasboard

Screenshots If applicable, add screenshots to help explain your problem.

Environment and Versions (please complete the following information): pyproject.toml (poetry)

[tool.poetry]
name = "article-analyzer-api"
version = "0.1.0"
description = ""
authors = []

[tool.poetry.dependencies]
python = "^3.10"
psycopg2-binary = "*"
boto3 = '*'
fastapi = "*"
uvicorn = "*"
datadog = "0.45.0"
setproctitle = "^1.3.2"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

[tool.poetry.scripts]
start = "article_analyzer_api.app:start"

Docker image

python:3.11-slim-buster

Usage

from datadog import initialize, statsd

            dd_options = {
                'statsd_port': 8125,
                'statsd_host': 'statsd',
                'statsd_namespace': 'my_namespace'
            }

            initialize(**dd_options)
            self.ddog = statsd

            ######

            statsd.increment('my_val')

Additional context

sgnn7 commented 1 year ago

@joshhoegen Can you provide some more info on your problem? It's not clear what the problem area may be from the description.

joshhoegen commented 1 year ago

@sgnn7, I've updated the issue above.

sgnn7 commented 1 year ago

@joshhoegen Thanks for the info! I have a few followups:

I don't have access to the dd-agent, is there somewhere I can find it in

You might be able to find it in the infrastructure list but it depends on your setup topology. The easiest way to find it is in the first log line that the agent emits (and stores in agent.log) when it initializes. It's also available in the agent pod with agent version command.

Have you tried using Python 3.10 with 0.45 to eliminate Python version change as the culprit?

I'm not quite clear if you meant that it was ok or not. From this matrix below, what was tested and what worked / didn't work (I filled in the things that I think you mentioned)?

Some other long-shot questions:

sgnn7 commented 1 year ago

@joshhoegen / @mark-a-dev I did a simple test using the same invocation code (without Poetry) with virtualenv on Ubuntu 22.04 and Agent 7.43.1. All combinations of Python 3.10/3.11 and Datadogpy 0.44.0/0.45.0 seem to send the data fine so we will for sure need more info on this ticket to proceed.

github-actions[bot] commented 1 year ago

Thanks for your contribution!

This issue has been automatically marked as stale because it has not had activity in the last 30 days. Note that the issue will not be automatically closed, but this notification will remind us to investigate why there's been inactivity. Thank you for participating in the Datadog open source community.

If you would like this issue to remain open:

  1. Verify that you can still reproduce the issue in the latest version of this project.

  2. Comment that the issue is still reproducible and include updated details requested in the issue template.

aexvir commented 1 year ago

@sgnn7

@joshhoegen / @mark-a-dev I did a simple test using the same invocation code (without Poetry) with virtualenv on Ubuntu 22.04 and Agent 7.43.1. All combinations of Python 3.10/3.11 and Datadogpy 0.44.0/0.45.0 seem to send the data fine so we will for sure need more info on this ticket to proceed.

we've hit the same issue with a standalone datadog/dogstatsd:7.35.2 deployment could you maybe test that version? our agent shows a bunch of errors like

2023-05-26 10:29:40 UTC | DSD | ERROR | (pkg/dogstatsd/server.go:592 in errLog) | Dogstatsd: error parsing metric message '"lynx.rule.cache:********@0.25|#state:hit,environment:production,color:purple|c:e98432af157746286894eb9fdc9d6e7294cd68e4413b20a06825d85563948830"': invalid dogstatsd message format
john-bodley commented 1 year ago

We've already run into the same issue, i.e., statsD metrics not being reported in Datadog, using Python 3.9 with datadog version 0.45.0. Downgrading to version 0.44.0 remedies the issue.

randori-ddejohn commented 9 months ago

@john-bodley @sgnn7 @aexvir @joshhoegen

0.45.0 added container IDs to metrics -- look into whatever you're using to consume your metrics as it may not support that new datagram format: https://docs.datadoghq.com/developers/dogstatsd/datagram_shell/?tab=metrics#dogstatsd-protocol-v12

Added in #720

asottile-sentry commented 7 months ago

we hit the same thing -- it's the new container metrics. there's no easy way to turn it off in code only an environment variable

can you add a similar opt-out to the telemetry opt-out?

for instance I can do:

statsd.disable_telemetry()

it would be nice to also be able to

statsd.disable_origin_detection()

especially because it silently breaks metric delivery when enabled