DataDog / dd-trace-py

Datadog Python APM Client
https://ddtrace.readthedocs.io/
Other
552 stars 415 forks source link

AioMysql integration span is mangled into service used in `@tracer.wrap` #8151

Closed kuchichan closed 6 months ago

kuchichan commented 10 months ago

Summary of problem

Aiomysql does not appear as an external service in the datadog dashboard (flamegraph and other views) like any other integration, when called within wrapped (@trace.wrap) function, but (for instance) redis does.

Which version of dd-trace-py are you using?

ddtrace 2.4.0

Which version of pip are you using?

pip 23.3.1

Which libraries and their versions are you using?

`pip freeze`
aiomysql==0.1.1
attrs==23.2.0
bytecode==0.15.1
cattrs==23.2.3
ddsketch==2.0.4
ddtrace==2.4.0
Deprecated==1.2.14
envier==0.5.0
greenlet==3.0.3
importlib-metadata==6.11.0
mypy==1.8.0
mypy-extensions==1.0.0
opentelemetry-api==1.22.0
protobuf==4.25.2
PyMySQL==1.1.0
redis==4.6.0
six==1.16.0
SQLAlchemy==2.0.25
typing_extensions==4.9.0
wrapt==1.16.0
xmltodict==0.13.0
zipp==3.17.0

How can we reproduce your problem?

Please consider the following (somewhat minimum) example:

import ddtrace.auto
import asyncio
import ddtrace
from sqlalchemy import Column
from sqlalchemy import MetaData
from sqlalchemy import select
from sqlalchemy import String
from sqlalchemy import Table
from sqlalchemy.ext.asyncio import create_async_engine
from redis.asyncio import ConnectionPool, Redis

connection_pool = ConnectionPool.from_url(url="redis://localhost:6380")
redis = Redis(connection_pool=connection_pool)

meta = MetaData()
t1 = Table("t1", meta, Column("name", String(50), primary_key=True))

async def propagate_data():
    engine = create_async_engine(
        "mysql+aiomysql://root:root@localhost:3308/dd",
        echo=True,
    )

    async with engine.begin() as conn:
        await conn.run_sync(meta.create_all)

        await conn.execute(
            t1.insert().prefix_with("IGNORE"), [{"name": "some name 1"}, {"name": "some name 2"}]
        )

    await engine.dispose()

@ddtrace.tracer.wrap(
    name="fetch-t1",
    span_type="sql",
    service="dummy-service",
)
async def fetch_t1():
    engine = create_async_engine(
        "mysql+aiomysql://root:root@localhost:3308/dd",
        echo=True,
    )
    async with engine.connect() as conn:
        result = await conn.execute(select(t1).where(t1.c.name == "some name 1"))

        print(result.fetchall())

    await engine.dispose()
    await redis.set("hello", 1)

asyncio.run(fetch_t1())

Corresponding docker-compose.yml

services:
  db:
    restart: "no"
    image: mysql:8.0
    container_name: dd-db 
    command: --default-authentication-plugin=mysql_native_password
    ports:
      - "3308:3306"
    environment:
      - MYSQL_DATABASE=dd
      - MYSQL_ROOT_PASSWORD=root
    volumes:
      - "./docker-entrypoint-initdb.d:/docker-entrypoint-initdb.d"
    cap_add:
      - SYS_NICE

  datadog:
    ports:
      - "8126:8126"
    image: datadog/agent
    pid: host
    environment:
      - DD_API_KEY=${DD_API_KEY}
      - DD_SITE=datadoghq.com
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /proc/:/host/proc/:ro
      - /sys/fs/cgroup:/host/sys/fs/cgroup:ro

  redis:
    restart: "no"
    image: redis:6.2.7
    ports:
      - "6380:6379"

If you want to propagate initial data to the db, just switch, fetch_t1 inside asyncio.run to propagate_data.

To reproduce, simply run this python script, and observe the flamegraph for dummy-service trace in the datadog APM dashboard.

What is the result that you get?

image

As you can see, sql query internals are mangled into dummy-service, but redis SET command is displayed as a separate service (which is cool thing, because it's an external library / client which I do not control directly).

What is the result that you expected?

I expected, that aiomysql integration behaviour will be identical as the redis integration. In many cases aiomysql Connection object is not used directly, but rather via some upper abstraction layer mechanisms like sessions, transactions and similar. These upper abstraction layers are later used in the (let's say) application / use-cases code, which indeed leverage @tracer.wrap a lot, so it's really hard to avoid the situation, when aiomysql span is not a child span.

ZStriker19 commented 9 months ago

Hi @kuchichan thanks for reaching out, we'll look into this.

github-actions[bot] commented 6 months ago

This issue has been automatically closed after a period of inactivity. If it's a feature request, it has been added to the maintainers' internal backlog and will be included in an upcoming round of feature prioritization. Please comment or reopen if you think this issue was closed in error.