search

DataDog / dd-trace-py

Datadog Python APM Client
https://ddtrace.readthedocs.io/
Other
543 stars 411 forks source link

What is the intended way to send logs to DataDog #7773

Closed Gitznik closed 10 months ago

Gitznik commented 10 months ago

Summary of problem

Hey everyone,

I'm trying to send logs from my fastapi application to DataDog. I've installed the datadog agent locally and activated log collection in the config. I then run my application using ddtrace-run.

The datadog information is correctly injected into the logs I see in stdout, but they never make it to datadog. Tracing works just fine, just the logs are missing.

I worked around this by logging to a file and have the agent tail that file or by sending the logs to via TCP and having a log collector for TCP setup in the agent. For this, I have to overwrite the SocketHandler, so it does not pickle the log. For either of these setups - the logs end up fine in datadog.

My question here: Is this the intended way? This feels awfully complicated, but I can't find a better way to set it up.

Which version of dd-trace-py are you using?

2.3.1

Which version of pip are you using?

23.3.1

Which libraries and their versions are you using?

annotated-types==0.6.0 anyio==3.7.1 anytree==2.12.0 appdirs==1.4.4 appnope==0.1.3 asgiref==3.7.2 asttokens==2.4.0 asyncpg==0.28.0 attrs==23.1.0 autoflake==2.2.1 black==23.10.1 bump-pydantic==0.7.0 bytecode==0.15.1 cachetools==5.3.2 cattrs==23.2.2 certifi==2023.7.22 cfgv==3.4.0 chardet==3.0.4 charset-normalizer==3.3.1 click==8.1.7 colorama==0.4.6 coverage==7.3.2 DateTime==5.2 ddsketch==2.0.4 ddtrace==2.3.1 decorator==5.1.1 Deprecated==1.2.14 deprecation==2.1.0 distlib==0.3.7 dnspython==2.4.2 eikon==1.1.18 email-validator==2.1.0.post1 envier==0.4.0 et-xmlfile==1.1.0 execnet==2.0.2 executing==2.0.0 Faker==19.12.0 fastapi==0.103.2 filelock==3.12.4 google-api-core==2.12.0 google-auth==2.23.3 google-auth-oauthlib==1.1.0 google-cloud-appengine-logging==1.3.2 google-cloud-audit-log==0.2.5 google-cloud-core==2.3.3 google-cloud-logging==3.8.0 googleapis-common-protos==1.61.0 greenlet==3.0.1 grpc-google-iam-v1==0.12.6 grpcio==1.59.0 grpcio-status==1.59.0 gspread==5.12.0 gunicorn==20.1.0 h11==0.14.0 h2==4.1.0 hpack==4.0.0 httpcore==0.16.3 httptools==0.6.1 httpx==0.23.3 hyperframe==6.0.1 icecream==2.1.3 identify==2.5.30 idna==2.10 importlib-metadata==6.8.0 inflect==7.0.0 iniconfig==2.0.0 ipython==8.17.2 isort==5.12.0 jedi==0.19.1 libcst==1.1.0 markdown-it-py==3.0.0 matplotlib-inline==0.1.6 mdurl==0.1.2 mypy-extensions==1.0.0 nest-asyncio==1.5.8 nodeenv==1.8.0 numpy==1.25.2 oauthlib==3.2.2 openpyxl==3.1.2 opentelemetry-api==1.21.0 opentelemetry-distro==0.42b0 opentelemetry-instrumentation==0.42b0 opentelemetry-instrumentation-asgi==0.42b0 opentelemetry-instrumentation-asyncpg==0.42b0 opentelemetry-instrumentation-aws-lambda==0.42b0 opentelemetry-instrumentation-dbapi==0.42b0 opentelemetry-instrumentation-fastapi==0.42b0 opentelemetry-instrumentation-grpc==0.42b0 opentelemetry-instrumentation-httpx==0.42b0 opentelemetry-instrumentation-logging==0.42b0 opentelemetry-instrumentation-requests==0.42b0 opentelemetry-instrumentation-sqlalchemy==0.42b0 opentelemetry-instrumentation-sqlite3==0.42b0 opentelemetry-instrumentation-system-metrics==0.42b0 opentelemetry-instrumentation-tortoiseorm==0.42b0 opentelemetry-instrumentation-urllib==0.42b0 opentelemetry-instrumentation-urllib3==0.42b0 opentelemetry-instrumentation-wsgi==0.42b0 opentelemetry-propagator-aws-xray==1.0.1 opentelemetry-sdk==1.21.0 opentelemetry-semantic-conventions==0.42b0 opentelemetry-util-http==0.42b0 packaging==23.2 pandas==1.5.0 parso==0.8.3 pathspec==0.11.2 pexpect==4.8.0 pickleshare==0.7.5 platformdirs==3.11.0 pluggy==1.3.0 pre-commit==3.5.0 prompt-toolkit==3.0.39 proto-plus==1.22.3 protobuf==4.24.4 psutil==5.9.6 psycopg2-binary==2.9.9 ptyprocess==0.7.0 pure-eval==0.2.2 pyasn1==0.5.0 pyasn1-modules==0.3.0 pydantic==2.4.2 pydantic-settings==2.0.3 pydantic_core==2.10.1 pyflakes==3.1.0 Pygments==2.16.1 pyright==1.1.333 pytest==7.4.3 pytest-asyncio==0.20.3 pytest-xdist==3.3.1 python-dateutil==2.8.2 python-dotenv==1.0.0 pytz==2023.3.post1 PyYAML==6.0.1 requests==2.31.0 requests-oauthlib==1.3.1 rfc3986==1.5.0 rich==13.6.0 rsa==4.9 ruff==0.0.289 six==1.16.0 sniffio==1.3.0 sqlacodegen==3.0.0rc3 SQLAlchemy==2.0.22 stack-data==0.6.3 starlette==0.27.0 traitlets==5.13.0 typer==0.9.0 typing-inspect==0.9.0 typing_extensions==4.8.0 urllib3==2.0.7 uvicorn==0.23.2 uvloop==0.19.0 virtualenv==20.24.6 watchfiles==0.21.0 wcwidth==0.2.9 websocket-client==1.6.4 websockets==12.0 wrapt==1.16.0 xmltodict==0.13.0 zipp==3.17.0 zope.interface==6.1

How can we reproduce your problem?

Run command DD_SERVICE=app DD_ENV=dev DD_LOGS_INJECTION=true DD_TRACE_ENABLED=true DD_VERSION=1.0.0 ddtrace-run uvicorn app.main:app

This is my main file:

import logging
import os
import time

from fastapi import FastAPI

from app.routers import main_router

logging.Formatter.converter = time.gmtime

FORMAT = (
    "%(asctime)s %(levelname)s [%(name)s] [%(filename)s:%(lineno)d] "
    "[dd.service=%(dd.service)s dd.env=%(dd.env)s "
    "dd.version=%(dd.version)s "
    "dd.trace_id=%(dd.trace_id)s dd.span_id=%(dd.span_id)s] "
    "- %(message)s"
)
logging.basicConfig(
    format=FORMAT,
    level=logging.INFO,
    handlers={
        logging.StreamHandler(),
    },
)

app = FastAPI(
    title="App",
    openapi_url="/openapi.json",
    docs_url="/docs",
)

app.include_router(main_router)

What is the result that you get?

No logs in datadog

What is the result that you expected?

Logs available in the datadog log explorer

ZStriker19 commented 10 months ago

Hi @Gitznik, thanks for reaching out! This is actually more of a logs question than a tracing question, so it's not really related to our library (as long as injection is occurring properly which you mention it is). It's purely on the Datadog agent to collect logs, and all the tracer does is inject contexts into the logs you're already generating.

Looking at the two ways you've set this up (log to file, have agent tail file) and tcp, I think those are pretty standard ways and would be recommended: https://docs.datadoghq.com/agent/logs/?tab=tailfiles#custom-log-collection If you're running in Docker/Kubernetes I'm pretty sure the log collection would be automatic since the agent would read from the container's stdout and stderr, but that doesn't seem to be your setup? If you'd like anymore assistance with someone who knows more about log collection, I'd recommend opening a support ticket: https://help.datadoghq.com/hc/en-us/requests/new

Going to close this for now but let us know if I've misunderstood or you think our library is affecting this in some way. Have a terrific Tuesday!

Gitznik commented 10 months ago

Hi @ZStriker19, thank's for the rapid response. That indeed answers my question, but I'd suggest adding this as a side node to the log injection documentation. For me it was not clear that I have to setup log collection manually.

Appreciate your help!