open-telemetry / opentelemetry-python

OpenTelemetry Python API and SDK
https://opentelemetry.io
Apache License 2.0
1.8k stars 625 forks source link

Invalid type _FixedFindCallerLogger for attribute '_logger' value. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types #3649

Closed Sammyjroberts closed 9 months ago

Sammyjroberts commented 9 months ago

Describe your environment python version 3.11.7 structlog = "^24.1.0" opentelemetry-distro = "^0.43b0" opentelemetry-instrumentation-fastapi = "0.43b0" opentelemetry-instrumentation-pika = "0.43b0" opentelemetry-instrumentation-requests = "0.43b0" opentelemetry-exporter-otlp-proto-http = "^1.22.0" opentelemetry-exporter-otlp-proto-grpc = "^1.22.0"

running in kubernetes and the logs are appearing in datadog, can provide any extra

Steps to reproduce Used structlogs as my default logger, this does not happen locally.

What is the expected behavior? no warn

What is the actual behavior? this error happens on every few seconds, likely related to a health check log

Additional context Add any other context about the problem here.

Invalid type _FixedFindCallerLogger for attribute '_logger' value. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types

the logger logging this log opentelemetry.attributes

if i can provide any additional info, let me know

lzchen commented 9 months ago

Could provide more information with how you are using your logging calls and how you are instrumenting with OpenTelemetry? Perhaps a code snippet.

ToTheMax commented 7 months ago

Hi @Sammyjroberts did you find a workaround / fix? For me this warning shows when I use structlog in combination with azure-monitor-opentelemetry.

waza-ari commented 4 months ago

Same issue here as @ToTheMax describes. I'm using structlog in combination with azure-monitor-opentelemetry.

folt commented 3 months ago

I have the same problem, I also use structlog

pmcollins commented 3 months ago

Thanks all for the reports. Can someone provide a reproducing script?

roger-collins-self commented 2 months ago

My current fix to get this working with OpenTelemetry (and Websockets):

from opentelemetry.sdk._logs import LoggingHandler
from opentelemetry.util.types import Attributes

class AttrFilteredLoggingHandler(LoggingHandler):
    DROP_ATTRIBUTES = ["_logger", "websocket"]

    @staticmethod
    def _get_attributes(record: logging.LogRecord) -> Attributes:
        attributes = LoggingHandler._get_attributes(record)
        for attr in AttrFilteredLoggingHandler.DROP_ATTRIBUTES:
            if attr in attributes:
                del attributes[attr]
        return attributes

OTEL's LoggingHandler doesn't call the super().emit nor does it call self.format(), so the processors are never run. This means the Record still has a the _logger attribute stored from structlog.stdlib.ProcessorFormatter.wrap_for_formatter which can't be serialized.

I believe the only way to fix this is a PR to the Python OpenTelemetry SDK

Likewise, this is a problem when you turn on DEBUG logging for the websockets library as well, as it stores the websocket object as an extra to its logs.

mdgilene commented 1 month ago

My current fix to get this working with OpenTelemetry (and Websockets):

from opentelemetry.sdk._logs import LoggingHandler
from opentelemetry.util.types import Attributes

class AttrFilteredLoggingHandler(LoggingHandler):
    DROP_ATTRIBUTES = ["_logger", "websocket"]

    @staticmethod
    def _get_attributes(record: logging.LogRecord) -> Attributes:
        attributes = LoggingHandler._get_attributes(record)
        for attr in AttrFilteredLoggingHandler.DROP_ATTRIBUTES:
            if attr in attributes:
                del attributes[attr]
        return attributes

OTEL's LoggingHandler doesn't call the super().emit nor does it call self.format(), so the processors are never run. This means the Record still has a the _logger attribute stored from structlog.stdlib.ProcessorFormatter.wrap_for_formatter which can't be serialized.

I believe the only way to fix this is a PR to the Python OpenTelemetry SDK

Likewise, this is a problem when you turn on DEBUG logging for the websockets library as well, as it stores the websocket object as an extra to its logs.

Could you maybe explain how you are getting opentelemetry to use that updated handler configuration?