Closed Sammyjroberts closed 9 months ago
Could provide more information with how you are using your logging calls and how you are instrumenting with OpenTelemetry? Perhaps a code snippet.
Hi @Sammyjroberts did you find a workaround / fix? For me this warning shows when I use structlog in combination with azure-monitor-opentelemetry.
Same issue here as @ToTheMax describes. I'm using structlog in combination with azure-monitor-opentelemetry.
I have the same problem, I also use structlog
Thanks all for the reports. Can someone provide a reproducing script?
My current fix to get this working with OpenTelemetry (and Websockets):
from opentelemetry.sdk._logs import LoggingHandler
from opentelemetry.util.types import Attributes
class AttrFilteredLoggingHandler(LoggingHandler):
DROP_ATTRIBUTES = ["_logger", "websocket"]
@staticmethod
def _get_attributes(record: logging.LogRecord) -> Attributes:
attributes = LoggingHandler._get_attributes(record)
for attr in AttrFilteredLoggingHandler.DROP_ATTRIBUTES:
if attr in attributes:
del attributes[attr]
return attributes
OTEL's LoggingHandler doesn't call the super().emit
nor does it call self.format()
, so the processors are never run. This means the Record still has a the _logger
attribute stored from structlog.stdlib.ProcessorFormatter.wrap_for_formatter
which can't be serialized.
I believe the only way to fix this is a PR to the Python OpenTelemetry SDK
Likewise, this is a problem when you turn on DEBUG logging for the websockets
library as well, as it stores the websocket object as an extra
to its logs.
My current fix to get this working with OpenTelemetry (and Websockets):
from opentelemetry.sdk._logs import LoggingHandler from opentelemetry.util.types import Attributes class AttrFilteredLoggingHandler(LoggingHandler): DROP_ATTRIBUTES = ["_logger", "websocket"] @staticmethod def _get_attributes(record: logging.LogRecord) -> Attributes: attributes = LoggingHandler._get_attributes(record) for attr in AttrFilteredLoggingHandler.DROP_ATTRIBUTES: if attr in attributes: del attributes[attr] return attributes
OTEL's LoggingHandler doesn't call the
super().emit
nor does it callself.format()
, so the processors are never run. This means the Record still has a the_logger
attribute stored fromstructlog.stdlib.ProcessorFormatter.wrap_for_formatter
which can't be serialized.I believe the only way to fix this is a PR to the Python OpenTelemetry SDK
Likewise, this is a problem when you turn on DEBUG logging for the
websockets
library as well, as it stores the websocket object as anextra
to its logs.
Could you maybe explain how you are getting opentelemetry to use that updated handler configuration?
Describe your environment python version 3.11.7 structlog = "^24.1.0" opentelemetry-distro = "^0.43b0" opentelemetry-instrumentation-fastapi = "0.43b0" opentelemetry-instrumentation-pika = "0.43b0" opentelemetry-instrumentation-requests = "0.43b0" opentelemetry-exporter-otlp-proto-http = "^1.22.0" opentelemetry-exporter-otlp-proto-grpc = "^1.22.0"
running in kubernetes and the logs are appearing in datadog, can provide any extra
Steps to reproduce Used structlogs as my default logger, this does not happen locally.
What is the expected behavior? no warn
What is the actual behavior? this error happens on every few seconds, likely related to a health check log
Additional context Add any other context about the problem here.
Invalid type _FixedFindCallerLogger for attribute '_logger' value. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types
the logger logging this log opentelemetry.attributes
if i can provide any additional info, let me know