search

open-telemetry / opentelemetry-cpp

The OpenTelemetry C++ Client
https://opentelemetry.io/
Apache License 2.0
834 stars 396 forks source link

C++ Logger API does not compliant with the OTel Logs Data Model #3000

Open yijiem opened 1 month ago

yijiem commented 1 month ago

In the OTel Logs Data Model, it is specified that the Body field must support any type to preserve the semantics of structured logs emitted by the application: https://opentelemetry.io/docs/specs/otel/logs/data-model/#field-body. This is specifically useful for the application which logs a JSON-structured data i.e. map<string, any> as the Body. But the OTel C++ Logger API does not support such a case, it supports string_view or AttributeValue type in the Body (code) but neither of those can preserve the semantics of structured data.

Ideally, if the Logger API preserves the semantics of structured data, the structure should be preserved across the stack (e.g. from application to OTel collector via OTLP and eventually the structured data gets exported to the backend log store).

Seems like there is also an Event API for logging structured data. But it is still a WIP and is a specialized Log API and would have the same issue in opentelemetry-cpp since the current API does not support structured data.

owent commented 1 month ago

Good point, do we need a new class to describe any and a new variant which contains map<string, any>? It may changes the ABI. @open-telemetry/cpp-approvers

lalitb commented 1 month ago

To be specs compliant with Log data model, we would need to add support for Any type with below variant types in addition to those existing in AttributeValue :

Supporting non-owning types for these heterogenous deeply nested structure would be complex to implement and maintain, and I believe we can wait for sufficient ask for this support before deciding on it.

As of now, other languages with stable log API - Java, JS, .Net, Python - are not fully compliant as well. Java uses string type for body, python reuses common attributes for log-attributes. I think most of the languages are taking decision based on whether the logging frameworks for these language support these types or not. And in C++, we provide a user-facing logging API where it is recommended to use the message-template format for the log body.

yijiem commented 1 month ago

Will the message-template formatted log body and its structure being preserved when OTel-Cpp exports the log data e.g. through OTLP to a collector (and eventually to a backend logging store)? gRPC wants to log JSON-structured data through OTel and ideally the structure of the data should be preserved in the backend logging store.

BTW, just throw in one idea of implementing a JSON-like nested data structure: https://godbolt.org/z/oeEh66aad.

ThomsonTan commented 1 month ago

Will the message-template formatted log body and its structure being preserved when OTel-Cpp exports the log data e.g. through OTLP to a collector (and eventually to a backend logging store)? gRPC wants to log JSON-structured data through OTel and ideally the structure of the data should be preserved in the backend logging store.

BTW, just throw in one idea of implementing a JSON-like nested data structure: godbolt.org/z/oeEh66aad.

OTel-Cpp is not supposed to change or format the log body, the message-template and attributes will be kept and exported.

For logging JSON, wondering that is the benefit of structured with map<string, any> than logging it as a plain string?

yijiem commented 1 month ago

Structured logging in supported backend logging stores will have benefits such as easy to query. This is especially true when the payload is structured at the point of its origin.

Edit: it seems like OTLP is able to carry structured data in LogRecord.body.

esigo commented 2 weeks ago

discussed in SIG meeting 7th August, https://github.com/open-telemetry/opentelemetry-cpp/issues/3000#issuecomment-2222150596 and https://github.com/open-telemetry/opentelemetry-cpp/issues/3000#issuecomment-2224043461 we would support only string type as body.

yijiem commented 2 weeks ago

How about supporting structured data type as body in the Event API which IIUC is for logging structured data with well-defined semantic/schema and requires an AnyValue typed Body field in its spec as well? Either way there needs to be an AnyValue type in opentelemetry-cpp.

IIUC, opentelemetry-java is in the process of stablizing AnyValue and APIs for recording AnyValue log record bodies, see https://github.com/open-telemetry/opentelemetry-java/issues/6581, https://github.com/open-telemetry/opentelemetry-java/pull/6591 and https://github.com/open-telemetry/opentelemetry-java/issues/6626 which will make them compliant with the Log data model.

opentelemetry-go already supports this since the language itself supports any type: https://github.com/open-telemetry/opentelemetry-go/blob/main/log/keyvalue.go#L48-L52.