Observability and IncompleteEvents

[pcuriel]

When events are resubmitted, their original tracing context is lost.

Keeping this information is specially relevant when events are externalized.

Any thoughts on persisting context on the DB and restoring it when the event is read? Maybe using nullable column(s) and extracting context via an optional Bean, configured by spring-modulith-observability when this module is added.

[odrotbohm]

Does it really make sense to fully reestablish that context? Wouldn't those traces then look like an actual request having been performed?

[pcuriel]

I think it makes sense regarding event traceability. I will try to explain why with an example.

Given this scenario:

• Service A receives a request with Trace ID ABC
• As a part of that request, it publishes MyEvent via RabbitMQ.
• Service B receives MyEvent and does whatever.
• Service Creceives MyEvent and does whatever.

Considering happy-path execution, Trace ID ABC would enable you to trace up from the initial request to Service A, down to MyEvent reception and processing in Service B and Service C.

However, if anything goes wrong publishing MyEvent (e.g., RabbitMQ is down for a few seconds) and it is finally published when retried from the EventRepository, it will be published with another Trace ID, and we won't be able to trace it down to Service B and Service C.

However, persisting the context and restoring it when retrying the event would solve this issue. (Note: to simplify the example I'm referring only to the Trace ID, but the same would apply to other Trace Context info).

[marcingrzejszczak]

We could use the concepts of linked spans. That means that the new resubmission could happen at any time so if we kept the same trace id we would pollute the trace view. However we could create a new trace id and link the old one to the resubmitted one. WDYT?