psavidis
commented
9 months ago Kickoff
Context
There is a subset of our users who are utilising the MDC to store data to log management systems (e.g graylog).
- The
MDCis populated with the same key-value properties as the Logging Context Parameters by them externally. - The User uses the engine through the Java API (e.g deploy processes, start process instances, completes tasks etc) and commands are produced.
- Each command is associated with one
ProcessDataContextwhich is going to collectMDCinformation to log its operation - At the end of the command execution, the
ProcessDataContextis cleared and anyMDCkeys that are associated withLogging Context Parametersare cleared as well. - The External Properties the user sets before any operation are the same with
Logging Context Parameters, thus their presence in the MDC is also cleared.
Business Value
- Our users will have the capacity to Preserve any property that is set Externally on the MDC.
- Currently this problem is mitigated by writing custom Spring Proxies that try to alter the existing behaviour. The feature will eliminate this need.
- Non need to control the Logging Context Parameter Values to avoid this problem (e.g
activityIdshould be renamed to another value to avoid this conflict). - Better isolation on logging (what is consumed by the engine's logging mechanism does interfere with external MDC properties).
Customer Requirements
As a user
- I want *all MDC properties that are set externally, prior to any command execution to be preserved after process engine finishes its processing.
*all MDC Properties = Logging Context parameter properties + custom third party properties used by the user
Business Cases
👉 Assumption: The access of MDC to store the external properties ( by the user) and the process-engine are both living in the same JVM process
SLF4J,Logback,Log4J&java.util.loggingall use implementations that rely on thread-local access of the MDC.- ❓ Open point: The insertion of a Custom User MDC tuple and executing a job in production never happens in the same thread (most likely scenario, we need to verify that with the customer)
Backwards compatibility
- If there were keys used by Customers that had the same values as the
Logging Context Parameterkeys, they used to be cleared (legacy behaviour). After this change, the keys are going to be preserved. - The user can also clear the
MDCmanually themselves (programmatically) to restore the legacy behaviour in case that is desired as a low-cost workaround. - Since the use case of expecting the aforementioned
Logging Context ParameterMDC keys to be cleared is not popular, we assume that this isn't a risky decision.
The above points render the feature non-backwards compatible. We will collect data about our customers usage of the feature and if required, we will deliver in a patch release the possibility to make it configurable.
Documentation
The Logging Context Parameters section needs to include a note which states that any external properties (third party or the same as logging context parameters) will be preserved after any command executions.
Backporting
This is a new feature request, it does not require any backports to previous versions. It will be available only in 7.21.
Technical Solution Proposal
How Current Design Works
- The
CommandContextcan be reused (e.g commands that spawn other commands to execute their logic) whereasCommandInvocationContextis created only once per each Command. - The
ProcessDataContexthas a 1-1 relationship withCommandInvocationContext - Each Command Invocation pushes its execution into the
ProcessDataContextfor eachAtomicOperationand it will be included in the MDC.
What needs to be added
- Store in
ProcessDataContextthe External Properties upon its creation & clear them when the command context is removed. - The External Properties that will be stored / restored will be the keys described by the Logging Context Parameters described in the
ProcessEngineConfiguration - External Properties should be parked only for Outer Commands
Breakdown
[camunda-bpm-platform] - Add Feature
- [x] Add the External Properties initialisation of
ProcessDataContextper eachCommandInvocationContext. Apply it only in the case of Outer Commands - [x] Restore them when
CommandInvocationContextis removed - [x] Add Unit Tests that cover:
- Preservation of External Properties after commands execution (e.g a start process instance & complete)
- Preservation of other custom properties in combination with the Logging Context Parameter properties
[camunda-docs-manual] - Add Documentation with the Changes under Sections:
- [x] Add a note under the Logging Context Parameters Section to state that any externally set properties will be preserved after the engine processing.
Migration guide 7.20 to 7.21
- [x] Highlight the breaking change character of the feature
- [x] Mention the workaround of clearing the MDC manually as a low cost alternative in case a customer wants to control the legacy behaviour
This issue was imported from JIRA:
What is this name?
This pseudonym name was generated based on the user name in JIRA to protect the personal data of our JIRA users. You can use this identifier to search for issues by the same reporter.Current behavior:
Problem
Suggested improvement
Links: