In NoSql appender, be able to use both default fields and custom fields

[jfontsaballs]

Right now, the fields of the documents that the NoSql appender creates are fixed, unless you use a MapMessage and a MessageLayout. In this case you get the fields from the MapMessage but none of the default fields.

If you do this (Kotlin code in this example, but Java would have the same problem):

logger.info(MapMessage(mapOf("aa" to "11", "bb" to "22")))

You can either get this:

{
  "_id": {
    "$oid": "64f1fc8ec553ec3593274f30"
  },
  "level": "INFO",
  "loggerName": "InjectedLogger.-1169081147",
  "message": "aa=\"11\" bb=\"22\"",
  "source": {
    "className": "InjectedLogger",
    "methodName": "execute",
    "fileName": "Main.kt",
    "lineNumber": 32
  },
  "marker": null,
  "threadId": {
    "$numberLong": "1"
  },
  "threadName": "main",
  "threadPriority": 5,
  "millis": {
    "$numberLong": "1693580430801"
  },
  "date": {
    "$date": "2023-09-01T15:00:30.801Z"
  },
  "thrown": null,
  "contextMap": {},
  "contextStack": [],
  "additionalFields": {}
}

Or this:

{
  "_id": {
    "$oid": "64f1dbbddb2eb17fd4809d51"
  },
  "aa": "11",
  "bb": "22",
  "additionalFields": {}
}

I want this (order of the fields is unimportant):

{
  "_id": {
    "$oid": "64f1fc8ec553ec3593274f30"
  },
  "level": "INFO",
  "loggerName": "InjectedLogger.-1169081147",
  "message": "aa=\"11\" bb=\"22\"",
  "source": {
    "className": "InjectedLogger",
    "methodName": "execute",
    "fileName": "Main.kt",
    "lineNumber": 32
  },
  "marker": null,
  "threadId": {
    "$numberLong": "1"
  },
  "threadName": "main",
  "threadPriority": 5,
  "millis": {
    "$numberLong": "1693580430801"
  },
  "date": {
    "$date": "2023-09-01T15:00:30.801Z"
  },
  "thrown": null,
  "contextMap": {},
  "contextStack": [],
  "additionalFields": {},
  "aa": "11",
  "bb": "22",
}

I have been doing some analysis and I believe that this functionality would be very difficult to achieve with the provided extension points, as the code that creates the document layout can't be swapped from configuration.

Then it would be easy to create a custom Message or a Layout that can control the message field and set it to something more interesting than repeating the custom properties (not part of this request). StructuredDataMessage, for instance, already has a message property that can be used via a custom layout.

I propose creating a configuration option that would enable this behavior.

I have been looking at the code and, apart from creating the configuration option, only a minor change would be required here:

if (serializable instanceof MapMessage) {
    // Add these three lines
    if (newOption){
        setFields(event, entity);
    }
    setFields((MapMessage<?, ?>) serializable, entity);
} else {
    setFields(event, entity);
}

Another option (maybe more flexible) would be to move all the code that sets the document properties from the event (writeInternal and setFields methods from NoSqlDatabaseManager) to a new class that can be swapped via configuration. Something in the lines of a NoSqlObjectLayout that would be conceptually similar to Layouts but that outputs (or sets fields to) a NoSql document instead of a String.

[ppkarwasz]

This is related to #2389.