[out_stackdriver] Support textPayload format in Cloud Logging LogEntry API

[shuaich]

Is your feature request related to a problem? Please describe. Stackdriver output plugin writes log payload as json format[1] as the only supported format. Cloud Logging LogEntry also support text payload format[2]. [1]https://github.com/fluent/fluent-bit/blob/master/plugins/out_stackdriver/stackdriver.c#L2437 [2]https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry

This is a feature request for out_stackdriver to support writing to textPayload field in LogEntry.

Describe the solution you'd like

Proposal Add a new configuration parameter text_payload_key, which default to unset.

• text_payload_key is unset The out_stackdriver plugin writes to jsonPayload field to be backwards compatible with current behaviour.
• text_payload_key is set non-empty value, e.g. message out_stackdriver determines the log payload format based on the residual log fields after stripping away special log fields. The following are examples for input log records.

// Example A
{
  ...
  // "message" is the only residual log field
  "message": "The application errored out",
  "logging.googleapis.com/severity": "ERROR"
}

// Example B
{
  ...
  // There are two residual fields: "message" and "errorCode" 
  "message": "The application errored out",
  "errorCode": "400",
  "logging.googleapis.com/severity": "ERROR"
}

// Example C
{
  ...
  // "message" is a non-scalar json object and there is one more residual field "errorCode"
  "message": {
    "application_name": "my_application",
    "error_message": "The application errored out"
  },
  "errorCode": "400",
  "logging.googleapis.com/severity": "ERROR"
}

Write to textPayload field only when message is the only residual scalar field. Therefore, the resulting log entries are:

// Example A
LogEntry {
  ...
  "severity": "ERROR",
  "textPayload": "The application errored out"
}

// Example B
LogEntry {
  ...
  "severity": "ERROR",
  "jsonPayload": {
    "message": "The application errored out",
    "errorCode": "400"
  }
}

// Example C
LogEntry {
  ...
  "severity": "ERROR",
  "jsonPayload":   {
    "message": {
      "application_name": "my_application",
      "error_message": "The application errored out"l
    },
    "errorCode": "400"
  }
}

Describe alternatives you've considered

Put log payload under a new special logging.googleapis.com/textPayload log field.

Additional context

[qingling128]

I'm slightly concerned about the part of the proposal that drops any fields other than "textPayload".

Is this ultimately trying to reach parity with how Fluentd's Stackdriver output plugin works? If so, that plugin automatically switches to textPayload if and only if there is only one field in the jsonPlayload. Can we follow that pattern instead?

[shuaich]

I'm slightly concerned about the part of the proposal that drops any fields other than "textPayload".

Updated the proposal to fallback to current jsonPayload behaviour when textPayload field is a non-scalar object and don't drop any fields.

Is this ultimately trying to reach parity with how Fluentd's Stackdriver output plugin works?

It is not. The purpose of this PR is to support writing to LogEntry.textPayload field in out_stackdriver plugin.