MeshAccessLogs: Multiple lines of logs are consumed as single line by Dynatrace

[lukidzi]

What happened?

Dynatrace requires [timestamp] [logLevel] {json_data} to correlate data as a single log. Because JSON format produced by Envoy is a full json string without these prefixes, logs are not properly indexed by Dynatrace. It would be nice to support this format or HTTP backend

type: MeshAccessLog
name: accesslogs
mesh: default
spec:
 targetRef:
   kind: Mesh
 from:
   - targetRef:
       kind: Mesh
     default:
       backends:
         - type: File
           file:
             path: /dev/stdout
             format:
               type: Json
                 json:
                   - key: "start_time"
                      value: "%START_TIME%"
                   - key: "bytes_received"
                      value: "%BYTES_RECEIVED%"

[bartsmykla]

triage: it's possible that Dynatrace support OTEL gRPC endpoints, which should solve this issue. We need to check if that's possible

[Icarus9913]

I found the way to handle with the stdout in json type. Use the MeshProxyPatch to supplement the JsonFormatOptions sort_properties (The output JSON string properties will be sorted.) In this way, each log would be single one that was analyzed by Dynatrace

apiVersion: kuma.io/v1alpha1
kind: MeshAccessLog
metadata:
  name: default
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Mesh
  from:
    - targetRef:
        kind: Mesh
      default:
        backends:
          - type: File
            file:
              path: /dev/stdout
              format:
                type: Json
                json:
                - key: "data_set"
                  value: "kuma"
                - key: "start_time"
                  value: "%START_TIME(%Y-%m-%d %H:%M:%S %s)%"
                - key: "kuma_mesh"
                  value: "%KUMA_MESH%"
                - key: "kuma_source_address_without_port"
                  value: "%KUMA_SOURCE_ADDRESS_WITHOUT_PORT%"
                - key: "kuma_source_service"
                  value: "%KUMA_SOURCE_SERVICE%"
                - key: "upstream_host"
                  value: "%UPSTREAM_HOST%"
                - key: "kuma_destination_service"
                  value: "%KUMA_DESTINATION_SERVICE%"
                - key: "duration_ms"
                  value: "%DURATION%"
                - key: "bytes_sent"
                  value: "%BYTES_SENT%"
                - key: "bytes_received"
                  value: "%BYTES_RECEIVED%" 
---
apiVersion: kuma.io/v1alpha1
kind: MeshProxyPatch
metadata:
  name: accesslog
  namespace: kuma-system
spec:
  targetRef:
    kind: Mesh
  default:
    appendModifications:
      - listener:
          operation: Patch
          match:
            origin: inbound
            tags:
              k8s.kuma.io/service-name: httpbin-1
          jsonPatches:
            - op: add
              path: /filterChains/0/filters/0/typedConfig/accessLog/0/typedConfig/logFormat/jsonFormatOptions
              value: {"sort_properties": true}

[Icarus9913]

Maybe we should also add that jsonFormatOptions property in our MeshAccessLog policy configuration

https://github.com/kumahq/kuma/blob/e21c751d782bc06eb7e8c779dacf71097bfe5732/pkg/plugins/policies/meshaccesslog/plugin/xds/configurer.go#L163-L172

[michaelbeaumont]

Waiting on feedback from user on MeshProxyPatch