Add support for additional fields at `message` level in pipeline tests

[narph]

Ex:

test-datastream-raw.log file contains message value: { valid json}

Pipeline tests are successful

if the file has the following format:

{'some_field': 'hello', 'message':'{valid json}'} then pipeline tests consider the entire log part of the message

Ex https://beats-ci.elastic.co/blue/organizations/jenkins/Ingest-manager%2Fintegrations/detail/PR-1462/8/tests where the external fields like azure_log_forwarder were placed inside message

[mtojek]

I'm not sure if I understand this issue correctly. Is it something you can address with dynamic fields (see docs)?

Could you please post fragments of real metrics/logs?

[narph]

@mtojek , the example is in the build test I have linked above, I have copy/pasted it here:

     "expected": [
         {
             "azure": {
                 "platformlogs": {
                    "@timestamp": "2020-10-28T11:59:26.373Z",
                    "agent": {
                        "name": "MicrosoftAgent",
                        "version": "1.0.0"
                    },
                    "azure_log_forwarder": {
                        "category": "kube-apiserver",
                        "resource_type": "Microsoft.AppPlatform/Spring",
                        "service_provider": "Microsoft.ContainerService"
                    },
                    "data_stream": {
                        "dataset": "azure.platformlogs",
                        "namespace": "default",
                        "type": "logs"
                    },
                    "event": {
                        "dataset": "azure.platformlogs",
                        "module": "azure"
                    },
                     "event_category": "Administrative",
                    "input": {
                        "type": "azure-log-forwarder"
                    },
                    "message": "{ \"time\": \"2021-07-01T19:30:30.535404056Z\", \"LogFormat\": \"RAW\", \"resourceId\": \"/SUBSCRIPTIONS/EDD63B67-0BA2-4837-A4EB-CD484E9FF623/RESOURCEGROUPS/SA-HEMANT/PROVIDERS/MICROSOFT.APPPLATFORM/SPRING/HM-SC-PETCLINIC\", \"operationName\": \"Microsoft.AppPlatform/Spring/logs\", \"category\": \"ApplicationConsole\", \"level\": \"Informational\", \"location\": \"westus2\", \"properties\": {\"Log\":\"2021-07-01 19:30:30.535 INFO 1 --- [oundedElastic-9] c.c.c.ConfigServicePropertySourceLocator : Located environment: name=admin-server, profiles=[mysql], label=null, version=638a1af7fc8d331d7eb26a571275e954632717e8, state=null\\n\",\"Stream\":\"stdout\",\"AppName\":\"admin-server\",\"InstanceName\":\"admin-server-default-12-8459d44f68-g4b5f\",\"ServiceId\":\"c41fd000b1a5450eb234039376da26de\",\"ServiceName\":\"hm-sc-petclinic\"}}",
                    "service": {
                        "type": "azure"
                    },
                    "tags": [
                        "forwarded"
                    ]
                }
             },
             "cloud": {
                 "provider": "azure"
             },
             "ecs": {
                 "version": "1.11.0"
             },
             "event": {
                "kind": "event",
                "original": "{\"@timestamp\":\"2020-10-28T11:59:26.373Z\",\"agent\":{\"name\":\"MicrosoftAgent\",\"version\":\"1.0.0\"},\"data_stream\":{\"namespace\":\"default\",\"type\":\"logs\",\"dataset\":\"azure.platformlogs\"},\"input\":{\"type\":\"azure-log-forwarder\"}, \"azure_log_forwarder\" : {\"service_provider\" : \"Microsoft.ContainerService\",\"category\" : \"kube-apiserver\",\"resource_type\": \"Microsoft.AppPlatform/Spring\"},\"message\":\"{ \\\"time\\\": \\\"2021-07-01T19:30:30.535404056Z\\\", \\\"LogFormat\\\": \\\"RAW\\\", \\\"resourceId\\\": \\\"\\/SUBSCRIPTIONS\\/EDD63B67-0BA2-4837-A4EB-CD484E9FF623\\/RESOURCEGROUPS\\/SA-HEMANT\\/PROVIDERS\\/MICROSOFT.APPPLATFORM\\/SPRING\\/HM-SC-PETCLINIC\\\", \\\"operationName\\\": \\\"Microsoft.AppPlatform\\/Spring\\/logs\\\", \\\"category\\\": \\\"ApplicationConsole\\\", \\\"level\\\": \\\"Informational\\\", \\\"location\\\": \\\"westus2\\\", \\\"properties\\\": {\\\"Log\\\":\\\"2021-07-01 19:30:30.535 INFO 1 --- [oundedElastic-9] c.c.c.ConfigServicePropertySourceLocator : Located environment: name=admin-server, profiles=[mysql], label=null, version=638a1af7fc8d331d7eb26a571275e954632717e8, state=null\\\\n\\\",\\\"Stream\\\":\\\"stdout\\\",\\\"AppName\\\":\\\"admin-server\\\",\\\"InstanceName\\\":\\\"admin-server-default-12-8459d44f68-g4b5f\\\",\\\"ServiceId\\\":\\\"c41fd000b1a5450eb234039376da26de\\\",\\\"ServiceName\\\":\\\"hm-sc-petclinic\\\"}}\",\"tags\":[\"forwarded\"],\"service\":{\"type\":\"azure\"},\"event\":{\"module\":\"azure\",\"dataset\":\"azure.platformlogs\"}}"
             },
             "tags": [
                 "preserve_original_event"
             ]
         }
     ]
 }

We expected the pipelines to handle the message and every property under it to be at azure.platformlogs level, instead it takes entire event. This is the pipeline https://github.com/elastic/integrations/blob/master/packages/azure/data_stream/platformlogs/elasticsearch/ingest_pipeline/default.yml