`event.dataset` combines the Fleet Custom Logs dataset name with the dataset name of the ECS formatted json log message damaging visualization

[cyrille-leclerc]

Kibana version:

7.15 BC3

Elasticsearch version:

7.15 BC3

Original install method (e.g. download page, yum, from source, etc.):

ESS

Describe the bug:

When using Elastic Agent + Fleet combined with a JSON application log file generated by an ECS compatible library the event.dataset contains 2 values which is confusing for the user and the readability of the "logs app" is damaged.

The ECS formatted log message contains "event.dataset":"frontend.log"

Original ECS formatted log entry on the application filesystem

```json { "@timestamp":"2021-08-30T21:11:42.517Z", "log.level":"INFO", "message":"SUCCESS createOrder([OrderController.OrderForm@7c615556 paymentMethod = 'PAYPAL'list[[OrderProductDto@6ca5f03d product = [Product@2a4cdb46 id = 4, name = 'Icecream', price = 5.0], quantity = 1]]]): price: 5.0, id:3255113", "ecs.version":"1.2.0", "service.name":"frontend", "event.dataset":"frontend.log", "process.thread.name":"http-nio-8080-exec-7", "log.logger":"com.mycompany.ecommerce.controller.OrderController", "transaction.id":"2f4d80a5f7f50197", "trace.id":"e64941a83550e2785179e1ad479fa493" } ```

In discover, we see "event.dataset": ["frontend", "frontend.log"]

Log entry as a datastream entry in Elasticsearch

```json { "_index": ".ds-logs-frontend-default-2021.08.30-000001", "_type": "_doc", "_id": "sv7nmHsBgGepNw45W1hC", "_version": 1, "_score": 1, "_source": { "@timestamp": "2021-08-30T21:11:42.517Z", "log.logger": "com.mycompany.ecommerce.controller.OrderController", "event": { "dataset": "frontend" }, "ecs": { "version": "1.10.0" }, "message": "SUCCESS createOrder([OrderController.OrderForm@7c615556 paymentMethod = 'PAYPAL'list[[OrderProductDto@6ca5f03d product = [Product@2a4cdb46 id = 4, name = 'Icecream', price = 5.0], quantity = 1]]]): price: 5.0, id:3255113", "trace.id": "e64941a83550e2785179e1ad479fa493", "ecs.version": "1.2.0", "input": { "type": "log" }, "elastic_agent": { "snapshot": false, "version": "7.14.0", "id": "8742eb04-bbba-4322-8d30-b4fb2386f5d9" }, "log": { "offset": 6130101, "file": { "path": "/usr/local/var/log/my-shopping-cart/frontend.log" } }, "service.name": "frontend", "process.thread.name": "http-nio-8080-exec-7", "agent": { "version": "7.14.0", "hostname": "MacBook-Pro.localdomain", "id": "8742eb04-bbba-4322-8d30-b4fb2386f5d9", "ephemeral_id": "9581d856-cbeb-4342-911c-9f28f39e9d7e", "name": "MacBook-Pro.localdomain", "type": "filebeat" }, "host": { "hostname": "MacBook-Pro.localdomain", "architecture": "x86_64", "os": { "kernel": "20.6.0", "build": "20G95", "type": "macos", "platform": "darwin", "version": "10.16", "family": "darwin", "name": "Mac OS X" }, "id": "04A12D9F-C409-5352-B238-99EA58CAC285", "ip": [ "fe80::aede:48ff:fe00:1122", "fe80::4ed:33e4:69e1:de9b", "192.168.1.46", "fe80::1c55:99ff:fe90:bdb8", "fe80::1c55:99ff:fe90:bdb8", "fe80::93de:95f4:1af0:56ea", "fe80::8810:5541:e830:a4c6" ], "mac": [ "ac:de:48:00:11:22", "fa:ff:c2:4e:d1:b1", "f8:ff:c2:4e:d1:b1", "1e:55:99:90:bd:b8", "1e:55:99:90:bd:b8", "82:bf:e9:40:48:01", "82:bf:e9:40:48:00", "82:bf:e9:40:48:05", "82:bf:e9:40:48:04", "82:bf:e9:40:48:01" ], "name": "MacBook-Pro.localdomain" }, "transaction.id": "2f4d80a5f7f50197", "log.level": "INFO", "event.dataset": "frontend.log", "data_stream": { "type": "logs", "dataset": "frontend", "namespace": "default" } }, "fields": { "elastic_agent.version": [ "7.14.0" ], "host.hostname": [ "MacBook-Pro.localdomain" ], "host.mac": [ "ac:de:48:00:11:22", "fa:ff:c2:4e:d1:b1", "f8:ff:c2:4e:d1:b1", "1e:55:99:90:bd:b8", "1e:55:99:90:bd:b8", "82:bf:e9:40:48:01", "82:bf:e9:40:48:00", "82:bf:e9:40:48:05", "82:bf:e9:40:48:04", "82:bf:e9:40:48:01" ], "log.logger": [ "com.mycompany.ecommerce.controller.OrderController" ], "transaction.id": [ "2f4d80a5f7f50197" ], "host.os.build": [ "20G95" ], "host.ip": [ "fe80::aede:48ff:fe00:1122", "fe80::4ed:33e4:69e1:de9b", "192.168.1.46", "fe80::1c55:99ff:fe90:bdb8", "fe80::1c55:99ff:fe90:bdb8", "fe80::93de:95f4:1af0:56ea", "fe80::8810:5541:e830:a4c6" ], "trace.id": [ "e64941a83550e2785179e1ad479fa493" ], "agent.type": [ "filebeat" ], "host.os.version": [ "10.16" ], "host.os.kernel": [ "20.6.0" ], "host.os.name": [ "Mac OS X" ], "log.level": [ "INFO" ], "agent.name": [ "MacBook-Pro.localdomain" ], "host.name": [ "MacBook-Pro.localdomain" ], "elastic_agent.snapshot": [ false ], "host.id": [ "04A12D9F-C409-5352-B238-99EA58CAC285" ], "process.thread.name": [ "http-nio-8080-exec-7" ], "host.os.type": [ "macos" ], "service.name": [ "frontend" ], "elastic_agent.id": [ "8742eb04-bbba-4322-8d30-b4fb2386f5d9" ], "data_stream.namespace": [ "default" ], "input.type": [ "log" ], "log.offset": [ 6130101 ], "agent.hostname": [ "MacBook-Pro.localdomain" ], "message": [ "SUCCESS createOrder([OrderController.OrderForm@7c615556 paymentMethod = 'PAYPAL'list[[OrderProductDto@6ca5f03d product = [Product@2a4cdb46 id = 4, name = 'Icecream', price = 5.0], quantity = 1]]]): price: 5.0, id:3255113" ], "data_stream.type": [ "logs" ], "host.architecture": [ "x86_64" ], "@timestamp": [ "2021-08-30T21:11:42.517Z" ], "agent.id": [ "8742eb04-bbba-4322-8d30-b4fb2386f5d9" ], "ecs.version": [ "1.10.0", "1.2.0" ], "host.os.platform": [ "darwin" ], "log.file.path": [ "/usr/local/var/log/my-shopping-cart/frontend.log" ], "data_stream.dataset": [ "frontend" ], "agent.ephemeral_id": [ "9581d856-cbeb-4342-911c-9f28f39e9d7e" ], "agent.version": [ "7.14.0" ], "host.os.family": [ "darwin" ], "event.dataset": [ "frontend", "frontend.log" ] } } ```

Steps to reproduce:

1. Generate java logs using the co.elastic.logging:logback-ecs-encoder library (example here)
2. Collect logs using Elastic Agent + Fleet
3. Visualize logs in the Kibana Logs app

Expected behavior:

I'm not clear if the dataset name specified in Fleet should overwrite the dataset name defined by the ECS log message but the visualization should look nice.

Screenshots (if relevant):

See above

[elasticmachine]

Pinging @elastic/logs-metrics-ui (Team:logs-metrics-ui)

[weltenwort]

@cyrille-leclerc the document you provide contains both values in "event.dataset": [ "frontend", "frontend.log" ], so they are correctly displayed in discover and the logs ui. This looks like a problem somewhere in the ingestion pathway, that can't be solved in either discover or the logs ui.

Does that make sense? Should this be reported to the integrations team instead?

[cyrille-leclerc]

@weltenwort I totally agree with you.

I don't have a clear answer but I feel the challenge is that Fleet Custom Log setup asks to specify a dataset.name even for ECS formatted logs that already specify a dataset.name. In this case, we have defined dataset.name in 2 places and I don't know which one should take precedence. My gut feeling is that the dataset.name defined the closest to the log source should take precedence and that in fact, we shouldn't ask users to specify a dataset.name in the fleet configuration when the log message already comes with a dataset.name (ie is an ECS compatible log message specifying event.dataset).

[weltenwort]

Ok, let's try to find the right team to assign, then. Would it be something to fix in fleet or in the integrations?

[elasticmachine]

Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs)