Open zmoog opened 3 weeks ago
Checking the first field, client.geo.location
in the logs-azure.graphactivitylogs-*
data stream. This field has an explicit mapping in the integration:
# packages/azure/data_stream/graphactivitylogs/fields/ecs.yml
- name: client.geo.location.lat
external: ecs
- name: client.geo.location.lon
external: ecs
- name: source.geo.location.lat
external: ecs
- name: source.geo.location.lon
external: ecs
This leads to the following mapping:
"client": {
"properties": {
"geo": {
"properties": {
"continent_name": {
"type": "keyword",
"ignore_above": 1024
},
"country_iso_code": {
"type": "keyword",
"ignore_above": 1024
},
"country_name": {
"type": "keyword",
"ignore_above": 1024
},
"location": {
"properties": {
"lat": {
"type": "geo_point"
},
"lon": {
"type": "geo_point"
}
}
}
}
},
"ip": {
"type": "ip"
}
}
}
So client.geo.location
here is an object.
Paradoxically, if I index the same document using a logs-*-*
data stream, I get the correct mapping from ecs@mappings:
GET logs-whatever-sdh5075/_mapping/field/client.geo.location
{
".ds-logs-whatever-sdh5075-2024.08.22-000001": {
"mappings": {
"client.geo.location": {
"full_name": "client.geo.location",
"mapping": {
"location": {
"type": "geo_point"
}
}
}
}
}
}
This seems a choice in the logs-azure.graphactivitylogs-*
data stream that does not align with ECS and other data streams.
We're getting a ~potential~ issue with host.os.version
which is no more defined on the System integration / processor
dataset.
While in most cases it is mapped as keyword
(it was mapped to keyword
in the past), some users seem to get sporadically get mapped to float
.
We can have "7.9"
and "7.9 (Maipo)"
, but those seem to be correctly coerced into keyword
. But it doesn't happen if Beats sends us 7.9
.
~I'm gathering a sample and I'll update the comment.~
We have a sample document where we clearly see Beats / Elastic Agent can send "version": 7.2
(where the version is not wrapped in quotes, so it is coerced to float
instead of being a keyword
).
A few users reported mapping errors on a few integrations. We suspect these problems may be related to integrations that migrated to ecs@mappings with recent updates.
Here is the list of fields with mapping issues:
client.geo.location
geo_point
object
logs-azure.graphactivitylogs-*
source.geo.location
geo_point
object
logs-azure.graphactivitylogs-*
destination.port
long
keyword
logs-cisco_aironet.log-*
event.duration
long
keyword
logs-azure.activitylogs.log-*
dns.authorities
dns.id
long
keyword
logs-logstash.tpot-*
error.code
keyword
long
logs-system.security-*
keyword
https://www.elastic.co/guide/en/ecs/current/ecs-error.html#field-error-codeevent.severity
long
keyword
logs-cisco_aironet.log-*
{"event.severity": "4"}
; ECS expected type islong
https://www.elastic.co/guide/en/ecs/current/ecs-event.html#field-event-severityhttp.request.body
object
flattened
logs-apm.error-*
http.request.headers
flattened
object
logs-apm.error-*
http.response.headers
flattened
object
logs-apm.error-*
input
object
keyword
logs-logstash.tpot-*
log.offset
long
keyword
logs-microsoft_exchange_server.httpproxy-*
keyword
observer.ip
ip
keyword
logs-ti_abusech_latest.dest_malware-*
request
text
object
logs-logstash.tpot-*
response
text
object
logs-logstash.tpot-*
session
sip.uri
status
keyword
long
logs-logstash.tpot-*
threat.indicator.first_seen
date
keyword
logs-ti_abusech.malware-*
threat.indicator.last_seen
date
keyword
logs-ti_abusech.malwarebazaar-*
timestamp
user_agent
object
keyword
logs-cisco_asa.log-*
object
is the expected mapping foruser_agent
; see https://www.elastic.co/guide/en/ecs/current/ecs-user_agent.htmlRoot Causes
fields/ecs.yml
filedate_detection: false
cause a few fields to not be mapped asdate
date_detection: true
or update ecs@mappingsecs@mappings+type-coercion
The ecs@mappings component template does not perform type coercion, so if the value is a string, ES maps it as a
keyword
.Here is an example, if I perform the following requests using the Dev Tools:
I get the following result:
ecs@mappings+date_detection:false
When date_detection is disabled, the following fields aren’t mapped correctly:
integration-mapping-problem
We probably need to change mappings in the integration to something similar (like most other integrations do):
Or remove these mappings and only use ecs@mappings.
integration-update
Mapping changed due to integration updates.