Original install method (e.g. download page, yum, from source, etc.): elastic-package stack up --version=8.14.0-SNAPSHOT
Description of the problem including expected versus actual behavior: For ti_abusech package, the 4 latest transforms exist (1 for each datastream) to copy only the latest indicators from source datastream into the destination index. The latest_malware transform is incorrectly creating extra field mappings inside destination index that are not defined in any of its fields definitions. Interestingly, these extra field mappings are coming from another transform latest_malwarebazaar field definitions. This could be due to a bug in Fleet transform which is taking field definitions based on a pattern, malware vs malwarebazaar?
Here's a diff of the mappings that show extra fields inside malware's destination index. Notice extra field mappings such as abusech.malwarebazaar.* that shouldn't exist.
Bring up the Elastic Stack using elastic-package stack up --version=8.14.0-SNAPSHOT -d -v
Install AbuseCH package enabling AbuseCH Malware payloads using Elastic Agent.
In Dev Tools, run GET logs-ti_abusech_latest.dest_malware-1/_mapping to get mappings of destination index created by the transform logs-ti_abusech.latest_malware-default-0.1.0.
Verify that fields named abusech.malwarebazaar.* exist which were never defined by the transform field definitions.. There are also many threat.* ECS fields such as threat.software.alias which are present in the mappings, but were never added in the field definitions.
Kibana version: 8.14.0-SNAPSHOT
Elasticsearch version: 8.14.0-SNAPSHOT
Original install method (e.g. download page, yum, from source, etc.):
elastic-package stack up --version=8.14.0-SNAPSHOT
Description of the problem including expected versus actual behavior: For
ti_abusech
package, the 4 latest transforms exist (1 for each datastream) to copy only the latest indicators from source datastream into the destination index. Thelatest_malware
transform is incorrectly creating extra field mappings inside destination index that are not defined in any of its fields definitions. Interestingly, these extra field mappings are coming from another transformlatest_malwarebazaar
field definitions. This could be due to a bug in Fleet transform which is taking field definitions based on a pattern,malware
vsmalwarebazaar
?Here's a diff of the mappings that show extra fields inside
malware
's destination index. Notice extra field mappings such asabusech.malwarebazaar.*
that shouldn't exist.diff <(cat ~/abusech-malware-source.json | flatten-json) <(cat ~/abusech-malware-dest.json | flatten-json)
abusech-malware-source.json abusech-malware-dest.jsonSteps to reproduce:
elastic-package stack up --version=8.14.0-SNAPSHOT -d -v
AbuseCH Malware payloads using Elastic Agent
.logs-ti_abusech_latest.dest_malware-1/_mapping
to get mappings of destination index created by the transformlogs-ti_abusech.latest_malware-default-0.1.0
.abusech.malwarebazaar.*
exist which were never defined by the transform field definitions.. There are also manythreat.*
ECS fields such asthreat.software.alias
which are present in the mappings, but were never added in the field definitions.