Field problems in SIEM dashboards

[SimonAccordDS]

I am not knowledgeable when it comes to the ELK stack so I have just followed the Easy Install doumentation to install LME by using the shell script, which has worked so far. However, when I enable the SIEM detection engine as in 4.2, I get an error every time I visit /app/SIEM:

[illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [event.module] in order to load field data by uninverting the inverted index. Note that this can use significant memory.

Also, the External alert count panel has a spinning circle and doesn't populate; changing the 'Stack by' in that panel will cause the error to repeat.

To Reproduce After following all LME instructions up to 4.2, with data being shipped to ELK from the Windows Event Collector via Winlogbeat, browse to /app/siem/ in Kibana.

Expected behavior No error

Screenshots

Windows Event Collector

• Server 2019 v1809
• WEC Config V0.3
• Winlogbeat Config V0.3
• Winlogbeat.exe version 7.9.1
• sysmon config V0.3
• sysmon executable V11.11

Linux Server

• Docker version 19.03.12
• Docker compose stack file version 3.8
• Ubuntu 18.04.5 LTS

I am not a developer and I'm not very experienced with Docker or ELK so it may well be user error!

[duncan-ncc]

Hello Simon, Thanks for the report. We are looking into this and will provide an update once we have a solution.

Thanks, Duncan

[ipswichschool]

Any updates on this? Having the same issue with my install.

[splurggy]

any updates on this ???

[a-d-a-m-b]

Hello. We are aware of the issue, the root cause and have a mechanism to fix - we are looking at the easiest way of providing that solution to everyone

[ipswichschool]

Perfect. Thank you.

[splurggy]

is there any timescale on this ? as i need to showcase this install i am doing to some top brass - wont look good if there are errors all over the place - thanks dudes

[kcmark1]

Did the solution to this issue get published somewhere?

[a-d-a-m-b]

Hi @splurggy, @ipswichschool @kcmark1 @SimonAccordDS (and everyone else who might be reading)

A new branch was pushed to the repo last night (https://github.com/ukncsc/lme/tree/0.4-pre-release). It is a pre-release of the next version which includes a fix for this issue.

Details of how to upgrade are in the https://github.com/ukncsc/lme/blob/0.4-pre-release/docs/upgrading.md document, which provides some background to the issue, and how to resolve.

If you are willing to test and report back if things are resolved, that would be helpful. Whilst the code changes have passed our testing, we would still recommend making a snapshot of your logs if you are relying on them.

A full changelog includes:

• Updated to ELK stack 7.11.1
• Added quotes to logstash passwords to prevent issues with certain characters in passwords
• Added additional output to deploy script to guide users through various stages of the install/update/uninstall process and reduce bug reports
• Merged dashboard versions into single file managed by git and removed outdated visualisations
• Updated existing documentation to fix inconsistencies and clarify instructions
• Updated dashboards to ECS compliant fields
• Updated dashboards to resolve several existing issues with null fields and to clarify the purpose of the MITRE alerts dashboard
• Updated the WEC config to resolve duplicate query IDs, include additional events and store events in RenderedText format
• Removed deprecated OSMap functionality
• Migrated GeoIP enrichment from logstash filter to elastic pipeline
• Included index mapping template exported from winlogbeat-7.11.1 and customised for LME, and updated install script to deploy this automatically for new installs
• Added LME config file storing versioning and hostname info, created on install
• Automated initial dashboard install and fixed dashboard update script so that it correctly deploys latest dashboards from git if enabled
• Added new deploy script method to deploy new index mapping, pipeline updates, dashboard installs and write the config version to support manual upgrade for existing users
• Added "upgrading" documentation to walk users through the manual steps required when calling the above method
• Bug fixes for existing install function to correctly enable automatic updates for Ubuntu and to resolve vm.max_map_count configuration within Docker
• Bug fixes for the uninstall function to ensure it properly cleans all the existing files up from the Docker stack and leaves the swarm
• Added checks to ensure that LME is stored in the correct location (/opt/lme) and modified the deploy script so that it can be referenced from any working directory provided it's stored in the correct location
• Modified the existing update and dashboard update methods to only run when the LME version is set to 0.4, to prevent existing installations automatically updating without manual user intervention (as this requires re-indexing)
• Added alternative suggested sysmon.xml configuration option
• Updated copyright and authors list

[adam-ncc]

Closing this issue as it should be resolved now.