securityonion-elastic: improve support for custom ingest parsers

[dougburks]

From Slack discussion with Jim Hranicky:

there at least 2 ways to add custom Elasticsearch ingest pipelines today: (1) Simply add your new pipelines to the existing directory /etc/elasticsearch/ingest/. The caveat here is that if you change our existing parsers your changes will be overwritten on next upgrade. OR (2) Copy the existing directory /etc/elasticsearch/ingest to a new directory (for example /etc/elasticsearch/ingest-custom/) and then add a new setting to /etc/nsm/securityonion.conf: ELASTICSEARCH_INGEST_PIPELINES="/etc/elasticsearch/ingest-custom" The caveat here is that your new directory wouldn't automatically get any improvements that we make to the upstream parsers. In either case, you would then need to run sudo so-elasticsearch-pipelines for the changes to take effect.

Jim's response:

So I created /etc/elasticsearch/ingest-custom , symlinked the files in /etc/elasticsearch/ingest there, added my pipeline, and set the variable in /etc/nsm/securityonion.conf . That seems to work. For custom pipelines there does still need to be a way to add an output to /etc/logstash/conf.d.minimal, I'm doing that by hand for now. FYI. Also added /etc/elasticsearch/ingest-custom to salt

[defensivedepth]

Looks good to me

[dougburks]

Published: https://blog.securityonion.net/2020/02/zeek-301-elastic-686-and-cyberchef-9120.html