support both ElasticSearch and OpenSearch output

[mmguero]

We've had a number of interested parties wanting to be able to store logs in both OpenSearch and ElasticSearch.

The index store that is managed by Malcolm will always (for the foreseeable future) be OpenSearch. This is due 1) to the licensing changes that initially caused us to go this direction and 2) to the features in Elasticsearch that would only be available to "premium" subscribers that we can get in open-source OpenSearch.

However, because we already support opensearch instances that are "local" vs. "remote" the code is already in place in Malcolm to use another data store into which to index the data. As of Q3 2023, the APIs between the two projects are still pretty similar.

So what I'm envisioning is this:

• During setup, if the user answers 'n' to question "Should Malcolm use and maintain its own OpenSearch instance? (Y/n)", then they'd be able to select
  • OpenSearch (remote)
  • ElasticSearch (remote)

Should they choose Elasticsearch, Malcolm won't run its opensearch container (the same way it doesn't today if they choose "OpenSearch (remote)") and it also won't run the dashboards container (since OpenSearch Dashboards isn't going to run against ElasticSearch).

Let's be clear: malcolm won't run an instance of ElasticSearch or Kibana. But it will be able to index the network data into an ElasticSearch instance and the user would be able to see that data in their own Kibana.

Here are a few areas we'd need to look at:

• the scripts used by dashboards-helper
• index templates
  • are they compatible? if not, how is the best way to make them so?
• dashboards
  • Can we programmatically convert them on-the-fly for creation in Kibana? I'd rather not have to maintain two sets
• the Malcolm API
• configuring Arkime forwarding for Malcolm and for Hedgehog Linux
• some of the non-network data stuff in filebeat
• logstash output pipeline

We shouldn't need to worry about:

• keystores
• Kibana itself

Honestly it's not too much that deals directly with the opensearch API. It's pretty easy to determine where those places are: since we're already handling remote opensearch instances, just look anywhere in the docker-compose file where we're referencing .opensearch.primary.curlrc and that is a place that is talking to OpenSearch, and as such a place we'd need to support ElasticSearch.

[pleasantencounter]

this would be very useful, thank you.

[mmguero]

I've got this mostly working in my development fork. At this point it's a lot of heavy testing and writing documentation, but it should work in the next release.

[mmguero]

things I still need to figure out:

• the Alerts - Tags visualization looks really weird
• ~metrics forwarders from fluent-bit aren't getting indexed for some reason~ (I just didn't have them enabled :/)

things I'm going to live with:

• the sankey visualization won't work
• the region map visualization won't work
• the links from kibana to netbox/arkime and from arkime -> "dashboards" won't work (maybe I can trim them out of dashboards at least)

[mmguero]

This all seems to be working. I'll release for v23.10.0 and then open individual issues if they crop up later.