idaholab / Malcolm

Malcolm is a powerful, easily deployable network traffic analysis tool suite for full packet capture artifacts (PCAP files), Zeek logs and Suricata alerts.
https://idaholab.github.io/Malcolm/
Other
327 stars 53 forks source link

investigate allowing specifying another index pattern for Arkime to query in addition to arkime_sessions3-* #423

Closed mmguero closed 3 months ago

mmguero commented 4 months ago

Now with #313, we can write the suricata and zeek logs to another index pattern. However, those logs aren't queryable in Arkime.

Arkime from what I can tell hard-codes the arkime_sessions3-* index pattern throughout the code. However, it maybe be possible to specify multiple index patterns to query:

GET /my-index-000001,my-index-000002/_search
{
  "query": {
    "match": {
      "user.id": "kimchy"
    }
  }
}

We are going to look into how feasible it would be to 1) specify via an arkime config.ini setting another index pattern to query and 2) pick up and use that index pattern and include it in the existing arkime queries. This change would need to be made to the upstream arkime repo and accepted there as a PR to be feasible for use in Malcolm, we're not interested in maintaining a downstream arkime fork.

mmguero commented 4 months ago

Another option: allowing arkime to use a specified alias to query rather than the arkime_sessions3* value.

supcom234 commented 4 months ago

Ideally it would be nice if Arkime could have a configurable param to accept multiple indexes or use an Elastic alias to combine the indexes.

Kassio filed a ticket with those Arkime folks which was subsequently closed. This may be an Arkime limitation. https://github.com/arkime/arkime/issues/2555

mmguero commented 3 months ago

See the PR work in progress for Arkime: arkime/arkime#2705

mmguero commented 3 months ago

With arkime/arkime#2705 merged this will make it into this Malcolm release (either via a small patch for v5.0.1 or in v5.0.2 once that's released).