search

robcowart / elastiflow

Network flow analytics (Netflow, sFlow and IPFIX) with the Elastic Stack
Other
2.49k stars 598 forks source link

Deploying elastiflow 4 without deleting old data #688

Closed photonofcode closed 3 years ago

photonofcode commented 3 years ago

Hello.

I want to be certain I understand the options for installing Elastiflow 4.0.

I am wondering why the 4.0 install guide mentions deleting all old elastiflow 3.x indexes ? Elastiflow 4 uses ECS so, in theory, it shouldn't affect or conflict with any previous indexes. Kibana might have some conflicts but I cannot immediate think on any.

What I'd like to do is install Elastiflow 4 to the existing production cluster without deleting the existing indexes. Then I will use a custom re-index strategy to import the flows into the new ECS format. (maybe it will involve using backup flow data from outside Elastic) Then I will delete the old elastiflow-3 indexes. The goal here is to keep the old data available for search until the backlog is imported.

Why would the approach above not work ?

Thanks

photonofcode commented 3 years ago

After reviewing the ES template with a diff to 3.5.3 I see where the differences are, although I don't fully understand how they would break 3.5.3 or possibly complicate a successful install of 4.0.1. Same for Kibana.

So as the INSTALL.md mentioned, it's non-trivial to maintain a working parallel instance of 3.5.3 on the same cluster.

Regardless, thanks to all who make this software possible.

robcowart commented 3 years ago

@photonofcode your strategy is sound. Reindexing the old data, with the necessary conversion of fields from the 3.x schema to the 4.x is the way to go if you need to convert your old data. It is however a very detailed process, that can be challenging to many when no tooling and specific instructions are provided. While no tooling for a migration has been provided, the ElastiFlow version is in each record as well as then index names. This was done to assist in the migration process for anyone wanting to do so on their own.