Migration to a modular architecture

[dgarros]

As we are adding support for more features and input formats in OpenNTI, the size of the container is becoming bigger and bigger, also it's not always ideal to rebuild everything if only one part needs to be upgraded

Considering that all components are communicating over IP inside the container, it's very easy to break down the architecture into multiple smaller containers.

For example

• 1 container for the database + GUI
• 1 container for the Data Collection Engine
• 1 container for the Data Streaming Collector
• 1 container for Syslog
• 1 container for Kapacitor As container don't have overhead in resources, the among of resources consumed globally won't be more important. With this model it will become very simple to add new type of input (snmp, netflow, etc ..)

To keep it as simple to deploy and hide the number of containers, the idea is to use docker compose by default. it's a very easy solution to create a multi-container architecture without adding complexity. Also, if needed it become possible to create multiple containers of the same type to distribute the load. For Syslog or JTI for example.

The transition should be transparent for most people as docker compose is installed by default with docker in most cases.

I started to create a standalone container for the Data Streaming Collector, it's available here https://github.com/Juniper/open-nti-input-jti I'll create a branch on this project to integrate it soon

Ideas, comments, help is welcome

[dgarros]

I created a branch and a pull request to share my progress https://github.com/Juniper/open-nti/pull/47

Let me know if you are interested to test/help

[dgarros]

It's been implemented