New Kibana index configuration

[ChaDup]

We can now have a direct subscription of Orion to Logstash with an http connector. Every new data is pushed on Logstash (entrance of elasticsearch).

For this new configuration, we have 2 options to discuss :

1- Every data on Orion, is arriving in the same index

We can configure logstash so that every data arriving on Orion, is also arriving on logstash. It is easier to configure because even when we create new data pipeline, with new service path, it is automatically pushed. The problem is that every data is on the same index, and if we want to restrain access to index, it will not be possible. We can only filter by service path or sensor name to deal with the data.

same index - filter by service path same index - filter by sensor name

2- Each data pipeline on Orion (service path) is arriving in separate index.

We can configure logstash so that every service path has its own index. It is less easier to configure, because if we have 100 services path and data pipeline, we must have 100 configuration files and index. Also, every time we have a new pipeline/service path, the configuration must be created on logstash, it will not be automatic. But if we can restrain access to index, it will be easier with this configuration.

different index

So the big question - that will help to choose - is : Who will access to Kibana platform and Elasticsearch index ?

The visualisations, will be embedded on platforms or websites, with restreint access. But the platforms will be accessible only by developpers. Is it a big problem that developpers can access all the data ? They are not end-users, and maybe the privacy issues don't apply here.

The answer to this question will help to choose between option 1 or 2

[bures]

I would generally go with one service path = one index.

It will be much easier to provide security. Also, I believe we don't have scenarios where data from different service paths should be aggregated on the side of Elasticsearch. Or do we?

The index is created automatically by feeder. We will just have to automate the configuration of feeder. Also, we can extend feeder in such a way that it scans the available service paths and then configures itself automatically. @cdupont already extended Orion to return the service path. What do you think?

[ChaDup]

Yes we can have the utility to compare different data from different location (different service path) but with the same application. But anyway with timelion plugin I know how to compare and display data from different indexes now

[cdupont]

I also agree one service path = one index is a good way to go. @bures what is the status of the feeder? Could you put it on Waziup repo so we can integrate it?

[cdupont]

closing as too old