search

telefonicaid / fiware-cygnus

A connector in charge of persisting context data sources into other third-party databases and storage systems, creating a historical view of the context
https://fiware-cygnus.rtfd.io/
GNU Affero General Public License v3.0
64 stars 105 forks source link

Bug in NGSIElasticsearchSink? #2188

Open santcorc opened 2 years ago

santcorc commented 2 years ago

Hi to everyone, I do not know if this could be consider a bug or a different way of implementation adapted to other scenarios, due to according to implementation it is a searched feature. I am talking about the Index convention number 6

append the created <date> such as yyyy.mm.dd.

In the implementation I made I had to modify the code to eliminate this because in long-term deployments Elasticsearch would crash due to the maximum number of indexes is reached. According to the default options of elasticsearch, each elasticsearch node have 1000 shards and each index need 2, so in a single-node-cluster only 500 indexes could be created. This, following the expected behaviour would be filled in a short amount of time, for example having 20 entities being updated everyday would take less than one month to not let create more indexes. Even if increasing the maximum number of shards which is not recommended by elasticsearch or adding more nodes, creating a multi-node-cluster would only be a temporal fix and is not scalable in long term scenarios.

If neccessary, I could send a pull request with the modified NGSIElasticsearchSink that is being used in my actual deployment, eliminating the addition of the date at the end of the created indexes.

AlvaroVega commented 2 years ago

Thanks @santcorc Any pull request that you can do in order to fix a bug is welcome!

santcorc commented 2 years ago

Thanks for the fast reply @AlvaroVega But the thing is that according to the documentation https://fiware-cygnus.readthedocs.io/en/latest/cygnus-ngsi/flume_extensions_catalogue/ngsi_elasticsearch_sink.html it is a searched feature. If this is changed that documentation should also be modified.

fgalan commented 2 years ago

Your PR could include also modifications in the documentation. Thanks!