search

telefonicaid / fiware-cygnus

A connector in charge of persisting context data sources into other third-party databases and storage systems, creating a historical view of the context
https://fiware-cygnus.rtfd.io/
GNU Affero General Public License v3.0
65 stars 105 forks source link

Trying to improve performance of cygnus with elasticsearch #2131

Open santcorc opened 2 years ago

santcorc commented 2 years ago

Hello everyone, I'm trying to use fiware cygnus with elasticsearch with orion. Once I made the setup and started the testing, I faced serious performance problems. To resolve them, I started looking at https://fiware-cygnus.readthedocs.io/en/1.7.1/cygnus-ngsi/installation_and_administration_guide/performance_tips/index.html, and saw that increasing the batch_size is a good way to avoid the congestion I have suffered, but when tried to increase the batch_size of elasticsearch in the agent.conf file I saw there is no option to increase that on it, while other options, like CKAN or Mongo have that option. Am I missing anything or is no way in elasticsearch to increase the batch_size?

Thank you in advance

fgalan commented 2 years ago

ElasticSearch sink was authored by @nmatsui (see PR https://github.com/telefonicaid/fiware-cygnus/pull/1523).

It would be great if he can provide feedback or support in this.