search

mczajkow / mediamail

Allows email communication with Social Media (Twitter, etc.) digests
GNU General Public License v3.0
0 stars 0 forks source link

Improve Elastic Search Store Performance #4

Open mczajkow opened 3 years ago

mczajkow commented 3 years ago

It seems that Elastic Search can not receive tweet data at pace with what comes in to a track Twitter Bot. Explore ways to improve performance which might be native to ES such as grouping messages in one larger bundle or compression, etc.

Ultimate goal is to handle a data volume rate of whatever can be thrown at it but with this issue, start with acheiving at-par performance with

"queries" : {
    "tracks" : ["sick","covid","help","lonely","depress","hospital","pray","christmas","jesus"]
},
mczajkow commented 3 years ago

Removing high priority status. This was OBE a little with the introduction of filtering lower scored tweets ahead of storage.