search

Mango-information-systems / twitto_be

twitto_be is a real-time tweets analytics dashboard
https://twitto.be
Other
12 stars 2 forks source link

runs out of memory under heavy load #138

Closed mef closed 4 years ago

mef commented 7 years ago

While the application behaves well under a moderate volume of tweets (.be, approx 25k tweets per day), a higher volume of tweets causes it to crash.

Server runs out of memory. node-persist file ends up corrupt as a consequence.

Possibly: something else causes the crash, then node-persist file gets corrupt, then pm2's aggressive restart of the app that keeps crashing causes the out of memory error.

This has been observed by @11digitlabs when launching twitto.es

mef commented 7 years ago

switching node-persist storage to async allowed the process to run well for approx 18 hours, handling 80k tweets.

storage.init({continuous:false, interval: 5000,

instead of 

storage.initSync({

Then the app crashed, without leaving track of the cause in the logs.

mef commented 4 years ago

Not relevant anymore in v4, closing this ticket