search

senecajs / seneca

A microservices toolkit for Node.js.
http://senecajs.org
MIT License
3.96k stars 311 forks source link

History ends up as a memory leak due to connection to timeout #857

Open wzrdtales opened 4 years ago

wzrdtales commented 4 years ago

History seems to have no disconnected clean up aside from the prune interval. This is directly connected with the timeout. This is pretty unhealthy, since even when these request are finished they will be kept in history way too long, which ends up in a high frequency service where you set this timeout higher to lower the failure rate by timeouts on high peak hours to completely hog and leak the memory. The application will become unresponsive and disfunctional.

Instead the history should reset the timelimit to a different value than timeout after the execution was successful and send back to the client.

rjrodger commented 4 years ago

Yeah this makes sense. How do we first define a (pseudo) test case to capture this?

wzrdtales commented 4 years ago

something like that

t (timeout): 60 * 1000 
set history prune interval to almost immediate
spawn n>1 requests, finishing work (nt1) in no time at all
spawn n>1 requests, finishing work (nt2) in a time window big enough to remain in the list
check history table after success of nt1 for length delayed by the prune interval and probably new cache/cleanup interval  -> expect to find only nt2 in list
cleanup
wzrdtales commented 4 years ago

see for a test quickly put together

https://github.com/senecajs/seneca/pull/861