search

senecajs / seneca-transport

Seneca micro-services message transport over TCP and HTTP.
MIT License
63 stars 45 forks source link

Seneca TCP transport consumes memory regularly, never frees it #146

Open SzybkiSasza opened 7 years ago

SzybkiSasza commented 7 years ago

For some considerable time we noticed that our Seneca instances using TCP transport as main way of communication consume memory in a very regular manner. Not so long time ago, due to another incompatibility/stability problems, we decided to migrate to Seneca 3.x from Seneca 2.1.0. However, this did not solve our core problem - we still se growing memory stamp on our instances and it happens only with TCP transport:

screen shot 2016-11-22 at 13 35 23

On the attached graph, you could see two services, sharing exactly the same codebase, initialised only with different transports. One, queue based (our own proprietary SQS transport, will source probably soon ;)) acts perfectly stable, whereas second one - TCP-based, behaves in a strange, yet pretty regular fashion. The drop-downs are obviously restarts.

On 11/18 we introduced Seneca 3.x to this particular service - you could notice that for some reason this saw-shaped characteristic got... Smoothed :) Still, it increases super-regularly, every hour by the same amount of memory.

Instances are hosted EC2 containers. The same behaviour is observed locally, in long-term runs. I'd like to add that our microservice isn't a very complicated one and all the act handlers are stateless (apart from MongoDB connection, but this one is present on non-TCP ones as well).

panva commented 7 years ago

Hello @SzybkiSasza, were you able to resolve this issue whilst still using TCP transport?

andybar2 commented 7 years ago

I'm facing this same issue. Is there a solution without changing the transport?

wzrdtales commented 6 years ago

Hey together,

I currently work on a new transport and while that I am getting some bits from this driver. So for me it seems this issue is the result of the same as the following bug #114 . For every send a new reconnect gets created, that never gets destroyed, which is one the reason for the memory leak and two the reason for the memory duplication.