search

altangent / ccxws

WebSocket client for 38 cryptocurrency exchanges
MIT License
619 stars 186 forks source link

Concurrent zlib operations cause memory fragmentation #140

Closed purplesmoke05 closed 4 years ago

purplesmoke05 commented 4 years ago

Exchange Huobi

Subscription type Level2 orderbook updates

Describe the bug I made 300 concurrency subscription with "Huobi Level2 orderbook updates" for a while(10min). This has resulted in our docker containers crashing due to memory exhaustion.

To Reproduce Steps to reproduce the behavior:

  1. Make a large number of subscription in huobi
  2. Wait for a while

It would be that crash is caused by "zlib". There are some issues about "zlib" memory leak. eg. https://github.com/nodejs/node/issues/8871

bmancini55 commented 4 years ago

Thanks for submitting the issue!

Looks like as resolution for the zlib module they added better documentation: https://nodejs.org/docs/latest-v13.x/api/zlib.html#zlib_threadpool_usage_and_performance_considerations

Based on the documentation, under high load, the Huobi client code could cause memory fragmentation issues. Two possible solutions would be switching to the unzipSync or using a semaphore to restrict concurrent zlib operations.

I can probably get to this sometime in the next week or two, but am also happy to accept a pull request in taking a stab at it.

bmancini55 commented 4 years ago

A fix for this has been released in: https://github.com/altangent/ccxws/releases/tag/v0.31.1