search

zeroc-ice / datastorm

Data centric pub/sub framework based on Ice
GNU General Public License v2.0
13 stars 7 forks source link

memory leak upon node-crashing #9

Closed zengdaiwei closed 3 years ago

zengdaiwei commented 3 years ago

At first, thanks for building this wonderful project which helps me a lot! Everything worked well with DataStorm, but I did encounter a serious memory leak problem. I tried to locate the problem, but my understanding stopped at leak location "makeXXXKeyReader"/"makeXXXKeyWriter", so I build a daemon for this problem, and hope for your insight & help. The main idea of my daemon is to automatically release specified-key writer when last specified-key reader disconnected.

How to reproduce the problem: 1, build Reader.exe & Writer.exe from https://github.com/zengdaiwei/TestDataStormMemoryLeak.git 2, run Writer.exe 3, run Reader with bash: "while true; do ./Reader & sleep 1s; kill %1; wait; sleep 3s; done" 4, now you can see there is a continuous memory leak from Writer.exe

Thanks

bentoi commented 3 years ago

Thanks for the bug report. I indeed found a leak for the dead session of the reader. I've pushed a fix on the main branch. Try it out!

zengdaiwei commented 3 years ago

Thanks for trying to fix this bug. But I found the leaking still exist after I tried the fix. The Writer.exe leaked more than 5MB in 10 minutes, while the Reader.exe ran & crashed with "while true; do ./Reader & sleep 1s; kill %1; wait; sleep 3s; done". I recommend you to try my test daemon. Thanks, Sincerely.

bentoi commented 3 years ago

Ok I didn't get such a big leak when running your test case. I only tried it on macOS, I'll try with Windows.

bentoi commented 3 years ago

I pushed another fix. However, I need to look further into it, it looks like it's still leaking on Windows in particular with a debug build. Also note that the writer tries to re-establish the connection up to 6 times with increasing delays before giving up. If the failure rate is higher than the time needed to reap the session of a failed node, it will leak. You will need to tune to DataStorm.Node.Retry* properties to ensure the session for the failed node is released quickly. You can for example set DataStorm.Node.RetryCount=0 to disable retries. See https://doc.zeroc.com/datastorm/latest/property-reference/datastorm-node for a description of the retry properties.

bentoi commented 3 years ago

I've pushed an additional fix. Please try again.

zengdaiwei commented 3 years ago

After testing with different Reader-crashing strategies, and multiple crashing-Readers, for more than 1 hour, I haven't found a continuous memory leaking.

But after about-12-simultaneous working & crashing Reader processes all existed, the Writer.exe's memory did not restore from peak 43MB to starting 5MB,

which does not seem like a problem, for I can see a large chunk of the memory is cached by Ice.

After all, Thanks for your wonderful work.

At 2021-09-02 18:20:55, "Benoit Foucher" @.***> wrote:

I've pushed an additional fix. Please try again.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.