search

andremussche / scalemm

Fast scaling memory manager for Delphi
https://code.google.com/p/scalemm/
Other
98 stars 22 forks source link

memory is growing #25

Open Zeus64 opened 8 years ago

Zeus64 commented 8 years ago

Hello,

I want to understand a problem of memory growing. i m not sure if it's a bug of scalemm or not

We use scaleMM inside an ISAPI DLL (IIS 7), so we are not responsible of creating thread, nor closing them.

in this dll, we create some objects in the calling thread and after move them to another thread who is responsible to "recycle" them (clean memory, string, etc). so when another calling thread ask again to create an object, we first look in the "recycled queue" if their is no object available, if yes gave it to him, else let the calling thread create the object.

you understand, here lot of interthread memory

1/ what's happen if IIS close one thread? when the memory of the object (actually in the recycled queue ) created by this thread will be free ? when other thread will free the object ? 2/ why under scalemm the memory grow much more faster than on fastmm ? 3/ is their something like memory fragmentation that could explain why memory grow so fast? or it's simply a bad idea to recycle used object ? because cleaning properties of the object can cause memory fragmentation ?

andremussche commented 8 years ago

Hi,

I think there can be a problem when a thread is suspended by IIS and/or it is closed/terminated in a way that ScaleMM2 doesn't know about it: in these cases the interthread memory is marked as free by another thread but not cleaned/processed by the owner thread. Because FastMM does not have a pool per thread it does not suffer from these thread specific issues.

One thing on my wishlist is to create a garbage thread which will check these situations and take over the ownership so the (interthread)memory is released properly.

A quick fix would be to unlink the thread memory manager after each IIS request. I need to do some IIS stuff too so hopefully I can look at it soon.

Zeus64 commented 8 years ago

"One thing on my wishlist is to create a garbage thread which will check these situations and take over the ownership so the (interthread)memory is released properly."

i guess yes this would be good, but i have absolutely no idea how to help you with this, because as you say "ScaleMM2 is not easy to understand, because MM's are not easy!" :( but i think this is a very important feature. even a very low priority thread who do this check once every minute will be enalf i think.

"A quick fix would be to unlink the thread memory manager after each IIS request. I need to do some IIS stuff too so hopefully I can look at it soon."

i think the solution must not be dedicated only for IIS but for generic DLL who are not responsible of the thread creation/pause/destruction (like ISAPI on IIS yes)

andremussche commented 8 years ago

I made a generic solution by using a background cleanup thread, which will cleanup interthread memory and also detects terminated (non delphi) threads (so also usable for Windows Queue and IIS thread?)

Please take a look at a proof of concept here: https://github.com/andremussche/scalemm/tree/version2-Background-cleanup-thread-

Zeus64 commented 8 years ago

it's sound pretty good, i will take a look right now on it !

IgorIvkin commented 8 years ago

Hello, my name is Igor and i work with loki5100. I just tested the branch "Background-cleanup-thread" and it seems to be working incorrectly.

I tried to run two application: an ISAPI-dll on the address space of IIS mentioned above and also simple GUI-application based on VCL. It seems that main thread of both apps never got a control (for GUI for example no windows events were fired) and applications were hanging and consuming no CPU.

andremussche commented 8 years ago

thanks for testing, I could reproduce the problem and made a small fix

IgorIvkin commented 8 years ago

Hello, sorry for the delayed response. I was trying to test everything carefully.

My test environment is an ISAPI-DLL application working under IIS. I use some simple stress-test that requests this application from 10 threads at the same time.

The main result. It seems memory is quite stable now. It is growing fastly in the beginning but then becoming to be stable. Standard memory manager produces the size of working set around 130-135 Mb on the same test. ScaleMM2 produces around 250-260 Mb. And then working set size seems to be jumping around this size.

But there is another problem. It seems the branch "Background-cleanup-thread" is more slow. The same operation (to draw the same web-page on this test application) costs around 950 ms on trunk version of ScaleMM and around 1050-1100 ms on this branch. I will try to make some very simple application to compare performance and probably to reproduce this problem.

I'm ready to provide you any required measurements and additional info. Thank you by advance.

P.S. When i'm compiling i detected also some warnings and hints (i compile on Delphi XE4), probably you already saw them but anyway i suggest it will probably help to find some possible problems:

  1. smmGlobal.pas. Line 221. W1022 Comparison always evaluates to True.
  2. smmGlobal.pas. Line 665. W1035 Return value of function "ThreadProc" might be undefined.
  3. smmGlobal.pas. Line 662. H2164 Variable "FreeThread" is declated but never used in "ThreadProc".
andremussche commented 8 years ago

Thanks for testing and the feedback! Yes it is slower due to coarse grained locks for each operation, I will think about a faster way since it seems to work Ok now. And SMM2 is optimized for speed, not for smallest mem usage. Each thread has it's own buffers/pool so it will have an initial higher usage (and free small blocks cannot be directly used by another thread). And because it is a more or less experimental version i did not do a cleanup yet, but thanks for noticing!

andremussche commented 8 years ago

note: I have checked in a small change to make it much faster

IgorIvkin commented 8 years ago

Hello again! First of all I would like to express my appreciation for your job and your valuable time. We will test this branch a little more carefully but at the first look the performance is much better.

I reported about an operation that costs 1050-1100 ms on the branch, now it's 950-970, so almost the same to original ScaleMM.

Thank you again, i will try to heavily test the branch performance and then i will report you about my results.

IgorIvkin commented 8 years ago

Hello! I would like to inform you about our results. Our application has some internal cache of objects in memory and we was working to minimize memory consumption of this cache (one of the target was to make a little more comfortable conditions for ScaleMM reducing memory overhead bringing by our process), this way we were not responding this time.

So, results (it is an ISAPI-dll under IIS, compiled by Delphi XE4, running on Windows Server 2008R2):

Standard memory manager (XE4 so it is some variation of FastMM4).

ScaleMM (branch Background-cleanup-thread)

ScaleMM (trunk)

But current branch version of ScaleMM (Background-cleanup-thread) has some issue that is not reproducible with standard memory manager and trunk version of ScaleMM.

After some period (we reproduced it twice, first time it was around 1.5 hours, next time - 3-3.5 hours) working set of one of the worker processes of IIS fastly jumps (we never catched the sequential growth, so it seems to be a deal of few minutes) to 2.5 Gb (2 500+ Mb).

Is it possible that some changes in this branch could force this burst growth? Trunk version of ScaleMM and standard manager were tested during few days and this problem doesn't appears. Thank you by advance.

andremussche commented 8 years ago

Sorry for the delay, but I think I found the problem. I will test it tomorrow first and will commit the fix afterwards. Many thanks for testing and the feedback!

andremussche commented 8 years ago

commited the fix, including some small speed improvements

IgorIvkin commented 8 years ago

Hello! We tested the branch version during some time and it seems there is no more burst growing, thank you!

To be fully sure i will run automatic heavy loading test this weekend on the test machine and then i will share my results to you.

andremussche commented 8 years ago

Great! And what about the speed? It should be slightly faster (not as fast as I want it yet though)

andremussche commented 8 years ago

I just committed a new change, which should be AND fast AND safe :)

can you test and confirm it works in your heavy load situation?

IgorIvkin commented 8 years ago

Hello! Yes, sure, we will run the test this night and tomorrow i will return back to you with our results. Thank you!

jaclas commented 7 years ago

Toledius and Loki, we still waiting for test result! :-)