OOM killer wreaking havok when compcache is used

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Load compcache with a measurable amount of swap (say 50% of machine RAM)
2. Run several processes which will bring the machine deep into swap area
3.

What is the expected output? What do you see instead?

Expected - all processes work fine.
What I see - the OOM killer starts killing random jobs.

What version of the product are you using? On what operating system?

0.5.1, on the 2.6.16.

Please provide any additional information below.

The only solution which comes to my mind is to preallocate the necessary 
space.
Unfortunately, usually swapping begins when the machine is already loaded, 
and then dynamically requesting memory for compressed pages can cause the 
kernel to look call the OOM killer to free up pages.

Original issue reported on code.google.com by luna.sta...@gmail.com on 5 Feb 2009 at 9:27

[GoogleCodeExporter]

pre-allocation is not acceptable. With prealloc you might be forced to swap even
under workloads where system might not otherwise touch swap.

Unfortunately, we don't yet have a way to move pages allocated to compcache to
physical swap disk. This would have helped in your case. Nor do we have option 
to
forward to physical swap the pages which compress poorly (or not at all) - we 
simply
store these pages as-is. Basically these two things will resolve your issue but 
I
could not find any solution to these till now.

So, if you are badly hit by this issue, just use smaller compcache - 25% of RAM 
seems
to be a good value to me.

Thanks

Original comment by nitingupta910@gmail.com on 6 Feb 2009 at 12:17

[GoogleCodeExporter]

Hi Nitin,

Thanks for the quick response.
What really troubles me is that I'm not sure I can characterize what causes 
this, 
and so I'm wondering if the problem might happen at 25% RAM as well.
I'm assuming the machine is getting to 95% (or any other high number :)) memory 
load, and then it tries to allocate memory for swap. This causes the OOM killer 
to 
go forth and annoy me :). I'm thinking that, in theory, it might always happen, 
even 
if I set a 25% number.

Maybe a "simple" solution would be to have compcache own the disk-based swap, 
but 
then again, if it tries to allocate space from the kernel before going for the 
disk 
swap, OOM might still be loosed. I wonder - does the kernel have any 
"opportunistic" 
memory allocation which just returns null and doesn't go for OOM.

And of course - thanks for you time :)

Original comment by luna.sta...@gmail.com on 6 Feb 2009 at 1:44

[GoogleCodeExporter]

If machine is getting to 95% usage then OOM killer is simply doing its job :)

Regarding "opportunistic" alloc: if we return failure from compcache during 
write
(say alloc failure) then page has nowhere to go and it will stay in memory. 
Kernel
does not try to forward failed swap requests to another device.

Original comment by nitingupta910@gmail.com on 6 Feb 2009 at 2:36

[GoogleCodeExporter]

With backing swap device support now in 0.5.2, we can now forward swap requests 
to
given swap device. Also with memlimit parameter you now have better control 
over how
much memory compcache can use.

Otherwise, if you don't provide a backing swap device to compcache, there is 
nothing
we can really do :)

Closing this "bug" now. Thanks.

Original comment by nitingupta910@gmail.com on 12 Mar 2009 at 10:26

• Changed state: Fixed