Garbage collector problem

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Compile the attached program
2. Run it with `emu -pimage=256M -pmain=128M -pheap=256M a.dis`
3.

What is the expected output? What do you see instead?
The program is expected to consume at most (99+1+100)*64kb=13Mb of memory
and all temporary buffers are supposed to be eliminated by the garbage
collector.
I got `[A] Broken: "out of memory: heap"` error even with 256Mb of heap
which is 20 times bigger than really needed to complete this task.

What version of the product are you using? On what operating system?
Inferno fourth edition on Windows XP

Please provide any additional information below.
This example is part of real software which reads a file (which size is
unknown a priori) into memory until EOF. I only changed the loop over
sys->read to loop until 100 to avoid providing here a big file.

Original issue reported on code.google.com by 4239512@gmail.com on 5 Oct 2008 at 6:40

Attachments:

• a.b

[GoogleCodeExporter]

That particular example seems fine on Linux and Plan 9 with default heap size,
both with and without jit enabled. Does it really fail on Windows?
(I can't easily try that where I am.)

Original comment by Charles....@gmail.com on 5 Oct 2008 at 7:37

• Changed state: Checking

[GoogleCodeExporter]

> Does it really fail on Windows?
yes :(
Nice to hear it is good on Linux.
At least it seems to be an emulator bug and a bug of design.
I will try with Linux and maybe it would help me to localize the bug.

Original comment by 4239512@gmail.com on 5 Oct 2008 at 8:07

[GoogleCodeExporter]

there isn't any straightforward difference between Windows and Linux (in a
straightforward way) wrt garbage collection.
that's all done by portable code. (indeed the platform-specific code is tiny, 
except
for graphics, which is always different, and usually strangely horrendous.)
so it's still a little odd.
i'll boot up windows shortly to find out.

Original comment by Charles....@gmail.com on 5 Oct 2008 at 9:39

[GoogleCodeExporter]

The problem consists in Windows' VirtualAlloc() is not an equivalent of Linux's 
sbrk().
sbrk() extends allocated block and VirtualAlloc() allocs a new one wherever it 
wants.
So you can merge chains on Linux and cannot on Windows.
Moreover on Windows you get a lot of allocated blocks instead of one big on 
Linux
thus make impossible to malloc big arrays at place of freed smaller arrays.
I do not know how to solve the issue easily.
May be by preallocating all memory needed for the pools ?

Original comment by 4239512@gmail.com on 5 Oct 2008 at 10:33

[GoogleCodeExporter]

thank you. i'd just discovered that it did indeed fail rapidly under Windows,
and i realised that it was probably related to sbrk (since that's the only code
that's seriously different between the ports in this case), but i hadn't yet 
looked 
at the implementation to work out the details. i'll have a hunt round MSDN. 
there's 
probably a better primitive to use or a better way to use the primitives. i'm 
now 
only a little surprised it hasn't surfaced before (to my knowledge).

Original comment by Charles....@gmail.com on 5 Oct 2008 at 10:48

• Changed state: Accepted

[GoogleCodeExporter]

Here is the patch attached
And perhaps sbrk() should not be used on Linux as well.
Although there is no problem with a.b because it use only `heap` poll, 
allocating
memory for all three pool in only one growing block may cause interleaving of 
blocks
of different pools and the same "out of memory" error.
I think it is better to reserve contiguous address spaces for whole size of 
each pool
and then commit reserved memory instead of doing sbrk().

Original comment by 4239512@gmail.com on 6 Oct 2008 at 12:38

Attachments:

• win.patch

[GoogleCodeExporter]

I'd like to do away with the pool limits, so adding more preallocation is at 
best a
temporary fix, indeed a patch.
It would be better to change the underlying allocator and in fact i've got an
alternative i've tested but not yet installed. It would avoid some of this 
problem.
The test program's allocation pattern breaks up memory to make it hard to 
allocate
large blocks, so it runs out sooner or later -- keeping the pool contiguous 
extends
the time of the run, which is why it works on the other systems, but doesn't 
really
address the trouble. The new allocator is much faster. The catch is that it 
currently
is in a test bed and needs work to make its interface match the current one.

Original comment by Charles....@gmail.com on 6 Oct 2008 at 8:19

[GoogleCodeExporter]

Is the new allocator a closed product or it is possible to see it in svn?
Say in /testing branch ?

Original comment by 4239512@gmail.com on 6 Oct 2008 at 9:46

[GoogleCodeExporter]

Just another idea: maybe it would be better for long-run application to make 
Limbo's 
refs handles of relocable memory blocks instead of pointers ?

Original comment by 4239512@gmail.com on 10 Oct 2008 at 2:51

[GoogleCodeExporter]

hello,

it seems the same problem occurs on Mac OS X. 
The OS version I use is as follows:
Darwin julius-chrobaks-computer.local 8.11.1 Darwin Kernel Version 8.11.1: Wed 
Oct 10 18:23:28 PDT 2007; 
root:xnu-792.25.20~1/RELEASE_I386 i386 i386

Has anyone else experienced this problem on Mac? 

Original comment by julo.chr...@gmail.com on 1 Feb 2009 at 11:41