runtime: windows DLL mappings interfere with 32-bit memory allocation

[gopherbot]

by zippoxer:

Before filing a bug, please check whether it has been fixed since
the latest release: run "hg pull", "hg update default", rebuild, and
retry
what you did to
reproduce the problem.  Thanks.

What steps will reproduce the problem?
1. Installing gomingw installer for build of release r60.1
2. Executing make, then make install in the (attached to this issue) package's directory.
3. Testing the installed package by calling Now().

What is the expected output?
The x position of the cursor.

What do you see instead?
An error:
>> throw: runtime: cannot reserve arena virtual address space

Which compiler are you using (5g, 6g, 8g, gccgo)?
8g

Which operating system are you using?
Windows 7

Which revision are you using?  (hg identify)
I ran "Installer for Win32 (experimental) build of release.r60.1"
(hd identify doesn't cooperate, sorry)

Please provide any additional information below.

Attachments:

1. somecgopackage.go (188 bytes)
2. Makefile (122 bytes)

[alexbrainman]

Comment 1:

I can't reproduce your problem on Windows 7:
$ cat Makefile
include $(GOROOT)/src/Make.inc
TARG=a
CGOFILES=\
        a.go\
include $(GOROOT)/src/Make.pkg
$ cat a.go
package a
/*
#include <windows.h>
int xPlease() {
        POINT p;
        GetCursorPos(&p);
        return p.x;
}
*/
import "C"
func Now() {
        println("I got that. It's: ", int(C.xPlease()))
}
$ cat a_test.go
package a_test
import (
        "a"
        "testing"
)
func TestA(t *testing.T) {
        a.Now()
}
$ make test
gotest
rm -f _test/a.a
CGOPKGPATH= cgo --  a.go
touch _obj/_cgo_run
8g  -o _gotest_.8  _obj/a.cgo1.go _obj/_cgo_gotypes.go
8c -FVw -Ic:/MinGW/go/pkg/windows_386 -I . -o "_cgo_defun.8" _obj/_cgo_defun.c
gcc -m32 -I . -g -fPIC -O2 -o _cgo_main.o -c   _obj/_cgo_main.c
_obj/_cgo_main.c:1:0: warning: -fPIC ignored for target (all code is position
independent)
gcc -m32 -I . -g -fPIC -O2 -o a.cgo2.o -c   _obj/a.cgo2.c
_obj/a.cgo2.c:1:0: warning: -fPIC ignored for target (all code is position independent)
gcc -m32 -I . -g -fPIC -O2 -o _cgo_export.o -c   _obj/_cgo_export.c
_obj/_cgo_export.c:1:0: warning: -fPIC ignored for target (all code is position
independent)
gcc -m32 -g -fPIC -O2 -o _cgo1_.o _cgo_main.o a.cgo2.o _cgo_export.o
cgo -dynimport _cgo1_.o >_obj/_cgo_import.c_ && mv -f _obj/_cgo_import.c_
_obj/_cgo_import.c
8c -FVw -I . -o "_cgo_import.8" _obj/_cgo_import.c
rm -f _test/a.a
gopack grc _test/a.a _gotest_.8  _cgo_defun.8 _cgo_import.8 a.cgo2.o _cgo_export.o
I got that. It's:  1115
PASS
$

Owner changed to @alexbrainman.

Status changed to WaitingForReply.

[gopherbot]

Comment 2 by zippoxer:

I have same files and I typed the same commands shown in your comment, but I get:
$ make test
gotest
rm -f _test/a.a
CGOPKGPATH= cgo --  a.go
touch _obj/_cgo_run
8g  -o _gotest_.8  _obj/a.cgo1.go _obj/_cgo_gotypes.go
8c -FVw -IC:/Go/pkg/windows_386 -I . -o "_cgo_defun.8" _obj/_cgo_defun.c
gcc -m32 -I . -g -fPIC -O2 -o _cgo_main.o -c   _obj/_cgo_main.c
_obj/_cgo_main.c:1:0: warning: -fPIC ignored for target (all code is position in
dependent)
gcc -m32 -I . -g -fPIC -O2 -o a.cgo2.o -c   _obj/a.cgo2.c
_obj/a.cgo2.c:1:0: warning: -fPIC ignored for target (all code is position indep
endent)
gcc -m32 -I . -g -fPIC -O2 -o _cgo_export.o -c   _obj/_cgo_export.c
_obj/_cgo_export.c:1:0: warning: -fPIC ignored for target (all code is position
independent)
gcc -m32 -g -fPIC -O2 -o _cgo1_.o _cgo_main.o a.cgo2.o _cgo_export.o
cgo -dynimport _cgo1_.o >_obj/_cgo_import.c_ && mv -f _obj/_cgo_import.c_ _obj/_
cgo_import.c
8c -FVw -I . -o "_cgo_import.8" _obj/_cgo_import.c
rm -f _test/a.a
gopack grc _test/a.a _gotest_.8  _cgo_defun.8 _cgo_import.8 a.cgo2.o _cgo_export
.o
throw: runtime: cannot reserve arena virtual address space
gotest: "./8.out.exe" failed: exit status 2
make: *** [test] Error 2
===================
Maybe cgo doesn't use the right C++ compiler? (I don't really understand these things, I
just installed mingw with it's compiler).
My UAC (security settings) are turned off, btw.

[alexbrainman]

Comment 3:

I do not what your problem is. But before we do anything else, maybe you can tell us why
this error is happening. If you could change your $GOROOT/src/pkg/runtime/windows/mem.c
file like this:
diff -r 6aa111bd2d11 src/pkg/runtime/windows/mem.c
--- a/src/pkg/runtime/windows/mem.c Tue Oct 04 15:07:28 2011 +1100
+++ b/src/pkg/runtime/windows/mem.c Wed Oct 05 14:21:57 2011 +1100
@@ -55,7 +55,12 @@
        return v;

    // Next let the kernel choose the address.
-   return runtime·stdcall(runtime·VirtualAlloc, 4, nil, n, (uintptr)MEM_RESERVE,
(uintptr)PAGE_EXECUTE_READWRITE);
+   v = runtime·stdcall(runtime·VirtualAlloc, 4, nil, n, (uintptr)MEM_RESERVE,
(uintptr)PAGE_EXECUTE_READWRITE);
+   if(v == nil) {
+       runtime·printf("VirtualAlloc failed with errno=%d\n", runtime·getlasterror());
+       runtime·throw("VirtualAlloc");
+   }
+   return v;
 }

 void
Then you would need to rebuild "runtime" package, like
cd $GOROOT/src/pkg/runtime
make clean install
Then re-run your program and tell us what errno VirtualAlloc function returns. Thank you.
Alex

[gopherbot]

Comment 4 by zippoxer:

Compiling with your patch added these lines:
VirtualAlloc failed with errno=8
throw: VirtualAlloc
Hope this helps.

[alexbrainman]

Comment 5:

errno=8 is ERROR_NOT_ENOUGH_MEMORY, so it sounds like your program fails to allocate
required block of address space, perhaps, because there is not enough room to do it.
I take it, some simple program like:
package main
func main() {
        println("Hello")
}
works. Right?
Could you, please revert change to your $GOROOT/src/pkg/runtime/windows/mem.c file, and
apply this one, instead:
diff -r 6aa111bd2d11 src/pkg/runtime/windows/mem.c
--- a/src/pkg/runtime/windows/mem.c Tue Oct 04 15:07:28 2011 +1100
+++ b/src/pkg/runtime/windows/mem.c Thu Oct 06 12:21:13 2011 +1100
@@ -48,14 +48,25 @@
 void*
 runtime·SysReserve(void *v, uintptr n)
 {
+   runtime·printf("SysReserve: v=%p n=%d\n", v, n);
    // v is just a hint.
    // First try at v.
    v = runtime·stdcall(runtime·VirtualAlloc, 4, v, n, (uintptr)MEM_RESERVE, (uintptr)PAGE_EXECUTE_READWRITE);
-   if(v != nil)
+   if(v != nil) {
+       runtime·printf("SysReserve: hint worked v=%p\n", v);
        return v;
+   } else {
+       runtime·printf("SysReserve: hint failed with errno=%d\n", runtime·getlasterror());
+   }

    // Next let the kernel choose the address.
-   return runtime·stdcall(runtime·VirtualAlloc, 4, nil, n, (uintptr)MEM_RESERVE,
(uintptr)PAGE_EXECUTE_READWRITE);
+   v = runtime·stdcall(runtime·VirtualAlloc, 4, nil, n, (uintptr)MEM_RESERVE,
(uintptr)PAGE_EXECUTE_READWRITE);
+   if(v == nil) {
+       runtime·printf("SysReserve: accepting any address failed with errno=%d\n",
runtime·getlasterror());
+       runtime·throw("VirtualAlloc");
+   }
+   runtime·printf("SysReserve: accepting any address worked v=%p\n", v);
+   return v;
 }

 void
Then re-run you test again and show us the output. Please, also run little program above
and send its output too. I would like to compare the two.
I still do not know what is going on, perhaps, when you link extra mingw windows
functions, they use some extra address space, maybe they load some extra dlls, that we
have not enough room to move afterwards.
Thank you.
Alex

[gopherbot]

Comment 6 by zippoxer:

Printing "Hello" works of course, or I wasn't moving on to Windows API :P
I have 3.49GB usable memory and I successfully called Windows API with C++ (using mingw
gcc).
Okay I applied your patch and here is the output for the test:
$ make test
gotest
SysReserve: v=0x920118 n=805306368
SysReserve: hint failed with errno=487
SysReserve: accepting any address worked v=0x38020000
rm -f _test/a.a
rm -f _test/a.a
gopack grc _test/a.a _gotest_.8  _cgo_defun.8 _cgo_import.8 a.cgo2.o _cgo_export
.o
SysReserve: v=0x8fa1e0 n=805306368
SysReserve: hint failed with errno=487
SysReserve: accepting any address failed with errno=8
throw: VirtualAlloc
gotest: "./8.out.exe" failed: exit status 2
make: *** [test] Error 2
And for the little program:
$ 8.out
SysReserve: v=0x832e58 n=805306368
SysReserve: hint failed with errno=487
SysReserve: accepting any address worked v=0x38020000
Hello

[alexbrainman]

Comment 7:

I am running out of ideas. :-)
The problem here is not the amount of memory you have in your computer, but free block
of address space available in your process. Go memory manager is designed to use 1 big
block of address space (see that request for 805306368 bytes). It asks OS to reserve
single contiguous block of addresses for future use. No real memory is associated with
these addresses at the start, but will get assigned as addresses are used by the
program. The problem here is that OS refuses to allocate this 805306368 bytes block. I
don't know why. Perhaps, there is no contiguous block of such size at the time, because
memory is used by other parts of your process (dlls and such).
What I would do next is put a bug in mem.c: replace line with
"runtime·throw("VirtualAlloc");" with "*(int32*)0 = 0;". This will raise an exception
at the right moment in your program. If you run it under debugger (I use
http://www.ollydbg.de/), you should be able to see memory map of your process, this will
give you a clue if there is free block of addresses or not and why. Here is my run:
SysReserve: v=0xa00000 n=805306368
SysReserve: hint failed with errno=487
SysReserve: accepting any address worked v=0xd30000
here is my map:
Memory map
Address   Size      Owner            Section    Contains                 Type  Access  
Initial >Mapped as
00010000  00001000 >                >           Environment              Priv >RW      
RW
00020000  00001000 >                >           Process Parameters       Priv >RW      
RW
0005F000  00001000 >                >                                    Priv >RW 
Guar>RW  Guar>
00060000  00010000 >                >           Stack of main thread     Priv >RW      
RW
00070000  00006000 >                >                                    Priv >RW      
RW
00170000  00003000 >                >                                    Map  >RW      
RW
00180000  00016000 >                >                                    Map  >R       
R        C:\WINNT\system32\unicode.nls
001A0000  0002F000 >                >                                    Map  >R       
R        C:\WINNT\system32\locale.nls
001D0000  00041000 >                >                                    Map  >R       
R        C:\WINNT\system32\sortkey.nls
00220000  00004000 >                >                                    Map  >R       
R        C:\WINNT\system32\sorttbls.nls
00230000  00008000 >                >                                    Map  >R E     
R E
002F0000  00002000 >                >                                    Map  >R E     
R E
00300000  00043000 >                >           GDI handles              Map  >R        R
00400000  00001000 >8_out           >           PE header                Img  >R       
RWE Copy>
00401000  000E9000 >8_out           >.text      Code                     Img  >R E     
RWE Copy>
004EA000  00421000 >8_out           >.data      Data                     Img  >RW 
Copy>RWE Copy>
0090B000  00001000 >8_out           >/4                                  Img  >R       
RWE Copy>
0090C000  0000E000 >8_out           >/18                                 Img  >R       
RWE Copy>
0091A000  0000A000 >8_out           >/30                                 Img  >R       
RWE Copy>
00924000  0002B000 >8_out           >/43                                 Img  >R       
RWE Copy>
0094F000  00010000 >8_out           >/55                                 Img  >R       
RWE Copy>
0095F000  00003000 >8_out           >/71                                 Img  >R       
RWE Copy>
00962000  00001000 >8_out           >/87                                 Img  >R       
RWE Copy>
00963000  00001000 >8_out           >/102                                Img  >R       
RWE Copy>
00964000  00001000 >8_out           >.idata     Imports                  Img  >RW 
Copy>RWE Copy>
00965000  00001000 >8_out           >.edata     Exports                  Img  >R       
RWE Copy>
00966000  00001000 >8_out           >.symtab                             Img  >R       
RWE Copy>
00970000  000B5000 >                >                                    Map  >R E     
R E
00C70000  00001000 >                >                                    Priv >RW      
RW
00C80000  00001000 >                >                                    Priv >RW      
RW
00C90000  00001000 >                >                                    Priv >RW      
RW
00D10000  00005000 >                >                                    Priv >RW      
RW
00D20000  00002000 >                >                                    Map  >R       
R        C:\WINNT\system32\ctype.nls
77570000  00001000 >winmm           >           PE header                Img  >R       
RWE Copy>
77571000  0001F000 >winmm           >.text      Code,imports,exports     Img  >R E     
RWE Copy>
77590000  00005000 >winmm           >.data      Data                     Img  >RW 
Copy>RWE Copy>
77595000  00009000 >winmm           >.rsrc      Resources                Img  >R       
RWE Copy>
7759E000  00002000 >winmm           >.reloc     Relocations              Img  >R       
RWE Copy>
77D30000  00001000 >RPCRT4          >           PE header                Img  >R       
RWE Copy>
77D31000  00067000 >RPCRT4          >.text,.orp>Code,imports,exports     Img  >R E     
RWE Copy>
77D98000  00002000 >RPCRT4          >.data      Data                     Img  >RW      
RWE Copy>
77D9A000  00001000 >RPCRT4          >.rsrc      Resources                Img  >R       
RWE Copy>
77D9B000  00004000 >RPCRT4          >.reloc     Relocations              Img  >R       
RWE Copy>
77E10000  00001000 >USER32          >           PE header                Img  >R       
RWE Copy>
77E11000  00052000 >USER32          >.text      Code,imports,exports     Img  >R E     
RWE Copy>
77E63000  00001000 >USER32          >.data      Data                     Img  >RW      
RWE Copy>
77E64000  00008000 >USER32          >.rsrc      Resources                Img  >R       
RWE Copy>
77E6C000  00003000 >USER32          >.reloc     Relocations              Img  >R       
RWE Copy>
77F40000  00001000 >GDI32           >           PE header                Img  >R       
RWE Copy>
77F41000  00038000 >GDI32           >.text      Code,imports,exports     Img  >R E     
RWE Copy>
77F79000  00001000 >GDI32           >.data      Data                     Img  >RW      
RWE Copy>
77F7A000  00001000 >GDI32           >.rsrc      Resources                Img  >R       
RWE Copy>
77F7B000  00002000 >GDI32           >.reloc     Relocations              Img  >R       
RWE Copy>
77F80000  00001000 >ntdll           >           PE header                Img  >R       
RWE Copy>
77F81000  0004D000 >ntdll           >.text,ECOD>Code,exports             Img  >R E     
RWE Copy>
77FCE000  00003000 >ntdll           >.data      Data                     Img  >RW 
Copy>RWE Copy>
77FD1000  00001000 >ntdll           >EDATA                               Img  >RW 
Copy>RWE Copy>
77FD2000  00027000 >ntdll           >.rsrc      Resources                Img  >R       
RWE Copy>
77FF9000  00003000 >ntdll           >.reloc     Relocations              Img  >R       
RWE Copy>
78000000  00001000 >msvcrt          >           PE header                Img  >R       
RWE Copy>
78001000  00031000 >msvcrt          >.text      Code                     Img  >R E     
RWE Copy>
78032000  00008000 >msvcrt          >.rdata     Imports,exports          Img  >R       
RWE Copy>
7803A000  00007000 >msvcrt          >.data      Data                     Img  >RW 
Copy>RWE Copy>
78041000  00001000 >msvcrt          >.rsrc      Resources                Img  >R       
RWE Copy>
78042000  00003000 >msvcrt          >.reloc     Relocations              Img  >R       
RWE Copy>
7C2D0000  00001000 >ADVAPI32        >           PE header                Img  >R       
RWE Copy>
7C2D1000  0005A000 >ADVAPI32        >.text      Code,imports,exports     Img  >R E     
RWE Copy>
7C32B000  00004000 >ADVAPI32        >.data      Data                     Img  >RW 
Copy>RWE Copy>
7C32F000  00002000 >ADVAPI32        >.rsrc      Resources                Img  >R       
RWE Copy>
7C331000  00004000 >ADVAPI32        >.reloc     Relocations              Img  >R       
RWE Copy>
7C340000  00001000 >Secur32         >           PE header                Img  >R       
RWE Copy>
7C341000  0000B000 >Secur32         >.text      Code,imports,exports     Img  >R E     
RWE Copy>
7C34C000  00001000 >Secur32         >.data      Data                     Img  >RW      
RWE Copy>
7C34D000  00001000 >Secur32         >.rsrc      Resources                Img  >R       
RWE Copy>
7C34E000  00001000 >Secur32         >.reloc     Relocations              Img  >R       
RWE Copy>
7C570000  00001000 >KERNEL32        >           PE header                Img  >R       
RWE Copy>
7C571000  0005A000 >KERNEL32        >.text      Code,imports,exports     Img  >R E     
RWE Copy>
7C5CB000  00004000 >KERNEL32        >.data      Data                     Img  >RW      
RWE Copy>
7C5CF000  00051000 >KERNEL32        >.rsrc      Resources                Img  >R       
RWE Copy>
7C620000  00004000 >KERNEL32        >.reloc     Relocations              Img  >R       
RWE Copy>
7F6F0000  00007000 >                >                                    Map  >R E     
R E
7FFB0000  00024000 >                >           Code pages               Map  >R        R
7FFDE000  00001000 >                >           Data block of main threa>Priv >RWE     
RWE
7FFDF000  00001000 >                >           Process Environment Bloc>Priv >RWE     
RWE
7FFE0000  00001000 >                >           User Shared Data         Priv >R        R
As you can see, OS gave me address 0xd30000 which is right after end of program
00D20000  00002000 >                >                                    Map  >R       
R        C:\WINNT\system32\ctype.nls
and before start of first dll
77570000  00001000 >winmm           >           PE header                Img  >R       
RWE Copy>
Perhaps, you will see that there is no hole big enough to fit 805306368 bytes.
Otherwise, OS is lying when returns ERROR_NOT_ENOUGH_MEMORY and there is a different
reason for failure. Perhaps, it is security related.
Alex

[gopherbot]

Comment 8 by zippoxer:

Here is my output:
SysReserve: v=0x8fa1d0 n=805306368
SysReserve: hint failed with errno=487
SysReserve: accepting any address failed with errno=8
And my memory map:
Memory map
Address   Size      Owner            Section    Contains                 Type  Access  
Initial >Mapped as
00010000  00010000 >                >                                    Map  >RW      
RW
00020000  00010000 >                >                                    Map  >RW      
RW
0006D000  00001000 >                >                                    Priv >RW 
Guar>RW  Guar>
0006E000  00002000 >                >           Stack of main thread     Priv >RW      
RW
00070000  00004000 >                >                                    Map  >R        R
00080000  00001000 >                >                                    Priv >RW      
RW
00090000  00067000 >                >                                    Map  >R       
R        C:\Windows\System32\locale.nls
00100000  0000C000 >                >                                    Map  >R        R
001C0000  00003000 >                >                                    Map  >R        R
001D0000  00001000 >                >                                    Priv >RW      
RW
001E0000  00001000 >                >                                    Priv >RW      
RW
00200000  00003000 >                >                                    Priv >RW      
RW
00230000  00006000 >                >                                    Priv >RW      
RW
00400000  00001000 >8_out           >           PE header                Img  >R       
RWE Copy>
00401000  000E2000 >8_out           >.text      Code                     Img  >R E     
RWE Copy>
004E3000  00418000 >8_out           >.data      Data                     Img  >RW 
Copy>RWE Copy>
008FB000  00001000 >8_out           >/4                                  Img  >R       
RWE Copy>
008FC000  0000D000 >8_out           >/18                                 Img  >R       
RWE Copy>
00909000  00009000 >8_out           >/30                                 Img  >R       
RWE Copy>
00912000  00027000 >8_out           >/43                                 Img  >R       
RWE Copy>
00939000  0000F000 >8_out           >/55                                 Img  >R       
RWE Copy>
00948000  00003000 >8_out           >/71                                 Img  >R       
RWE Copy>
0094B000  00001000 >8_out           >/87                                 Img  >R       
RWE Copy>
0094C000  00001000 >8_out           >/102                                Img  >R       
RWE Copy>
0094D000  00001000 >8_out           >.idata     Imports                  Img  >RW 
Copy>RWE Copy>
0094E000  00001000 >8_out           >.edata     Exports                  Img  >R       
RWE Copy>
0094F000  00001000 >8_out           >.symtab                             Img  >R       
RWE Copy>
00950000  00101000 >                >           GDI handles              Map  >R        R
00A60000  00108000 >                >                                    Map  >R        R
0DCE0000  00001000 >KERNELBASE      >           PE header                Img  >R       
RWE Copy>
0DCE1000  00043000 >KERNELBASE      >.text      Code,imports,exports     Img  >R E     
RWE Copy>
0DD24000  00002000 >KERNELBASE      >.data      Data                     Img  >RW      
RWE Copy>
0DD26000  00001000 >KERNELBASE      >.rsrc      Resources                Img  >R       
RWE Copy>
0DD27000  00003000 >KERNELBASE      >.reloc     Relocations              Img  >R       
RWE Copy>
38010000  00001000 >                >                                    Img  >R       
RWE Copy>
402C0000  00001000 >LPK             >           PE header                Img  >R       
RWE Copy>
402C1000  00006000 >LPK             >.text      Code,imports,exports     Img  >R E     
RWE Copy>
402C7000  00001000 >LPK             >.data      Data                     Img  >RW      
RWE Copy>
402C8000  00001000 >LPK             >.rsrc      Resources                Img  >R       
RWE Copy>
402C9000  00001000 >LPK             >.reloc     Relocations              Img  >R       
RWE Copy>
41840000  00001000 >IMM32           >           PE header                Img  >R       
RWE Copy>
41841000  00017000 >IMM32           >.text      Code,imports,exports     Img  >R E     
RWE Copy>
41858000  00001000 >IMM32           >.data      Data                     Img  >RW      
RWE Copy>
41859000  00005000 >IMM32           >.rsrc      Resources                Img  >R       
RWE Copy>
4185E000  00001000 >IMM32           >.reloc     Relocations              Img  >R       
RWE Copy>
6F8E0000  00001000 >USP10           >           PE header                Img  >R       
RWE Copy>
6F8E1000  0005B000 >USP10           >.text      Code,imports,exports     Img  >R E     
RWE Copy>
6F93C000  00002000 >USP10           >.data      Data                     Img  >RW      
RWE Copy>
6F93E000  0002A000 >USP10           >Shared                              Img  >R       
RWE Copy>
6F968000  00012000 >USP10           >.rsrc      Resources                Img  >R       
RWE Copy>
6F97A000  00003000 >USP10           >.reloc     Relocations              Img  >R       
RWE Copy>
6FF50000  00001000 >msvcrt          >           PE header                Img  >R       
RWE Copy>
6FF51000  0009F000 >msvcrt          >.text      Code,imports,exports     Img  >R E     
RWE Copy>
6FFF0000  00007000 >msvcrt          >.data      Data                     Img  >RW 
Copy>RWE Copy>
6FFF7000  00001000 >msvcrt          >.rsrc      Resources                Img  >R       
RWE Copy>
6FFF8000  00004000 >msvcrt          >.reloc     Relocations              Img  >R       
RWE Copy>
70990000  00001000 >MSCTF           >           PE header                Img  >R       
RWE Copy>
70991000  00083000 >MSCTF           >.text      Code,imports,exports     Img  >R E     
RWE Copy>
70A14000  00002000 >MSCTF           >.data      Data                     Img  >RW 
Copy>RWE Copy>
70A16000  00041000 >MSCTF           >.rsrc      Resources                Img  >R       
RWE Copy>
70A57000  00005000 >MSCTF           >.reloc     Relocations              Img  >R       
RWE Copy>
77B60000  00001000 >GDI32           >           PE header                Img  >R       
RWE Copy>
77B61000  00048000 >GDI32           >.text      Code,imports,exports     Img  >R E     
RWE Copy>
77BA9000  00002000 >GDI32           >.data      Data                     Img  >RW      
RWE Copy>
77BAB000  00001000 >GDI32           >.rsrc      Resources                Img  >R       
RWE Copy>
77BAC000  00002000 >GDI32           >.reloc     Relocations              Img  >R       
RWE Copy>
77D10000  00001000 >user32          >           PE header                Img  >R       
RWE Copy>
77D11000  00068000 >user32          >.text      Code,imports,exports     Img  >R E     
RWE Copy>
77D79000  00001000 >user32          >.data      Data                     Img  >RW      
RWE Copy>
77D7A000  0005B000 >user32          >.rsrc      Resources                Img  >R       
RWE Copy>
77DD5000  00004000 >user32          >.reloc     Relocations              Img  >R       
RWE Copy>
77DE0000  00001000 >kernel32        >           PE header                Img  >R       
RWE Copy>
77DE1000  000C5000 >kernel32        >.text      Code,imports,exports     Img  >R E     
RWE Copy>
77EA6000  00001000 >kernel32        >.data      Data                     Img  >RW      
RWE Copy>
77EA7000  00001000 >kernel32        >.rsrc      Resources                Img  >R       
RWE Copy>
77EA8000  0000C000 >kernel32        >.reloc     Relocations              Img  >R       
RWE Copy>
77EC0000  00001000 >ntdll           >           PE header                Img  >R       
RWE Copy>
77EC1000  000D5000 >ntdll           >.text      Code,exports             Img  >R E     
RWE Copy>
77F96000  00001000 >ntdll           >RT                                  Img  >R E     
RWE Copy>
77F97000  00009000 >ntdll           >.data      Data                     Img  >RW 
Copy>RWE Copy>
77FA0000  00057000 >ntdll           >.rsrc      Resources                Img  >R       
RWE Copy>
77FF7000  00005000 >ntdll           >.reloc     Relocations              Img  >R       
RWE Copy>
7F6F0000  00005000 >                >                                    Map  >R        R
7FFB0000  00023000 >                >           Code pages               Map  >R        R
7FFD6000  00001000 >                >           Process Environment Bloc>Priv >RW      
RW
7FFDF000  00001000 >                >           Data block of main threa>Priv >RW      
RW
7FFE0000  00001000 >                >           User Shared Data         Priv >R        R
80000000  7FFF0000 >                >           Kernel memory            Kern >
So you are right :P
The program ends at 0xB68000 (0xA60000 + 0x108000) and the first dll starts at
0xDCE0000. The size of the hole is 219643904 (smaller than 805306368) bytes.
Why? :/

[alexbrainman]

Comment 9:

I do not know how KERNELBASE.DLL happened to be in the middle of your address space. And
I don't know how to control it. When I run your process on all computers I have: XP,
W2K, Windows 7; none of them have KERNELBASE.DLL loaded when I stop process in
runtime·SysReserve as you did. Perhaps, someone else will suggest something new.

Labels changed: added os-windows.

Status changed to Accepted.

[gopherbot]

Comment 10 by zippoxer:

I just tried running a go program called Pokemon Universe (http://pokemon-universe.com/)
and got the same error (throw: runtime: cannot reserve arena virtual address space).
This game worked not long time ago, on my old computer.
So this might be a problem with my current windows installation - it can't run go
programs, or more exactly, go programs using cgo.
I'm only worried because so many other applications are working perfectly. So part of
the problem is with go and the other is with my windows installation.

[alexbrainman]

Comment 11:

I think the problem you are seeing is because KERNELBASE.DLL happend to be loaded in the
middle of your address space. Your address space got fragmented, so go runtime can't
allocate this big chunk of memory it uses for garbage collector.
Non-cgo programs shouldn't be affected, because go runtime loads most of its dlls after
this big block of memory is allocated. So Windows just find another place for all these
dlls.
Cgo programs, on the other hand, have dlls loaded during process load time, even before
process started running. Dlls are listed in the pe executable file header. And Windows
process loader just loads them all alongside with the executable. Loader picks whatever
addresses it likes. This makes your address fragmented even before your process starts.
I am surprised, no-one reported anything like that before.
If nothing else, we would have to "reserve" this big contiguous block of addresses
somehow in our pe file description. Maybe create a special pe section for it, or include
it as part of existing BSS section.
Alex

[gopherbot]

Comment 12 by zippoxer:

Or to reinstall my windows. I tried adding a section to the PE file with the size of
805306368, but windows complained about validity. I'm (almost) sure I will be able to
execute cgo programs with the right person editing the PE file, but after every
compilation?
So I will reinstall windows soon, unless someone resists.
And if the problem ever appears again, I will know if it's a software that I installed
or a windows setting that I changed.
Proper cgo programs ran successfully in so many windows installations (I take Pokemon
Universe as an example) and nobody complained about "throw: runtime: cannot reserve
arena virtual address space", so what's the chance that this problem will appear again?
Thank you Alex for exploring the problem (until actually discovering it). Seriously,
without your help I was quitting go (until the world moves unix or until I format my
windows and accidentally trying a cgo program).

[alexbrainman]

Comment 13:

Adding big pe section is not enough (even if you do it properly). You also need to make
go use it during runtime.
I have a feeling we will hear more about your problem. Lets keep this issue opened.

[rsc]

Comment 14:

Labels changed: added priority-later, removed priority-medium.

[rsc]

Comment 15:

Labels changed: added priority-go1.

[robpike]

Comment 16:

Owner changed to builder@golang.org.

[gopherbot]

Comment 18 by waqas20:

Some quick googling hints at kernelbase.dll being loaded in 32-bit applications running
on 64-bit Windows. I don't have a 64-bit Windows install available at the moment to test.

[rsc]

Comment 19:

This won't be fixed for Go 1.

Labels changed: added priority-later, removed priority-go1.

Status changed to HelpWanted.

[gopherbot]

Comment 20 by novermore:

http://www.abtinforouzandeh.com/2012/04/08/Do-Not-Use-Go-For-32bit-Development.html

[4ad]

Comment 21:

I did some more research on this issue, particularly the issue reported in comment #20.
I've talked to the author of that blog post, and cgo is NOT used.
Windows shared libraries are not relocatable, if the dynamic linker can't load it at the
preferred base address specified in the PE header, it has to rebase it. Apart from
slowing loading of binaries, rebasing has the side effect of not allowing that dubious
benefit dynamic linking fans brag about -- it's not possible to have a single copy of a
shared object mapped in every process.
Windows tries really really hard to avoid rebasing, for obvious reasons. All the system
DLLs are crafted with a base address close to the 2GB mark in a way that rebasing will
never be necessary. Also, by default, the compiler will generate shared objects with a
base address far from the space system DLLs are using, so 3rd party DLLs will not ask
for the same part of the address space as the system DLLs.
  >dumpbin /headers KernelBase.dll | findstr /C:"image base"
          7D850000 image base (7D850000 to 7D895FFF)
We can see that KernelBase.dll prefers to be loaded at 0x7D850000, but it rebased at
0x0DCE0000. This is rather unusual in the Windows world. It's very common for 3rd party
DLLs to be rebased, but it's unusual for system DLLs to be rebased. Even if a 3rd party
DLL asks for the same address space reservation as a system DLL, the dynamic loader will
rather rebase the 3rd party DLL, and map in the process the same system DLL used by
other processes.
If a system DLL was rebased, it means it was rebased in the first process of that
session (a session is a Windows thing irrelevant to the discussion), and the rebased
copy is shared between all processes in that session. It has to be the first process,
because it's the only process the system DLL has not been loaded yet.
If someone, accidentally or not, forced a system DLL to be rebased in the first process,
that someone injected a DLL in the first process where the system one wants to be. This
is a way of doing user mode hooking, generally by malware, but it's also done by
purposely legitimate "security" software. In fact, before Windows Vista, the only
sanctioned way of doing registry monitoring, was through a user mode hook!
I don't care about "fixing" Go to run with malware, but I do care about cgo, which has
the same effect (and is the purpose of this issue report anyway). We could make the
address space reservation in the PE header, but it pains me that it would require a
different mechanism for Windows and non-Windows. It also wouldn't be such a great idea
if we ever get shared libraries, but I don't care about that crap the slightest.

[alexbrainman]

Comment 22:

Creating special pe section (with IMAGE_SCN_CNT_UNINITIALIZED_DATA set so it does not
take up any disk space) will not work. This memory gets committed at program load as far
as I can see. This is unacceptable for our purpose.
Alex

[4ad]

Comment 23:

If that's the case, we are out of luck.

[gopherbot]

Comment 24 by ericsroy:

I also see this problem while running my application (it uses cgo) on a VMWare virtual
machine running a fresh install of Windows XP Pro.  The virtual machine and os are 32
bit, but the host system is 64 bit running Windows 7.
I've read the discussion above, but are we moving towards a solution to this issue? 
This is a pretty serious problem, one that pretty much eliminates Go as a viable option
for Windows software development.

[gopherbot]

Comment 25:

Go implementation contains code like this one:
   // Words outside the arena cannot be pointers.
   if((byte*)obj < arena_start || (byte*)obj >= arena_used)
       continue;
In order to fix this Windows issue, Go would need to abandon the single arena approach
and use a different way of detecting whether an arbitrary pointer points to memory
allocated by the Go runtime.
One possible approach would be to put the allocated regions in a tree-like structure
(like paging on CPUs).

[alexbrainman]

Comment 26:

@ericsroy,
> ... are we moving towards a solution to this issue? ...
As far as I know, there are no proposals.
Alex

[gopherbot]

Comment 27 by ericsroy:

Quick question -
I understand why the memory arena is a contiguous block of virtual memory (it would be
significantly more complex and less efficient to spread it over several memory ranges),
but why must the memory information bitmap be contiguous with the arena memory?
Rather than trying to make one huge 768mb allocation (bitmap + arena), could the memory
info bitmap not be allocated elsewhere?  So that we make a 256mb allocation and then
another 512mb allocation?
On the system that I have which is exhibiting this problem, the two allocations would
almost certainly succeed, while the monolithic one does not.
(This question is based on what I see in the code in malloc.goc, correct me if i've
misunderstood anything)
Thanks,
-Eric

[gopherbot]

Comment 28 by ericsroy:

Here's a workaround for anyone experiencing this problem with cgo (rather than with
strangely rebased system dlls):
Custom and 3rd party dlls seem to be based at 0x10000000 by default (at least if you
create them with Visual Studio they are).  If you rebase your dlls to somewhere closer
to the executable image, you can clear a hole in the address space that (in my case) is
large enough to allow the memory allocation to succeed.
You can use Visual Studio's editbin.exe tool to rebase your dlls.  However, I can't seem
to rebase dlls that have been digitally signed, d3dx9_43.dll is one example.  But if
you're only using unsigned dlls then this might be an acceptable workaround.
-Eric

[4ad]

Comment 29:

Please not that ASLR causes DLLs to be rebased anyway.

[gopherbot]

Comment 30 by ericsroy:

Good point, I don't think my dlls had ASLR enabled, but I also explicitly disabled it
with editbin's /DYNAMICBASE option.

[alexbrainman]

Comment 31:

@ericsroy,
What you have to be aware of, that any non-cgo Go executable specifies to load only 2
dlls: kernel32.dll and winmm.dll. These are loaded by Windows program loader before Go
executable even runs, and we have no control over that. We could, probably, get rid of
winmm.dll if that is an issue, so that would leave only kernel32.dll to fight with. Any
other dlls that Go program might use are loaded during run time well after gc memory is
reserved, so they wouldn't affect the allocation.
On the other hand, if you use cgo - then you include code that is build with gcc and
uses whatever gcc runtime provides. So, whatever other dlls are included by cgo program
are out of Go control.
Also, sometimes Windows will load some dlls into your process without you asking for it.
Some antivirus, firewalls, "drivers" just load some dlls into every process to do what
they need to do. In that instance you are at the mercy of whoever written that software.
And sometimes that software is buggy and poorly designed.
Alex

[gopherbot]

Comment 32 by ericsroy:

@Alex,
Yes, I understand.  I've only been seeing this problem with dlls that I've introduced
into my program by using cgo.  In my particular case, winmm and kernel32 are in high
memory where we'd like them to be.  I haven't as of yet seen the scenario where unwanted
dlls are loaded, or where system dlls are rebased to significantly lower memory
locations.
For the sake of others, I just thought it was worth mentioning that naive use of even a
single dll through cgo could be enough to cause the issue.
-Eric

[alexbrainman]

Comment 33:

@ericsroy,
> Rather than trying to make one huge 768mb allocation (bitmap + arena),
> could the memory info bitmap not be allocated elsewhere?  So that we
> make a 256mb allocation and then another 512mb allocation?
I do not know anything about how memory manager works, but I tried allocating bitmap and
arena in separate / disjoint areas of memory and that fails, because some code, for
example pkg/runtime/mgc0.c:1256, assumes that bitmap and arena are together in one chunk
in that order. Perhaps it is easy to change all that code, but I do not know.
I could be wrong about that, so leaving for others to comment.
Alex

[rsc]

Comment 34:

Labels changed: added go1.1maybe.

[robpike]

Comment 35:

Labels changed: removed go1.1maybe.

[gopherbot]

Comment 36 by jack.radiance:

How about SizeOfHeapReserve in IMAGE_OPTIONAL_HEADER ?
    The number of bytes to reserve for the local heap. Only the memory specified by the SizeOfHeapCommit
member is committed at load time; the rest is made available one page at a time until
this reserve size is reached.
http://msdn.microsoft.com/en-us/library/windows/desktop/ms680339%28v=vs.85%29.aspx

[alexbrainman]

Comment 37:

@jack.radiance, how do you propose to use that memory?
Alex

[gopherbot]

Comment 38 by jack.radiance:

Well, you need a solid memory block to fit 0x30000000 bytes? I thought this will do.

[gopherbot]

Comment 39 by jack.radiance:

Here http://i.imgur.com/h2UEHmy.png
I thought that if this memory range is reserved, then it will not be occupied with
kernelbase

[alexbrainman]

Comment 40:

> Well, you need a solid memory block to fit 0x30000000 bytes? ...
That is too simple of a description. What we really need is to re-implement
runtime·SysReserve
(http://code.google.com/p/go/source/browse/src/pkg/runtime/mem_windows.c#50) and
runtime·SysMap
(http://code.google.com/p/go/source/browse/src/pkg/runtime/mem_windows.c#63) functions.
How do you propose we do that?
Alex

[rsc]

Comment 41:

Labels changed: added go1.3maybe.

[rsc]

Comment 42:

Labels changed: added release-none, removed go1.3maybe.

[rsc]

Comment 43:

Labels changed: added repo-main.

[gopherbot]

Comment 44 by ollie.walsh:

This used to be a common problem for HPC applications on 32bit windows.
Adding -largeaddessaware to LDFLAGS should address this (or editbin /LARGEADDRESSAWARE).
On 32bit windows this gives an additional 1GB block of virtual address space for 32bit
processes. On x64 this gives an additional 2GB address space. The library loader doesn't
appear to touch this address space - keeping all dlls below the 2GB boundary - so it's a
contiguous block.

[davecheney]

Comment 45:

@ollie. I'm not sure how much that would help. The problem is not that there is not
enough virtual address space available to the process, but that it has been fragmented
by the system (and other dll's which like to insert themselves into processes) dlls. The
Go garbage collector currently requires a contiguous heap, and it is these dlls which
limit the maximum size of the heap by fragmenting it.
passing /LARGEADDRESSAWARE may help, but it feels like a loosing battle.

[alexbrainman]

Comment 46:

@ollie.walsh, re "... Adding -largeaddessaware to LDFLAGS should address this ..." - how
exactly does it work?
Alex

[mattn]

I tried this. not reproduced. prefer closing.

[rsc]

Closing per @mattn.