Open ghost opened 6 years ago
/cc @alexbrainman
@xoviat what is the problem that you are having? I need to be able to reproduce it here. So, please, provide all steps I will need to follow to reproduce this.
Thank you
Alex
PS: I won't have computer until end of July. I will have a look at this then.
what is the problem that you are having?
I haven't tried it yet, but I would like to specifically call c functions in msvc object files by linking them as .syso with the go linker. Everything that I have read indicates that this is not possible but I will create a procedure to reproduce the specific error that occurs.
specifically call c functions in msvc object files by linking them as .syso with the go linker
Have you tried building these into a DLL, and use them from inside of DLL?
I will create a procedure to reproduce the specific error that occurs.
Please, do. Thank you.
Alex
Have you tried building these into a DLL, and use them from inside of DLL?
That was actually my original plan. I am using swig so it is not so convenient to compile the generated c code separately and then rewrite the functions as DLL exports. It's not difficult, but it is annoying when the workflow with gcc is just go generate; go build
.
Alright, I have a procedure. Start with these files:
hello.go:
package main
/*
extern void hello();
*/
import "C"
func main() {
C.hello()
}
hello.c:
#include <stdio.h>
extern void hello()
{
printf("Hello World from C");
}
then run these commands (with msvc on path):
cl /c hello.c
mv hello.obj hello.syso
mv hello.c hello.c.bak
go build
Result:
Warning: corrupt .drectve at end of def file
When running the produced file:
Exception 0xc0000005 0x8 0x13 0x13
PC=0x13
signal arrived during external code execution
main._Cfunc_hello()
_//_obj/_cgo_gotypes.go:41 +
main.main()
C://hello.go:9 +0x27
goroutine 17 [syscall, locked to thread]:
runtime.goexit()
C:/Program Files/Go/src/runtime/asm_amd64.s:2197 +0x1
rax 0x4a5960
rbx 0xc042045f78
rcx 0x4a9a20
rdi 0xc042045f78
rsi 0x4adc60
rbp 0xc042045f38
rsp 0x6dfd68
r8 0xc042016340
r9 0x0
r10 0xc04204faa0
r11 0x4783c2
r12 0x0
r13 0x6
r14 0x0
r15 0xf1
rip 0x13
rflags 0x10216
cs 0x33
fs 0x53
gs 0x2b
I do not have cl command installed on my pc. How do I install msvc?
Thank you.
Alex
You need the "build tools 2017" here. Microsoft actually now allows anyone to use visual studio for free as long as it's not for commercial development. If it's too much trouble, I can just give you the object file if you want.
If only c++ was not a thing, I wouldn't have to worry about this. But it is, so the pain continues...
You need the "build tools 2017" here.
Got it. Thank you.
If it's too much trouble, I can just give you the object file if you want.
Yes, please, post hello.obj somewhere.
Thinking about this some more. You are actually using gcc to link object file compiled with msvc compiler. Can you try and do your exercise, but replacing "go build" step with gcc linking hello.obj to a C program? Perhaps it has been done before. I suspect, if we know how to do that, we might be able to do similar with Go.
Alex
AFAIK lld https://github.com/llvm-mirror/lld supports msvc object files.
Object file is here: https://github.com/xoviat/msvcgo/blob/master/hello.syso
lld https://github.com/llvm-mirror/lld supports msvc object files
Go uses gcc linker (not lld) on Windows.
Object file is here: https://github.com/xoviat/msvcgo/blob/master/hello.syso
I will try it when I get home in August. thank you.
Alex
Go uses gcc linker (not lld) on Windows.
I know. But lld is the best open source documentation of msvc object format.
I actually get same error from ld so the error is definitely coming from ld.
I actually get same error from ld so the error is definitely coming from ld.
Yes. We need to work out how to build C program by compiling part with msvc, and linking with gcc.
Alex
Forgive my ignorance, but what exactly gcc linking? Is it linking output of the go linker?
Running objconv on the object files to convert to elf:
objconv -felf hello.obj hello.syso
We now have these errors:
hello.syso: In function `__local_stdio_printf_options':
(.text$mn+0x3): undefined reference to `?_OptionsStorage@?1??__local_stdio_printf_options@@9@9'
hello.syso: In function `_vfprintf_l':
(.text$mn+0x3a): undefined reference to `__stdio_common_vfprintf'
hello.syso: In function `printf':
(.text$mn+0x28): undefined reference to `__acrt_iob_func'
collect2.exe: error: ld returned 1 exit status
Possibly stdio is off limits?
Forgive my ignorance, but what exactly gcc linking? Is it linking output of the go linker?
You use 2 programs to build your Go program:
Sometimes (when one of your packages uses Cgo), the Go linker calls external linker to find all bits that are implemented in C. Imagine you call printf from your C code. The C printf compiled code needs to be part of Go executable, but Go linker does not know where to get it. So Go linker calls external linker to include that code.
Current Go uses gcc compiler/linker to compile and link C code (we use mingw gcc). If you are going to compile your C code with different compiler (by Microsoft), you would have to use correspondent linker (by Microsoft) to find all external C code that the compiler created.
So, I guess, I was wrong about suggesting to use gcc linker. For general scenario, you would have to use both Microsoft compiler and linker. We would have to figure out what Microsoft linker requires as input and match that.
You might get away without MC linker, if your C code does not have any external code. Please try some really simple C program (like the one that adds 2 integers or something). That might just work as you have described above.
Alex
Possibly stdio is off limits?
I suspect you need to call Microsoft linker to find that code.
Alex
I don't make sure but
objconv -felf hello.obj hello.syso
Maybe, you should try to do coff or omf instead of elf?
Maybe, you should try to do coff or omf instead of elf?
@xoviat yes, you should not convert .obj files to elf, Windows version of gcc generates pe/coff files just like any other Windows compiler.
Alex
the Go linker calls external linker to find all bits that are implemented in C.
What is the specific command used? If I know the mingw command, then perhaps I can proceed down a file comparison path to try to get msvc to match what mingw is putting out.
You can see the exact comment by running go build -ldflags=-v
.
Okay, so from what I can tell:
ld (link -> go.obj) + (gcc -> obj files) ==> a.out.exe
Go appears to create a temporary directory with these files. Is there any way to preserve the temporary directory so that I can inspect its contents?
Try go build -work
WORK=C:\Users\mattn\AppData\Local\Temp\go-build566171254
Object files are remaining in this directory.
The -work
option actually won't preserve the temporary files that the linker creates. At the moment you just have to edit the linker sources to not remove the directory. We should probably add an option for that, one way or another.
At the moment you just have to edit the linker sources to not remove the directory.
If you don't mind, I'm going to wait until this is implemented in HEAD so that I don't have to do repetitive work.
Is there any way to preserve the temporary directory so that I can inspect its contents?
cmd/link program has -tmpdir flag for that. You can use it like this:
c:\Users\Alex\dev\src\a>dir
Volume in drive C has no label.
Volume Serial Number is 9012-A870
Directory of c:\Users\Alex\dev\src\a
06/08/2017 02:02 PM <DIR> .
06/08/2017 02:02 PM <DIR> ..
06/08/2017 02:02 PM 77 main.go
1 File(s) 77 bytes
2 Dir(s) 430,809,088,000 bytes free
c:\Users\Alex\dev\src\a>type main.go
package main
import "fmt"
import "C"
func main() {
fmt.Println("Hello")
}
c:\Users\Alex\dev\src\a>go build -o a.exe -ldflags="-tmpdir=c:\Users\Alex\dev\src\a" main.go
c:\Users\Alex\dev\src\a>dir
Volume in drive C has no label.
Volume Serial Number is 9012-A870
Directory of c:\Users\Alex\dev\src\a
06/08/2017 02:02 PM <DIR> .
06/08/2017 02:02 PM <DIR> ..
06/08/2017 02:02 PM 2,055 000000.o
06/08/2017 02:02 PM 22,376 000001.o
06/08/2017 02:02 PM 2,017,382 a.exe
06/08/2017 02:02 PM 135 fix_debug_gdb_scripts.ld
06/08/2017 02:02 PM 2,402,226 go.o
06/08/2017 02:02 PM 77 main.go
06/08/2017 02:02 PM 24 trivial.c
7 File(s) 4,444,275 bytes
2 Dir(s) 430,804,631,552 bytes free
c:\Users\Alex\dev\src\a>
Alex
This is just for my own reference, but this needs porting to msvc:
void
_cgo_sys_thread_start(ThreadStart *ts)
{
uintptr_t thandle;
thandle = _beginthread(threadentry, 0, ts);
if(thandle == -1) {
fprintf(stderr, "runtime: failed to create new OS thread (%d)\n", errno);
abort();
}
}
static void
threadentry(void *v)
{
ThreadStart ts;
ts = *(ThreadStart*)v;
free(v);
ts.g->stackhi = (uintptr)&ts;
ts.g->stacklo = (uintptr)&ts - STACKSIZE + 8*1024;
/*
* Set specific keys in thread local storage.
*/
__asm {
"movq %0, %%gs:0x28\n" // MOVL tls0, 0x28(GS)
"movq %%gs:0x28, %%rax\n" // MOVQ 0x28(GS), tmp
"movq %1, 0(%%rax)\n" // MOVQ g, 0(GS)
:: "r"(ts.tls), "r"(ts.g) : "%rax"
}
crosscall_amd64(ts.fn);
}
I would not underestimate the time that it will take me to complete this task as I am not at all familiar with assembly.
Also, undefined symbols:
Okay, so moving along here more quickly than expected!
All: is asm_amd64.s assembled by go's assembler (with the odd assembly) or gcc's assembler?
It seems that gcc will not assemble it, which means that it's probably go assembly. And then the question becomes: how to assemble it with the go assembler into an object.
The intent is that runtime/cgo/asm_amd64.s gets assembled into a Go object, then cmd/link links it together with all the other Go objects into a single system object, and then the system linker links that single system objects plus all the cgo dependencies into the final program.
Is there a way to assemble that object for now for testing purposes? Like gcc -c asm_amd64.s
except with go?
The intent is that runtime/cgo/asm_amd64.s gets assembled into a Go object, then cmd/link links it together with all the other Go objects into a single system object, and then the system linker links that single system objects plus all the cgo dependencies into the final program.
So far I found these objects:
_cgo_.o
: generated by gcc, unusablego.o
is the biggest object.
@ianlancetaylor
I don't think what you said is correct. crosscall_amd64
is located in 000001.o, but that file doesn't have "all of the go code":
000001.o: file format pe-x86-64
SYMBOL TABLE:
[201](sec 1)(fl 0x00)(ty 20)(scl 2) (nx 0) 0x0000000000000440 crosscall_amd64
[206](sec 0)(fl 0x00)(ty 20)(scl 2) (nx 0) 0x0000000000000000 free
This file has obviously been generated by gcc, which is too late in the process.
Is there a way to assemble that object for now for testing purposes? Like gcc -c asm_amd64.s except with go?
asm_amd64.s is part of runtime package. You can see how "go build" command uses asm_amd64.s file, like:
$ touch asm_amd64.s
$ GOOS=windows go build -x runtime 2>&1 | grep asm_amd64.s
/home/a/go/pkg/tool/linux_amd64/asm -trimpath $WORK -I $WORK/runtime/_obj/ -I /home/a/go/pkg/include -D GOOS_windows -D GOARCH_amd64 -o $WORK/runtime/_obj/asm_amd64.o ./asm_amd64.s
$
Alex
Thanks.
The symbol crosscall_amd64
is defined in the file runtime/cgo/gccamd64.s. That file is compiled by GCC (as are all the runtime/cgo/gcc* files). So it is not combined into the single go.o file that contains all the Go code. The file we were talking about before, runtime/cgo/asm_amd64.s, defines the symbol crosscall2
. You will find that symbol in go.o.
Thanks.
Okay, I've managed to link an executable that crashes on an access violation while in go.runtime.rt0_go+5F --> go.___acrt_stdio_initializer.
That's all the time I have for now; I'll come back to this later.
@alexbrainman That's correct.
@alexbrainman You'll need MSVC to proceed. Let me know whether you need help with this.
You'll need MSVC to proceed.
I am installing this https://www.visualstudio.com/downloads/#build-tools-for-visual-studio-2017
What should I do once installed?
Alex
Once you have installed that, you need to build "libgo," which is the go library. Copy the following files from the cgo folder:
We need to adjust gcc_windows_amd64.c to be compatible with MSVC. Modify the function as follows:
static void
threadentry(void *v)
{
fprintf(stderr, "threadentry: started");
abort();
ThreadStart ts;
ts = *(ThreadStart*)v;
free(v);
ts.g->stackhi = (uintptr)&ts;
ts.g->stacklo = (uintptr)&ts - STACKSIZE + 8*1024;
/*
* Set specific keys in thread local storage.
*/
__writegsqword(0x28, (unsigned long long) ts.tls);
*(void **)ts.tls = (void *) ts.g;
crosscall_amd64(ts.fn);
}
Create a new file in the folder with all of these files called "build.bat":
REM TMP use gcc for assmebly file, in future port to ml64
gcc -c gcc_amd64.S
cl -c gcc_fatalf.c
cl -c gcc_libinit_windows.c
cl -c gcc_windows_amd64.c
cl -c gcc_util.c
ren gcc_amd64.o gcc_amd64.obj
lib gcc_amd64.obj gcc_fatalf.obj gcc_libinit_windows.obj ^
gcc_windows_amd64.obj gcc_util.obj ^
/OUT:libgo.lib
Let me know when you have the folder structured as requested.
I understand that the go linker cannot currently link msvc object files and also recognize that this issue is likely to be a low priority. However, it would be nice to support this because it would somewhat simplify windows workflow. This issue is mainly to understand how much effort this would be and/or what would be required.