`JULIA usr/lib/julia/sys.ji` fails when building 64-bit Julia on Windows

[vharavy]

During the final steps of the build I got the following error

    JULIA usr/lib/julia/sys.ji
Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.

*** This error is usually fixed by running 'make clean'. If the error persists, try 'make cleanall'. ***

Running make clean or make cleanall as suggested does not help.

[pao]

Could you please add the git hash of Julia, and a complete build log as a gist (https://gist.github.com/)?

cc @vtjnash

[vtjnash]

the build log isn't necessary. a gdb backtrace would be most helpful. here's how to create one on windows from the msys bash shell:

$ make debug
$ export PATH=`pwd`/usr/lib:`pwd`/usr/lib/julia:`pwd`/usr/bin:$PATH
$ cd base
$ gdb --args ../usr/bin/julia-debug-basic -bf sysimg.jl
gdb> run
....
gdb> bt
...

[vtjnash]

also only 32-bit julia on windows can print that error message... (edit: actually, i realized i had turned this on for windows 64 also, but thought it was non-functional)

[vharavy]

Okay, here is what I have got after running Julia under GDB as @vtjnash suggested:

Starting program: d:\Temporary\Julia\usr\bin\julia-debug-basic.exe -bf sysimg.jl
[New Thread 3400.0xdc8]

Program received signal SIGSEGV, Segmentation fault.
0x00000000031f004c in ?? ()
(gdb) bt
#0  0x00000000031f004c in ?? ()
#1  0x0000000000deee30 in ?? ()
#2  0x00000000025d02b0 in ?? ()
#3  0x00000000025d02b0 in ?? ()
#4  0x0000000000deedd0 in ?? ()
#5  0x00000000031f001b in ?? ()
#6  0x0000000000000000 in ?? ()
(gdb) list
285             jl_show(jl_stderr_obj(),jl_exception_in_transit);
286             JL_PUTS("\n",JL_STDOUT);
287             goto again;
288         }
289         uv_tty_reset_mode();
290         return iserr;
291     }
292
293     int main(int argc, char *argv[])
294     {

Julia is built from commit 3c3cc4ee, but I have encountered the same error in earlier commits too.

[vtjnash]

That isn't very helpful. Can you put a breakpoint in jl_throw and see what error message was being generated?

[vharavy]

Setting breakpoint in jl_throw does not work, because it never called. As you may see from my previous post the program crashed due to segmentation fault:

(gdb) run
Starting program: d:\Temporary\Julia\usr\bin\julia-debug-basic.exe -bf sysimg.jl
[New Thread 2536.0xd28]

Program received signal SIGSEGV, Segmentation fault.
0x000000000031048a in ?? ()
(gdb) bt
#0  0x000000000031048a in ?? ()
#1  0x00000000025e7570 in ?? ()
#2  0x0000000000def188 in ?? ()
#3  0x0000000000b70000 in ?? ()
#4  0x000000000353e680 in ?? ()
#5  0x0000000000117dc0 in ?? ()
#6  0x000000000000000a in ?? ()
#7  0x0000000000deef80 in ?? ()
#8  0x0000000063d93476 in jl_apply (f=0x353e680, args=0xb70000, nargs=0) at julia.h:1021
Backtrace stopped: frame did not save the PC
(gdb) frame 8
#8  0x0000000063d93476 in jl_apply (f=0x353e680, args=0xb70000, nargs=0) at julia.h:1021
1021        return f->fptr((jl_value_t*)f, args, nargs);
(gdb) print f
$1 = (jl_function_t *) 0x353e680
(gdb) print *f
$2 = {type = 0x64954a70, fptr = 0xbaadf00d00000102, env = 0x345b8a0, linfo = 0x0}

On Windows allocated memory is usually initialized with constants 0xbaadf00d (kind of "bad food").

[vtjnash]

gdb on Windows is extremely unreliable -- in this case, the stack trace you've given is almost completely useless: Only the #0 entry typically contains a completely valid frame, the rest is just pretty much just random numbers. (consider: if the code f->fptr had been executed when fptr = 0xbaadf00d, the program would have segfaulted right then, not 8 levels further up the call stack.)

Unfortunately, gdb also fails to stop at breakpoints relatively often. Sometimes you have to trick it just right (I find getting it to stop at a few other breakpoints along the way sometimes works). Or using sometimes using QtCreator, modified to use the 64-bit gdb can work.

[pao]

Will WinDbg work?

[vharavy]

No, this is the first thing I tried. MinGW does not produce debugging symbols in *.PDB format, so using WinDbg reduced to working with disassembled code. There is not symbol information at all, with exception to exported functions.

Right now, I working on building Julia using Visual Studio and Intel C++ Composer. I have noticed that Julia itself does not depend on LAPACK, FFTW and other libraries. All I need is to build LibUV, LLVM, FemtoLisp and support libraries. Everything else is not required for now.

Also, could it be possible that Julia fails due to some package in Base that is not ported to Windows? I know mmap.jl is not ported.

[pao]

Right, forgot about the difference in debugging symbol formats.

[vtjnash]

You can use the libraries created by mingw with files compiled by Visual Studio. This may require creating .lib files from the .dll files, which Visual studio has a tool for (The program is called lib).

On 32-bit windows, compatibility may be improved by setting the gcc stack to honor the platform ABI (http://math-atlas.sourceforge.net/errata.html#gccCrazy). On 64-bit windows, that issue, at least, does not exist.

[ViralBShah]

@VHaravy You can pretty much avoid much of deps. You need to avoid loading the relevant files in base.

It used to be the case that pcre was required as part of the julia startup, and you do need double-conversion as well.

[vharavy]

Just as a progress report. I have managed to create makefiles that gives the ability to build Julia using NMake (Microsoft's implementation of make) and Intel C++ Compiler. Currently, I am able to build FemtoLisp (both library and executable), libsupport, libuv and most parts of Julia (the only part is left is LLVM and Julia's UIs). As soon as I will have something more complete I will present it to your judgement.

P.S. Initially, I have been trying to build Julia with Microsoft's compiler, but Julia source code contains a huge amount of C99 code (Microsoft does not support C99). I believe this has to change.

[JeffBezanson]

Let's see, 1999 was...14 years ago. Hurry up, microsoft. On Apr 26, 2013 6:42 AM, "VHaravy" notifications@github.com wrote:

Just as a progress report. I have managed to create makefiles that gives the ability to build Julia using NMake (Microsoft's implementation of make) and Intel C++ Compiler. Currently, I am able to build FemtoLisp (both library and executable), libsupport, libuv and most parts of Julia (the only part is left is LLVM and Julia's UIs). As soon as I will have something more complete I will present it to your judgement.

P.S. Initially, I have been trying to build Julia with Microsoft's compiler, but Julia source code contains a huge amount of C99 code (Microsoft does not support C99). I believe this has to change.

— Reply to this email directly or view it on GitHubhttps://github.com/JuliaLang/julia/issues/2896#issuecomment-17066479 .

[vharavy]

@JeffBezanson Sadly they are not going to support C99 standard. They shifted their priority to C++ and then to C# and .NET Framework. Currently, there is no progress regardless support of any C standards. On the other hand, why not to use ANSI C standard? It is known to be supported by all C compilers on all OSs. I think it is best not to use any OS or compiler specific extensions when developing truly cross-platform software. You can use such extensions if they produce improvements, but give the possibility to compile your code without them. You can criticize Microsoft as much as you want, but the reality is not going to change. So, I suggest to adjust.

[JeffBezanson]

C99 is a well-established standard. There are multiple high-quality free implementations of it. On Apr 26, 2013 7:59 AM, "VHaravy" notifications@github.com wrote:

@JeffBezanson https://github.com/JeffBezanson Sadly they are not going to support C99 standard. They shifted their priority to C++ and then to C# and .NET Framework. Currently, there is no progress regardless support of any C standards. On the other hand, why not to use ANSI C standard? It is known to be supported by all C compilers on all OSs. I think it is best not to use any OS or compiler specific extensions when developing truly cross-platform software. You can use such extensions if they produce improvements, but give the possibility to compile your code without them. You can criticize Microsoft as much as you want, but the reality is not going to change. So, I suggest to adjust.

— Reply to this email directly or view it on GitHubhttps://github.com/JuliaLang/julia/issues/2896#issuecomment-17070252 .

[vharavy]

@JeffBezanson What would you suggest to use on Windows in that case besides MinGW?

[StefanKarpinski]

@VHaravy, really cool that you've made so much progress towards getting this to work with NMake. I personally would be willing to change our code to work with MS compilers, although I couldn't really be of any help since I don't have such a system to even test on. I also wonder if the whole effort isn't somewhat doomed since we rely quite heavily on a variety of other unix tools that end up necessitating at least msysgit.

[Keno]

It has been my goal to make Julia compile with MSVC for a while, so it's great that you already got this far. What C99-isms are we using in Julia that break MSVC?

[vharavy]

@StefanKarpinski MSYSGit (at least the one downloaded from git-scm.com) has the option "Run Git from Windows command prompt". When it is installed with this option, Git can be run from native Windows command prompt.

@loladiro Mostly, it is related to declaring local variables in the middle of the block. But, there are other issues. For example, Intel C++ Composer does support C99 (with /Qstd=c99) but it has some problems with variadic macros. I had to redefine JL_GC_PUSH macro in order to build Julia.

[JeffBezanson]

We can start incorporating your changes incrementally. Assuming the change to JL_GC_PUSH doesn't break anything, it would be good to make a pull request for it, and other minor portability tweaks.

[Keno]

@VHaravy re:MSYSGit while that is true, it's not all that useful for any applications other than the command prompt as it relies on some prompt magic in order to run and has some problems with argument passing. In any case, it shouldn't really matter as it's certainly possible to use MSYSGit without the fancy wrappers (It's mostly a question of putting everything in the right path). I've been doing repeatedly over the past year, so I'll take care of that when the time comes.

[ViralBShah]

@VHaravy Thanks for the persistence with this. All of this should make Julia work well on Windows. Please do keep submitting incremental pull requests, so that we can keep merging bits and pieces - otherwise we will have a merge nightmare.

[vharavy]

@ViralBShah @JeffBezanson @StefanKarpinski @vtjnash I have created a pull request with first portion of changes.

[vharavy]

I am preparing the next portion of changes that will allow build Julia on Windows with Intel C++ Compiler (and eventually with Microsoft), but I need that my pull request to libuv accepted.

[StefanKarpinski]

Merged.

[StefanKarpinski]

I'm not sure if you need the libuv submodule updated too, but that seems non-trivial. When I checkout master and attempt to pull it starts a merge that for some reason has lots of conflicts. I'm not sure what kind of nonsense is going on since I haven't made any local libuv changes, but I'm not going to mess with that.

[vharavy]

@StefanKarpinski Yes, I need to update libuv submodule too, but I will deal with it myself and let it be a part of my pull request.

[vtjnash]

I have no recollection of why this is open.