JIT on Windows

[trungnt2910]

Opening this issue to track the problems preventing JIT on Windows from happening yet.

From my own experience, the problem is due to the difference between Windows calling convention and Sys-V.

• x86_64 Windows and x86_64 Sys-V passes arguments in different registers. A solution might be just keep using the Sys-V ABI in the JITted code and make a trampoline before returning to compiled Windows code.
• FS - GS register differences. Although as JIT already works on macOS (which also uses different FS/GS than Linux), I guess the same can be applied to Windows?
• x86_64 Sys-V requires a 128-byte red zone below rsp, while x86_64 Windows does not. When jumping to JIT code, the value of rsp must be 128 bytes lower than it is supposed to be, and all read/write operations involving rsp must be hooked to modify the value. Things will get tricky when dealing with opcodes such as call,...

[jart]

The solution I'd propose is to put __attribute__((__sysv_abi__)) on every single op and micro-op function. Then require that GCC 6+ be used to build on Windows if JIT support is desired. Note that this is only a problem for Cygwin. Cosmopolitan Libc uses SysV ABI on Windows.

[trungnt2910]

The solution I'd propose is to put __attribute__((__sysv_abi__)) on every single op and micro-op function. Then require that GCC 6+ be used to build on Windows if JIT support is desired. Note that this is only a problem for Cygwin. Cosmopolitan Libc uses SysV ABI on Windows.

This only affects the calling convention. The last problem is the most difficult: While on Linux, a 128-byte red zone is guaranteed, on Windows, the space below rsp can be overwritten by arbitrary OS mechanisms. Therefore, some kind of rsp emulation is still needed.

[jart]

That problem is easily solved by passing the -mno-red-zone flag. I've always used it for Cosmopolitan Libc since (1) operating system code has this same problem, and (2) Cosmo's garbage collector and lock nopl optimizations violate the stack in such a way that the compiler can't assume a red zone. The performance impact of not having it is negligible.

[trungnt2910]

That problem is easily solved by passing the -mno-red-zone flag.

The problem here is normal, userland Linux binaries found in the wild are not compiled with that flag set, and as a Linux emulator we should be able to emulate these binaries as well.

[jart]

Those red zone binaries will emulate just fine, since they're running in a virtualized space. This will make sense if you flip the LOG_COD bit in blink/log.h and look at the /tmp/blink.s assembly output. For example, here's an instruction that uses the red zone:

/       mov     -4(%rsp),%eax # 8b 44 24 fc @ 4011ab
/       a1iqm
/       Br0C
/       Ltmtm
/       a2ia1iqm
/       s0a1=a0=m
        mov     $4,%esi
        mov     %rbx,%rdi
        add     %rsi,(%rdi)
        mov     %sil,8(%rdi)
        mov     %rbx,%rdi
        mov     $4,%edx
        mov     $-4,%rsi
        mov     %rbx,%rdi
        mov     0x40(%rdi,%rdx,8),%rax
        add     %rsi,%rax
        mov     %rax,%rdi
        movabs  $0x88800000000,%rax
        add     %rdi,%rax
        mov     %rax,%rdi
        mov     (%rdi),%eax
        mov     %rbx,%rsi
        mov     %rax,%rdi
        mov     %edi,%eax
        mov     %rax,0x40(%rsi)

As we can see, Blink's JIT doesn't generate any red zone instructions. Consider also a CALL instruction. It doesn't even use CALL.

/       call    0x561f57ca552e # e8 69 fc ff ff @ 4013a2
/       a2ia1iqm
/       a1imq
        mov     $7,%edx
        mov     $5,%esi
        mov     %rbx,%rdi
        add     %rdx,(%rdi)
        mov     %sil,8(%rdi)
        mov     %rbx,%rdi
        mov     $-0x397,%rsi
        mov     0x60(%rdi),%rdx
        mov     (%rdi),%rax
        lea     -8(%rdx),%rcx
        mov     %rcx,0x60(%rdi)
        movabs  $0x887fffffff8,%rcx
        mov     %rax,(%rdx,%rcx)
        add     %rsi,%rax
        mov     %rax,(%rdi)
        mov     %rbx,%rdi
        jmp     0x561f57ca2000

[jart]

We're two steps away from having JIT on Cygwin.

[jart]

I got JIT mostly working on Cygwin. I had to change nearly every line in the entire codebase to do it. However there's still a lot more work that needs to be done on this before it can be merged into master. See the Cygwin branch I've just created. https://github.com/jart/blink/commit/686b7e6d5335c5b3c6b1db36d617477b68779ae5

[trungnt2910]

I can see that JIT has been disabled on Cygwin by default, but how can I enable it (from the code on the master branch)?

Also, would you mind a GitHub Actions for testing on Cygwin, and possibly all other supported OSes on GitHub Actions other than Linux? If that's the case, I can upstream my actions on the fork.

[trungnt2910]

Don't know if this is related to the JIT, or more related to #26, but tests are still failing:

o//blink/blink third_party/cosmo/2/sigaction_test.com
I2023-01-17T03:05:07.743414:blink/syscall.c:2261:3638 unrecognized sigaction() flags: 0x2
error:test/libc/calls/sigaction_test.c:200: sigaction_autoZombieSlayer() on fv-az449-110 pid 3638 tid 3638
    ASSERT_EQ(0, sigaction(SIGCHLD, &sa, &sa))
        need 0  (or 0 or  =
         got -1 (or 0xffffffffffffffff)
    EINVAL/22/Invalid argument
    third_party/cosmo/2/sigaction_test.com @ fv-az449-110
1 / 9 tests failed
make: *** [third_party/cosmo/cosmo.mk:20: o//third_party/cosmo/2/sigaction_test.com.ok] Error 1
make: *** Waiting for unfinished jobs....

The branch is based on unmodified code from master, no special configuration.

See the full logs here.

[jart]

I can see that JIT has been disabled on Cygwin by default, but how can I enable it (from the code on the master branch)?

Right now you need to comment this out:

https://github.com/jart/blink/blob/8ae81a27595269685a1d395464b1127920b7f376/blink/blink.c#L148-L151

Also, would you mind a GitHub Actions for testing on Cygwin, and possibly all other supported OSes on GitHub Actions other than Linux? If that's the case, I can upstream my actions on the fork.

I would happily merge that.

Don't know if this is related to the JIT, or more related to https://github.com/jart/blink/issues/26, but tests are still failing:

Sorry I spoke too soon in that commit. I've fixed a bunch of failing Cygwin tests in https://github.com/jart/blink/commit/8ae81a27595269685a1d395464b1127920b7f376 I'm still working on clearing out the rest.

[trungnt2910]

With #34 closed and #26 completely solved, let's see how JIT works on Cygwin...

https://github.com/trungnt2910/blink/actions/runs/3946815784/jobs/6754936540

Seems like all is failing with a segfault.

[jart]

According to GDB the segfault is happening in Actor() because the callee-saved register %rsi is being clobbered. I'm still not sure yet why.