Coordinate porting to Windows

[straight-shoota]

UPDATE 2023-04-24:

All the main platform features for Windows are finished! 🚀

There are still some smaller stories pending. Project board: https://github.com/orgs/crystal-lang/projects/11/views/5

You can support the ongoing development financially: Windows suppport project on Open Collective.

UPDATE 2021-11-18:

By now, the compiler works pretty well on windows and most of the standard library has been ported as well. Besides a couple other smaller missing pieces, the major gap is in concurrency-related features held back by the missing event loop implementation.

---

With #5339 being merged, the first step of bringing Crystal to Windows has been accomplished! With current master branch it is possible to cross-compile and run simple programs (alá Hello, World!) on native Windows.

Obviously we're still far away from porting the compiler or entire standard library to windows. This issue is supposed to coordinate and keep track of ongoing efforts to add support for more and more of the crystal standard library.

TODO

The primary focus should be on the core library (somewhat odered by priority):

• [x] File descriptors #5339
• [x] GC #5339
• [x] Errno #5339
• [x] Basic Exceptions #5339
• [x] Regex #5339
• [x] Random #5539
• [x] Time #5422
• [x] Raising #6419
• [x] File #5623
• [x] Dir #5623
• [x] ENV #6333
• [x] Fibers #7995
• [x] Event loop #12149
• [x] Signal #7339
• [x] Mutex #12213
• [x] Sockets #10696
• [x] Process
• [x] Backtraces #11461
• [x] Iconv (#11480)
• [x] System #6972
• [x] Threads (#11647)
• [x] Compiler #9032

And of course a CI infrastructure once specs are running on windows.

Proceedings

The general course of action is to extract platform-specific implementations and then add the implementation for win32 platform.

A lot of work has already been done in #3582 but it needs to be updated and chopped into smaller pieces of changes.

The first goal should probably be to get specs running on windows to be able to verify the windows implementations. Essential for this are as far as I can see file, dir, exceptions (backtrace!), process and signal.

If you want to help with that effort, or want to "take" a piece of that work to work on, please comment below.

Useful References

• Wiki page Porting to Windows

• 26 Support Windows as a first class platform (note: not WSL or emulation)

• 3582 [WIP] Windows by @lbguilherme

• 5339 "hello world" windows port by @RX14

• 4832 Win: define Win32 API functions by @txe

• 5448 Remove c/winapi and rename some windows types (some pieces from removed c/winapi can be salvaged for implementations)

Please feel free to add and edit this post as appropriate (core team).

[RX14]

@straight-shoota that's a good point. We need to figure out the basics of getting a blank spec suite working on windows (we should do this first without unwind support, just the raise==exit we have now). Then we need to create some kind of win32_spec.cr which requires only the parts of spec/std that work. I'll take on this task (tomorrow).

[RX14]

Another area of priority is looking at https://github.com/crystal-lang/crystal/commit/c168035853fe8ee9b06a22e5890414e5efa61776 and working out ways of reducing that stubbing. For example: get iconv working so we can remove the stubbing in src/io/encoding.cr. The modified parts of the corelib are actually I think worth focus much more than the missing parts (for now).

[straight-shoota]

A very basic working spec shouldn't be too hard I think. I've tried it earlier to figure out what might be needed for that. It seems to be missing mostly some basic file and dir methods. After disabling all feature options I even got a simple spec to compile, allthough it resulted a LLVM error at codegen.

[s5bug]

I'll take Fibers, I'll be able to start work on them after January 2nd.

[RX14]

@straight-shoota looks like my spec PR has turned into a Dir PR first.

[RX14]

Also of importance: deciding what to do about windows's UTF-16 use. There should definitely be a really quick and easy way of going between String -> UTF-16 WCHAR* and the reverse. We should also have a policy of no A functions, only W functions even this early in the port.

This is an important issue: we shouldn't rely on iconv for this (I presume it's slow for small strings, and we don't have it in windows anyway). We need to write UTF-16 infrastructure for all platforms first.

[RX14]

Good news: spec compiles and works on my fork.

Bad news: it only works sometimes.

From this spec failure, looks like we have a codegen or ABI bug somewhere:

[RX14]

Here's my branch porting spec: https://github.com/RX14/crystal/commits/feature/windows-spec

I'll turn "Remove c/winbase" and "Add Crystal::System::Dir" into PRs.

[RX14]

Another idea before I forget it:

Consolidate all stubbing out into one huge windows_stubs.cr file so that we can keep track of them all.

[straight-shoota]

You mean to remove winapi.cr, not winbase. And this commit should also rename DWord.

[RX14]

@straight-shoota yep, thanks, commit message is wrong. Shouldn't be working at 1am...

[RX14]

Made those PRs: #5447 and #5448. I hope you don't mind me editing the checklist in your issue @straight-shoota.

[straight-shoota]

Looks great!

I've deliberatly added a note at the end of the OP for that purpose ;)

[felipetesc]

Great news! Thank you for this.

[drhuffman12]

In https://github.com/crystal-lang/crystal/wiki/Porting-to-Windows, should:

Ongoing efforts will be tracked and coordinated in #

... be:

Ongoing efforts will be tracked and coordinated in #5430?

[straight-shoota]

@drhuffman12 thanks, fixed!

[drhuffman12]

@straight-shoota , yw

[s5bug]

Due to time constraints and last minute events I won't be able to work on Fibers as expected :(. If someone else wants to claim them, that would be amazing.

[RX14]

If you can get openssh server working on WSL (hint: you need to edit /etc/ssh/sshd_conf to set UsePrivilegeSeparation no and change Port 2222), you can use the following script to make the windows testing process seamless:

#!/bin/sh
set -euo pipefail

windows=192.168.xxx.xxx

bin/crystal build --progress --cross-compile --target x86_64-unknown-windows-msvc -o .build/cross "${@}"
scp -P 2222 .build/cross.o $windows:/mnt/c/crystal/cross.o

cat <<EOF | ssh -p 2222 $windows bash
export PATH="$PATH:/mnt/c/Windows/System32"
cmd.exe /S /c "C:\Program^ Files^ ^(x86^)\Microsoft^ Visual^ Studio\2017\BuildTools\VC\Auxiliary\Build\vcvarsall.bat x64 && cd \crystal && cl cross.o /Fecross pcre.lib gc.lib libcmt.lib"

cd /mnt/c/crystal

echo
./cross.exe
EOF

This script assumes you have gc.lib and pcre.lib in C:\crystal and that you're using the visual studio 2017 build tools. MSVC 2015 can surely just change the path to vcvarsall.bat.

[RX14]

A script to automatically generate a compilable manifest of the entire spec suite, with broken tests commented out with a vague reason:

cd spec
for spec in ./std/**/*_spec.cr; do
  (cd .. && crystal-windows spec/$spec) >/dev/null 2>/dev/null && echo "require \"$spec\"" ||
    (../bin/crystal build --no-codegen --target x86_64--windows-msvc $spec >/dev/null 2>/dev/null && echo "# require \"$spec\" (failed to run)" || echo "# require \"$spec\" (failed to compile)")
done

crystal-windows is the script from above.

[straight-shoota]

The script posted by @RX14 is intended to be run on an external system which connects to WSL through SSH to upload the object file and run the linker through Windows interoperability feature. If you want to run the compiler locally on WSL, you don't need to go through SSH. I have put together a crystal-windows wrapper for this: https://gist.github.com/straight-shoota/e4393c9e3f0febb7dc290470f92f5c0d

[txe]

As I see, there is no dir.cr in crystal/system/win32, so I'm going to make it. Is there any idea what to do with UTF-16?

[txe]

About UTF-16: I suppose we need just a couple of functions to encode\decode utf-16 without any len, find and others functions and can keep utf-16 string as array of wchar. FYI, Go has this code for that and Nim this

[asterite]

@txe Please coordinate with @RX14 , as far as I know he was porting File to Windows (not sure about Dir). For UTF16 we now have this: https://github.com/crystal-lang/crystal/pull/5541 . But check the PR's comments, we might need to change it a bit to include a zero at the end when encoding to Slice(UInt16).

[RX14]

@txe Fantastic to see you back!

I'm working on File and Dir right now, they're both mostly complete but the new code isn't in my branch (uncomitted on my PC). UTF-16 also has an added tweak in my branch.

I have a lot of work not in master, perhaps committing my branch would help you have more of a base to work off of?

Namely, working in my branch but not master: File, Dir, Spec. I also have a spec/windows_spec.cr which lists all working specs.

[j8r]

@RX14 Yes the best is to create a dedicated branch, with a "WIP" pull request to master, even if it's a bit trashy. Everyone will know you work on it - this can avoid duplicate work and help others.

[asterite]

In fact, since none of this will "matter" until we can bootstrap the compiler in Windows, I would prefer small PRs that do some small functionality, even if incomplete. Then we can merge it (there's no risk of breaking anything) and everyone can already play with that and continue improving it. I prefer this instead of working on a branch.

[txe]

Good! I'm glad to hear that there is so much progress. Then I'll take a look at porting Fiber, it shouldn't be hard, because it already worked in @lbguilherme's branch.

[RX14]

@txe Feel free to ignore this suggestion - but actually I think the most important part of the port right now is stack unwinding and exception handling on windows. You seem to have this work largely done on your old windows branch, it'd be fantastic if you could PR it to master.

[txe]

OK, lets see what I can do with exceptions.

[RX14]

OK, i've completed my File port, and file_spec.cr now compiles and runs on windows. The results are at feature/windows-spec. I've split 3 new PRs out from that branch: #5579, #5580, and #5581.

After that, I'll PR a full File and Dir port, then PR a Spec port and we can run from there!

[txe]

What can I do next after exceptions?

[RX14]

@txe probably fiber (your original plan) would be fairly easy.

I think a lot of things are stuck on me porting spec/File/Dir right now. Though my exams are over so I should be able to get that PRed soon.

[RX14]

Just PRed #5622 and #5623. The latter ports File and Dir to win32.

I changed my "windows" branch to just be called "windows", so if you want to follow the latest work on the windows port it should now be https://github.com/RX14/crystal/tree/windows.

[BobG1983]

Hey. So, if I'm interested in getting involved here, do we have a setup guide?

[bew]

Yes @ShriekBob, you can find setup informations on the wiki: https://github.com/crystal-lang/crystal/wiki/Porting-to-Windows and in this thread.

[straight-shoota]

If you encounter any issues, feel free to ask on Gitter/IRC https://gitter.im/crystal-lang/crystal

[RX14]

Currently halfway through porting Process.

[nof1000]

So is there any news on this?

[RX14]

Just a few notes to people building pcre and gc (mostly me):

for bdwgc:

1. git clone -b v7.6.4 https://github.com/ivmai/bdwgc/
2. cd bdwgc
3. git clone -b v7.6.2 https://github.com/ivmai/libatomic_opts
4. Download https://gist.github.com/ynkdir/688e62f419e5374347bf as C:\path\to\bdwgc\ntwin32.mak
5. nmake -f NT_X64_STATIC_THREADS_MAKEFILE nodebug=1 _CL_=-DDONT_USE_USER32_DLL
6. use gc.lib

for pcre:

1. download pcre-8.41
2. mkdir build && cd build
3. cmake .. -G "Visual Studio 15 2017 Win64" -DBUILD_SHARED_LIBS=OFF -DPCRE_SUPPORT_UTF=ON -DPCRE_SUPPORT_JIT=ON -DPCRE_STATIC_RUNTIME=ON
4. cmake --build . --config release
5. use release/pcre.lib

[RX14]

And a new crystal-windows script for use with a windows-native ssh server:

#!/bin/sh
set -euo pipefail

windows=192.168.122.76

bin/crystal build --progress --cross-compile --target x86_64-unknown-windows-msvc -o .build/cross "${@}"
scp .build/cross.o $windows:'C:\Crystal\cross.obj'

cat <<'EOF' | ssh $windows cmd /S /C "C:\Program^ Files^ ^(x86^)\Microsoft^ Visual^ Studio\2017\BuildTools\VC\Auxiliary\Build\vcvarsall.bat x64 && powershell -Command -"
cd C:\Crystal
del cross.exe

cl cross.obj /Fecross pcre.lib gc.lib advapi32.lib libcmt.lib # C:\data\programming\crystal-lang\crystal\windows\src\llvm\ext\llvm_ext.obj "$(llvm-config --libs --system-libs --ldflags --link-static)"

echo ""
./cross.exe --verbose
EOF

[RX14]

And for building LLVM:

cmake .. -G "Visual Studio 15 2017 Win64" -Thost=x64 -DCMAKE_INSTALL_PREFIX="C:\Program Files\llvm" -DLLVM_TARGETS_TO_BUILD="X86" -DLLVM_PARALLEL_LINK_JOBS=1 -DLLVM_BUILD_LLVM_DYLIB=ON -DLLVM_USE_CRT_RELEASE=MT
cmake --build . --config release
cmake --build . --config release --target install

needs python installed and in PATH

[straight-shoota]

@RX14 Are step 4 and 5 for libpcre are in wrong order?

I updated the instructions on the wiki page.

[RX14]

@straight-shoota yep, thanks

[RX14]

@straight-shoota instructions are wrong: should be saved as ntwin32.mak. The "known issues" can be deleted since -DDONT_USE_USER32_DLL solves that.

[rdp]

Just curious, why target msvc vs. mingw-w64?

[RX14]

@rdp far less dependencies and it's not that much more difficult.

[rdp]

mingw-w64 is just way easier for cross compiling, but I suppose support can be added for it "later" as it were :) Also should sockets be added to the list?

[RX14]

@rdp cross-compiling is already simple

[sam0x17]

msvc is also pretty much as "first class" as you can get on windows, and I am sure there are some optimization / performance reasons to back that up, so targeting it is definitely a good idea (correct me if I'm wrong). This is why big tools like node-gyp use msvc for the most part.