Coordinate porting to Windows

[straight-shoota]

UPDATE 2023-04-24:

All the main platform features for Windows are finished! 🚀

There are still some smaller stories pending. Project board: https://github.com/orgs/crystal-lang/projects/11/views/5

You can support the ongoing development financially: Windows suppport project on Open Collective.

UPDATE 2021-11-18:

By now, the compiler works pretty well on windows and most of the standard library has been ported as well. Besides a couple other smaller missing pieces, the major gap is in concurrency-related features held back by the missing event loop implementation.

---

With #5339 being merged, the first step of bringing Crystal to Windows has been accomplished! With current master branch it is possible to cross-compile and run simple programs (alá Hello, World!) on native Windows.

Obviously we're still far away from porting the compiler or entire standard library to windows. This issue is supposed to coordinate and keep track of ongoing efforts to add support for more and more of the crystal standard library.

TODO

The primary focus should be on the core library (somewhat odered by priority):

• [x] File descriptors #5339
• [x] GC #5339
• [x] Errno #5339
• [x] Basic Exceptions #5339
• [x] Regex #5339
• [x] Random #5539
• [x] Time #5422
• [x] Raising #6419
• [x] File #5623
• [x] Dir #5623
• [x] ENV #6333
• [x] Fibers #7995
• [x] Event loop #12149
• [x] Signal #7339
• [x] Mutex #12213
• [x] Sockets #10696
• [x] Process
• [x] Backtraces #11461
• [x] Iconv (#11480)
• [x] System #6972
• [x] Threads (#11647)
• [x] Compiler #9032

And of course a CI infrastructure once specs are running on windows.

Proceedings

The general course of action is to extract platform-specific implementations and then add the implementation for win32 platform.

A lot of work has already been done in #3582 but it needs to be updated and chopped into smaller pieces of changes.

The first goal should probably be to get specs running on windows to be able to verify the windows implementations. Essential for this are as far as I can see file, dir, exceptions (backtrace!), process and signal.

If you want to help with that effort, or want to "take" a piece of that work to work on, please comment below.

Useful References

• Wiki page Porting to Windows

• 26 Support Windows as a first class platform (note: not WSL or emulation)

• 3582 [WIP] Windows by @lbguilherme

• 5339 "hello world" windows port by @RX14

• 4832 Win: define Win32 API functions by @txe

• 5448 Remove c/winapi and rename some windows types (some pieces from removed c/winapi can be salvaged for implementations)

Please feel free to add and edit this post as appropriate (core team).

[malte-v]

@bew Done. #7339 I'll keep my comments here so it's clear what this conversation was about. Or should I delete them?

[hanyuone]

Should the current build process on Windows be simpler? Some of the steps in the "Porting to Windows" page on the Wiki, such as downloading LLVM and GC, could be automated via shell script, and the cl command is really unwieldy if you use libraries. For example, when compiling an example for my library, hedron, on Windows, I have to use this:

cl control_gallery.o /Fecontrol_gallery _rc2bin\resources.res gc.lib pcre.lib libui.lib user32.lib kernel32.lib gdi32.lib comctl32.lib uxtheme.lib msimg32.lib comdlg32.lib d2d1.lib dwrite.lib ole32.lib oleaut32.lib oleacc.lib windowscodecs.lib msvcrt.lib legacy_stdio_definitions.lib advapi32.lib

It'd be nice if the compilation process on Windows were at least simplified - that way, more people could test for problems in the new platform.

[straight-shoota]

The wiki page already provides a link to an installation script in the very first step. It should build/install all required libraries. The last section How to Build Libs just explains how to build manually.

For compiling Crystal binaries, the cross compilation script should be good enough for most simple use cases. Currently, you can't do much more on windows anyway until we get fiber and proper IO implemented. So I don't expect many people would benefit from experimenting on windows in the current state.

But IIRC @RX14 said he isn't too far away from porting the compiler to be self-hosting on windows. Going native would obviously simplify the build process and remove the need to cross compile.

[sam0x17]

I came across this and thought it might have tidbits useful for the team trying to add windows fibers support: https://nullprogram.com/blog/2019/03/28/

@straight-shoota @RX14 @asterite

[rishavs]

Hi guys. Any updates on the overall porting?

[RX14]

I've been super busy with uni this year unfortunately, if people want quick progress on windows I suggest they make sure I'm busy with crystal over the summer by sponsoring me.

[rishavs]

Hey Chris How does one go about it?

[Sija]

@RX14 perhaps new GH feature would come handy here, see https://help.github.com/en/articles/receiving-sponsorships-as-a-sponsored-developer

[RX14]

@RX14 perhaps new GH feature would come handy here, see https://help.github.com/en/articles/receiving-sponsorships-as-a-sponsored-developer

Well, that's good timing for github to announce that right after I posted.

I'll probably have time to set that up after my exam on Tuesday. If people want to sponsor me with another method I suggest they contact me through the email on my profile.

[danieldaeschle]

I'm waiting for it <3 Please finish it. I don't know if i could help. I have no experience with building a compiler.

[wdhwg001]

Should we update the Fibers section, changing the related PR from #6955 to #7995 ?

[mrakobeze]

I may not know something, but, huh, is it such a necessary to use MSVC? Could not it be easier to use MinGW/MSYS/Cygwin to build Crystal on windows?

[asterite]

@mrakobeze It's an easier step for us Crystal developers but it means you need those tools as a developer which is harder for a user. I also think MSVC might be faster than using MinGW/others.

[straight-shoota]

@mrakobeze This was already asked and answered in https://github.com/crystal-lang/crystal/issues/5430#issuecomment-387780310 and following

[Edregol]

I'm not well informed regarding this topic, but as far as I know using MSVC brings some licensing conditions if the endproduct is used commercially. The only reason to use MSVC is to have the linker for the windows platform, right? As far as I know LLVM brings a complete toolchain for windows targets including a linker. Is there a reason not to pick LLVM? Or am I completely on the wrong track here?

[straight-shoota]

It might eventually be an option to use only the LLVM toolchain, but we not only need a linker but also a number of system libraries and they're provided by MSVC.

Rust had this idea as as well, but it seems they still haven't gotten anywhere either. It's a huge task and I think we're better off taking the short way to get windows support working first.

[rdp]

regarding MinGW see https://github.com/crystal-lang/crystal/issues/6170

[mrakobeze]

@asterite as of my expirience, MinGW compiled projects are a bit faster on my machine than MSVC ones. About difficulties to end-developer using MinGW/MSYS/etc, for example ruby needs MSYS env to compile gems native exts, and, as far as I know almost nobody sees this as a problem. And, finally, after MSVC support comes, will there be MInGW/MSYS/etc support? I think I could contribute to porting Crystal for MSYS, but, unfortunately, not MSVC as I don't use it. Moreover, I haven't MSVC on my machine, MSYS only.

[mrakobeze]

And finally, support of LLVM Clang on Windows - is it available yet?

[straight-shoota]

@mrakobeze Support for MinGW is very welcome! We'd very much appreciate if you wan to contribute to that. There is even a separate issue to track that: #6170 I suppose it's probably going to to be easier, to not run two tracks parallel but wait until MSVC support is at least somewhat stable.

Regarding LLVM on windows, see my previous comment: https://github.com/crystal-lang/crystal/issues/5430#issuecomment-519410752

[cfsamson]

I just wanted to bring up the natural question of what's next?

Now that basic fiber support is unblocked there are no more tracking issues for the other bullet points. I could possibly contribute with some work on sockets depending weather we're talking basic blocking socket support or or if we're thinking of non-blocking, overlapped sockets meant to be used in an IOCP event loop.

[straight-shoota]

@cfsamson Basic fiber support is working, but we're still missing the entire event loop, which is a requirement for any IO work. We need at least the first three steps from #6957 to commence porting sockets. The event loop is probably going to be quite a challenge to work out a platform-indepemdemt design and implement. We'll need to start a dedicated discussion about that. Help is of course always welcome. Other features like backtraces could probably be worked on right now, but they're not critical (although it would certainly help with the ongoing efforts on windows).

[cfsamson]

Ok, so I guess #6957 is the best issue to follow going forward then. I've been looking into cross platform event loops covering epoll/kqueue/IOCP for a different project and a safe and performant implementation with an ergonomic api is not a trivial task, but we got to start somewhere.

[asterite]

@cfsamson I recommend waiting for #8112 to be merged first.

[rishavs]

Now that #8112 is merged, I suppose #6957 and #7995 are next. Also, should someone start a thread on a platform independent implementation of event loop?

[wdhwg001]

Now #7995 has been merged, I suppose it's time to update the to-do list.

Also, maybe we should add another item Events to the list?

[AxDSan]

9 more implementations left!!! SO CLOSE!

[RX14]

Sorry to break the bad news, but we're not close to close, and my resources to work on this have basically vanished. Each one of those remaining tasks is hard, and many API issues still are yet to be solved. Don't expect this any time soon.

On the plus side, I'm having way more fun not battling with microsoft's crusty APIs, instead building a cubesat :)

[AxDSan]

Dang it, anyways thanks for keepin' it real, and letting us know, I thought this was gonna be a breeze :(

[rubyFeedback]

People on reddit etc... often reference this issue, which is now ~2 years old.

I would like to recommend to the crystal team to eventually think about closing the issue, BUT continue with a NEW one, including updated stuff, in particular what is missing - since evidently things improved quite a lot in the last 2 years too.

It's difficult for non-crystal centric users to keep track of what is going on in this regard.

[straight-shoota]

This issue is supposed to track and coordinate efforts for porting to win32. For this purpose, it doesn't matter how old it is. The progress tracker in the first post is updated continuously. Please avoid detail and meta discussions.

[rdp]

for discussion on the eventloop (libuv et al) see #6957 (maybe it could be added to the checkbox at the top?)

[jkthorne]

I donate to Crystal generally but is there a way to donate to help this issue move along?

[oprypin]

For the purpose of coordination, I'm currently porting Process and Signal.

[oprypin]

Oh nevermind, guess @jan-zajic is farther along in that anyway. https://github.com/crystal-lang/crystal/pulls?q=is%3Apr+author%3Ajan-zajic+process

[RX14]

@oprypin there hasn't been movement in those PRs for some time, and I'd really love to get them merged. Perhaps you could contact @jan-zajic and work out if they need to be picked up

[oprypin]

There has indeed been a reply, so I picked up Process after all.

Tracking this task in https://github.com/crystal-lang/crystal/issues/9032

[oprypin]

So, now there's a minimal port of Process for Windows (turned out mostly based on @RX14's earlier work, though there's also significant inspiration from @jan-zajic's work).

And... the Crystal compiler can compile itself on Windows. See https://github.com/crystal-lang/crystal/issues/9032 for details.

I also reworked the Porting to Windows wiki page.

There's still some work I need to get merged, including advanced features for Process by @jan-zajic. It'll be in my branch for now.

[asterite]

@oprypin You are AWESOME

[jkthorne]

I was having trouble with the old wiki page. Thanks for the update!!

[wdhwg001]

@ oprypin you are a legend! Does that mean we can finally have crystal_i386.exe and crystal_x86_64.exe compilers in the next release before 1.0 if it gets merged?

(A month ago we were planning to release 1.0 without Windows support... You've changed the world.)

[oprypin]

This is not Windows support, just a small (I'd even say relatively easy) but particularly juicy part of it. See https://github.com/crystal-lang/crystal/issues/5430#issuecomment-548759783, that hasn't really changed.

But I suppose yes, distributing some .exe as part of the release should be considered, to simplify a few first steps of the initial setup.

[oprypin]

So, Windows support is in a state where you can try it out quite easily and build programs natively, without WSL or cross-compilation. But very important parts of the functionality are still broken or missing -- most notably, networking of any kind.

For the curious, there's a quick start guide.

[rishavs]

Maybe its time to look at the packaging then. A simple way to install and setup the environment would be great to get people try it out and share feedback.

While a full-on GUI based installer may still be too much at this point in time, but how about adding a scoop installer?

[oprypin]

Packaging is https://github.com/crystal-lang/distribution-scripts/issues/64

[straight-shoota]

@nu8 That's completely unrelated here. There is however already a shard with libcurl bindings: https://github.com/blocknotes/curl-crystal

[straight-shoota]

Na we just need to port Socket to win32. That shouldn't be too difficult, I suppose. Libcurl can't replace that. And we don't want to have another dependency for stdlib.

[watzon]

@nu8 he was referring to networking in the standard library. LibCurl bindings are unlikely to be included there when we can just port Socket. But yes, in the mean time people on Windows are free to use LibCurl bindings.

[kubo]

In https://github.com/crystal-lang/crystal/pull/9257#issuecomment-639447695:

It's intended that we migrate from the "half-posix" file API to the full windows API. I used the unix-like wrappers only because of the ease of storing a fd instead of a HANDLE*, but this all needs refactoring to ideally use handles everywhere.

May I create a pull request to use HANDLE directly instead of fd in IO::FileDescriptor? I think it is necessary to invoke file I/O in the event loop because IOCP requires file handles with FILE_FLAG_OVERLAPPED. It passes cross-compilation of spec/std_spec.cr on linux. I have not run it on Windows. I bet there are some flaws. (See https://github.com/kubo/crystal/commit/15f0ecebaa4b6c8f5a567475e503bdbaa5bbc106)

EDITED: File handles opened with FILE_FLAG_OVERLAPPED works well except one critical issue. They cannot be passed to standard I/O of child processes. I tried to drop FILE_FLAG_OVERLAPPED by using ReOpenFile with no luck. I checked what nodejs does on Windows. It seems to use IOCP only for sockets, named pipes and monitoring file system changes. It seems to use separate threads for file I/O. I won't create the pull request in the end.

[rubyFeedback]

Is there an update to this? Any showstoppers?