search

crystal-lang / crystal

The Crystal Programming Language
https://crystal-lang.org
Apache License 2.0
19.5k stars 1.62k forks source link

Coordinate porting to Windows #5430

Open straight-shoota opened 6 years ago

straight-shoota commented 6 years ago

UPDATE 2023-04-24:

All the main platform features for Windows are finished! 🚀

There are still some smaller stories pending. Project board: https://github.com/orgs/crystal-lang/projects/11/views/5

You can support the ongoing development financially: Windows suppport project on Open Collective.

UPDATE 2021-11-18:

By now, the compiler works pretty well on windows and most of the standard library has been ported as well. Besides a couple other smaller missing pieces, the major gap is in concurrency-related features held back by the missing event loop implementation.

With #5339 being merged, the first step of bringing Crystal to Windows has been accomplished! With current master branch it is possible to cross-compile and run simple programs (alá Hello, World!) on native Windows.

Obviously we're still far away from porting the compiler or entire standard library to windows. This issue is supposed to coordinate and keep track of ongoing efforts to add support for more and more of the crystal standard library.

TODO

The primary focus should be on the core library (somewhat odered by priority):

And of course a CI infrastructure once specs are running on windows.

Proceedings

The general course of action is to extract platform-specific implementations and then add the implementation for win32 platform.

A lot of work has already been done in #3582 but it needs to be updated and chopped into smaller pieces of changes.

The first goal should probably be to get specs running on windows to be able to verify the windows implementations. Essential for this are as far as I can see file, dir, exceptions (backtrace!), process and signal.

If you want to help with that effort, or want to "take" a piece of that work to work on, please comment below.

Useful References

Please feel free to add and edit this post as appropriate (core team).

straight-shoota commented 6 years ago

Also should sockets be added to the list?

done.

straight-shoota commented 6 years ago

There's no point in arguing mingw64 would be easier when we've already got msvc working quite nice. And for now the focus should be on actually porting the stdlib to win32 API instead of supporting another compiler tool. Eventually, we'll be supporting both like we have multiple backends for other platforms as well. But right now I don't see any benefit from switching to mingw64.

konovod commented 6 years ago

Mingw allows to compile windows programs without using windows or wine at all, using only linux tools and linux compiler. This can be useful in some situations until Windows support is self-hosted, e.g. for people without WSL. As for performance - afaik there is no significant difference - there is no "extra layer" like in cygwin, so both mingw and msvc are "native". Many (most?) opensource tools and libraries support both toolchains. That said, of course having Windows support is much better then switching the toolchains, and as I remember msvc was chosen because there was some problem with Exceptions handling in mingw.

RX14 commented 6 years ago

Mingw comes with the win32 API, which is what we're coding against. It's the same API, so all the current work is applicable to both. Arguing about which is like arguing about glibc or musl: the answer is both.

I wouldn't be surprised if mingw is in fact already supported simply by copying the libc folder to the right triple. Any more talking on the topic is pointless.

RX14 commented 6 years ago

@svnpenn correct me if i'm wrong, but that just seems to be minor header definition incompatibilities. Crystal doesn't use or parse header files so we don't have the same problem.

Either way, it'll be far far less work than porting to windows initially. If someone wants to work on it, they can send PRs.

RX14 commented 6 years ago

@svnpenn that's referring to src/lib_c, crystal's bindings to libc (written in crystal). There's no headers or header parsing involved. As long as mingw provides all the functions crystal uses, with the same names, and the same arguments, it should be trivial to port.

RX14 commented 6 years ago

If mingw is "sizeof() compatible" with msvc (any calling convention differences are handles by LLVM), then the port becomes a single cp command. If not, but the function names are still the same, it's still probably trivial.

straight-shoota commented 6 years ago

This issue's purpose is to coordinate efforts, not arguing about how to proceed. Please keep this issue clear of such detailed discussions. If someone feels to talk about mingw64 support, open an issue for it. Thanks.

straight-shoota commented 6 years ago

I'll work on porting the missing pieces of ENV. Should be fairly straightforward.

Update: #6333

Neltherion commented 6 years ago

It's always great to hear new progress on Windows... Thanks...

sdogruyol commented 6 years ago

Updated: File and Dir are support just landed with #5623 :+1:

hanyuone commented 6 years ago

Raising should be updated to #6419, shouldn't it? Also, what's the progress on #6333 @straight-shoota?

straight-shoota commented 6 years ago

Todo list is updated.

ENV is good to go and waiting for review & second approval.

thiagomajesk commented 6 years ago

I'm curious, does this Issue pace represent future Windows support/ investment by the core team or is just a rough path for specific stuff related to Crystal compiler? (No offense, just plain curiosity)

RX14 commented 6 years ago

@thiagomajesk i'm not quite sure what you mean? We intend to support windows equally as any other platform going into the future

thiagomajesk commented 6 years ago

@RX14 I'm relieved to 'hear' this. I'd hate to see Crystal going the path Ruby went, creating a completely closed environment culture (regarding OS support). I was just wondering if this port was getting enough attention as other *nix systems.

sam0x17 commented 6 years ago

@thiagomajesk a big problem in the ruby ecosystem is a lot of the FFI-based gems (gems that use local compilation of C/C++ to do things) do not support compilation on windows, and therefore are broken on windows. This has the potential to be a problem with crystal, but I think the difference is since crystal is a compiled language, there are going to be many more people focused on making the cross platform experience better because they want to use crystal to build cross platform desktop apps. This didn't make as much sense with ruby because no one wants to distribute an interpreter-based desktop app, so no one built the desktop app tooling.

So I think the prospects are good -- I for one want to use crystal to develop cross platform desktop experiences, so once windows support is ready, all my shards are going to be cross platform compatible.

edit: except for mongo-orm because I'm limited by what the mongo.cr maintainer decides to do

thiagomajesk commented 6 years ago

@sam0x17 I couldn't agree more. Besides that, I think that the core team has a very important job setting the pace for the community regarding this matter, like discouraging OS specific implementations in favor of cross-platform ones. I'm really anxious to see how things will end up for the community. Thanks for sharing your thoughts.

straight-shoota commented 6 years ago

I think we've come quite a good way on the path to Windows support. We're about halfway through in the TODO list. And the already ported stdlib features are probably more complex then the ones still pending. The next major milestone is Fiber, which also includes porting Thread and Crystal::Event. A first step in that direction is #6955.

Work on porting the compiler could also begin soon, once the Path API has been merged (#5635) and Process is ported.

As an additional note, travis-ci has announced it now provides CI runners on Windows, which could be a great way to add automatic tests for the compiler and stdlib specs.

RX14 commented 6 years ago

libevent won't be used on windows. Fiber must be ported without it. spawn and channel can still work even if IO is blocking and that should be the first step.

bew commented 6 years ago

libevent won't be used on windows

Why? iirc Windows is a supported plateform for libevent

RX14 commented 6 years ago

libevent uses select on windows, and select doesnt support pipes on windows, only sockets. Besides, it's an additional dependency, whereas using IOCP directly isn't that much harder and allows us to support sockets, pipes, files, etc.

Windows's nonblocking IO model is entirely different to how libevent works. On windows you submit entire IO operations and wait for them to complete. With epoll you're waiting for "readable/writable status" of a FD then you can perform an IO.

asterite commented 6 years ago

@RX14 Pipes can be implemented in pure crystal, there's no need to create OS pipes. Go's io.Pipe works like that.

That said, I guess that pipe isn't very useful to share data when forking... but then again, you can't really fork in Go, so that's not a problem there (and it wouldn't be a problem in Crystal if we stopped supporting fork).

RX14 commented 6 years ago

@asterite you need to be able to spawn processes and send IO to them... We need pipes.

RX14 commented 6 years ago

And even if pipes weren't a problem, libevent would still be using select on windows which has terrible performance. IOCP is the only way to go.

asterite commented 6 years ago

Ah, makes sense. I didn't know libevent for Windows just supported select.

RX14 commented 6 years ago

BTW I already pretty much ported the compiler in a branch, I just havent had time to work on the branch recently. Porting the compiler depends on porting Process.

j8r commented 6 years ago

libuv seems to support pipes in its design - I haven't the knowledge to confirm this. I think the best would be to reuse an existing library, unless we have the resources and the knowledge to implement a better, custom solution.

RX14 commented 6 years ago

I seem to recall there were other reasons crystal didn't use libev/libuv, but not what those reasons were. Either way, they can't be used.

RX14 commented 6 years ago

Oh yes, the problem is that libuv's event loop isn't thread safe.

j8r commented 6 years ago

At least uv_async is thread-safe - don't know if its sufficient for our purposes.

straight-shoota commented 6 years ago

Please, lets discuss the details of porting fiber/events in a dedicated issue.

neatorobito commented 6 years ago

For System is it just cpu_count and hostname that need to be ported?

neatorobito commented 6 years ago

I have a working implementation of System for Win32. I'm not super familiar with Crystal's style guide and I'm not sure if I'm correctly handling the unsafe code in hostname. Any pointers (no pun intended) are welcome.

I'm also interested in helping get some CI stuff setup for Windows, we could likely use Azure Pipelines.

wooster0 commented 6 years ago

Just run crystal tool format to follow Crystal's style guide.

straight-shoota commented 6 years ago

@markrjr Looks great! Why don't you just open a PR and we can discuss details there?

rishavs commented 6 years ago

I'm also interested in helping get some CI stuff setup for Windows, we could likely use Azure Pipelines.

I actually work in the AzDev Pipelines team. Let me know if there's anything I can help with.

rishavs commented 5 years ago

Any updates on the porting? Christmas is coming near 😀

straight-shoota commented 5 years ago

Unfortunately not much.

RX14 commented 5 years ago

I'm busy with university currently, but christmas break is coming up soon where I'll likely resume work.

Neltherion commented 5 years ago

Thank you all for the hard work. I'm really excited for the day I'm able to port my current Ruby scripts to Crystal on Windows.

wdhwg001 commented 5 years ago

I think it's time to update the to-do list, #6972 is now merged and is for System instead of Signal.

AxDSan commented 5 years ago

I'm so excited to finally see Crystal in windows 💃 Keep on guys. I wish I could help in something

malte-v commented 5 years ago

How are we going to port Signal? From what I've read, Signals aren't supported at all on Windows. Please correct me if I'm wrong.

RX14 commented 5 years ago

signals are supported: https://docs.microsoft.com/en-us/previous-versions/xdkz3x12(v%3Dvs.140)

just less of them, and badly.

malte-v commented 5 years ago

I already looked at this page, and I think these signals are only there because they are part of the C standard library. From wikipedia:

The C standard defines only 6 signals. They are all defined in signal.h header (csignal header in C++):

SIGABRT - "abort", abnormal termination. SIGFPE - floating point exception. SIGILL - "illegal", invalid instruction. SIGINT - "interrupt", interactive attention request sent to the program. SIGSEGV - "segmentation violation", invalid memory access. SIGTERM - "terminate", termination request sent to the program.

Additional signals may be specified in the signal.h header by the implementation. For example, Unix and Unix-like operating systems (such as Linux) define more than 15 additional signals; see Unix signal.

And these 6 signals are the only ones "supported" by Windows.

There is already an issue about this topic on the Node.js repo. sam-github commented there:

[...] Windows as an O/S has absolutely no signal support. [...]

If SIGTERM was supported on Windows, you could listen for it and receive it... but you cannot. You can listen..., but you'll never receive it.

And I guess this is exactly the case. You have a function for listening to signals (signal()), but there is nothing like kill() for sending signals on Windows.

rdp commented 5 years ago

IIRC Ruby supports it by translating it (or "faking it"), ex: SIGTERM they send a Terminate call or what have you...

On Sat, Jan 26, 2019 at 4:34 AM Malte Voos notifications@github.com wrote:

I already looked at this page, and I think these signals are only there because they are part of the C standard library. From wikipedia https://en.wikipedia.org/wiki/C_signal_handling:

The C standard defines only 6 signals. They are all defined in signal.h header (csignal header in C++):

SIGABRT - "abort", abnormal termination. SIGFPE - floating point exception. SIGILL - "illegal", invalid instruction. SIGINT - "interrupt", interactive attention request sent to the program. SIGSEGV - "segmentation violation", invalid memory access. SIGTERM - "terminate", termination request sent to the program.

Additional signals may be specified in the signal.h header by the implementation. For example, Unix and Unix-like operating systems (such as Linux) define more than 15 additional signals; see Unix signal.

And these 6 signals are the only ones "supported" by Windows.

There is already an issue https://github.com/nodejs/node/issues/12378 about this topic on the Node.js repo. sam-github commented there:

[...] Windows as an O/S has absolutely no signal support. [...]

If SIGTERM was supported on Windows, you could listen for it and receive it... but you cannot. You can listen..., but you'll never receive it.

And I guess this is exactly the case. You have a function for listening to signals (signal()), but there is nothing like kill() for sending signals on Windows.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/crystal-lang/crystal/issues/5430#issuecomment-457824082, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAw0F2eH3U0B2je7yzPuqIeJkyPya4Kks5vHD1TgaJpZM4RKQoA .

malte-v commented 5 years ago

I think "faking" signals is not that easy... For example SIGTERM would politely ask a process to terminate on UNIX. On Windows, you only have TerminateProcess, but this would stop the process immediately, like SIGKILL on UNIX. The only SIGTERM replacement I found would be running taskkill.exe with the PID, which is a rather inelegant solution. My approach would be not to "fake" signals on Windows since this could be misleading and potentially produce different behaviour. I would suggest to find replacements where Crystal itself uses signals (Process for example).

malte-v commented 5 years ago

@rdp Probably you are right. I already found some replacements for signals one might want to send:

bew commented 5 years ago

This issue is for coordination around windows support, and keeps track of where we are. I think the discussion around signals on windows would be better in a separate issue (and maybe move the comments there? Not sure it's possible)