Plans about WebAssembly

[sergei-kucher]

Do you have any plans about WebAssembly support?

[vendethiel]

LLVM already has a branch for that, so Crystal should be covered already ;)

[asterite]

Maybe when there's a WebAssembly specification we can start discussing this, but right now Crystal is more geared towards server and desktop applications, not the web. I'll close this until then.

[Svenskunganka]

A year later, now that a spec is in the works I figured that we could open discussion for this again? Chrome Canary and Firefox Nightly has experimental support for WebAssembly.

@asterite I understand that this is probably not a priority right now - but is this something you guys might be interested in the future?

[lbguilherme]

I believe there is not much to be done from crystal side until both browsers and the llvm target get mature enough. Clang is not ready for this yet.

We can for sure make a proof of concept that works, but not maintain it or port the whole stdlib to work on it yet. I suggest creating a milestone "post-1.0" and leaving this issue open there.

[vendethiel]

Clang is not ready for this yet.

Not sure how Clang is related to Crystal? Once it gets down to LLVM IR, does it matter?

[asterite]

@Svenskunganka I guess so, if you could write Crystal to target the browser it would be awesome. I know nothing about WebAseembly, and just a bit about asm.js, but I don't know what kind of things you can do... dom manipulation? Or just games? In any case, I think we should wait a bit.

[lbguilherme]

@vendethiel It is not in any way related or needed for crystal to work. But it is the one compiler that is working most towards targeting wasm. In fact, this is the goal of the first release of wasm: Have clang targeting wasm and running it in multiple browsers. It is safer and less work for us to let clang test the ground, solve most bugs, and get there before us.

@asterite It let's you access all standard web apis thought some kind of "LibJS", that includes DOM manipulation and everything else. All the stdlib (libc included, I think) would have to be implemented on top of that.

[jwaldrip]

@waj It may be a time to revisit/reopen this issue.

[RX14]

@jwaldrip why?

[pannous]

because the standards are now fixed and the toolchain is working nicely. full support in most browsers and node.js

There is still a big opportunity to become the language of the binary Internet, racing against c++, rust and typescript.

[Svenskunganka]

Today WebAssembly is available in all the major stable browser releases, however its features are limited. It currently has no way to access the DOM or the Web APIs except if you're using JavaScript as the middle man. The WebAssembly Roadmap looks very promising however. Planned features are to allow wasm to directly access and modify the DOM, and call Web APIs without needing any JS at all.

Wasm has an (experimental) LLVM backend, and since Crystal uses LLVM I can imagine it would ease the work required to compile to wasm. Wasm post-MVP roadmap shows plans to implement access to low-level GC, however I'm not sure how this would interact with the Boehm GC that Crystal currently uses. I imagine that a new GC would have to be built specifically for wasm, or that the Boehm GC is adjusted to support the wasm "runtime". But this is only speculation, I'm not knowledgeable enough in that area to draw a conclusion.

Looking at these in progress features, the possibilities are very promising. Being able to compile to wasm would allow the community to create "isomorphic, fullstack" Meteor-like frameworks that shares the same code on both server & client (browser). To simplify, it would allow Crystal to do everything that JS currently does, like having a React-like library with full server-side rendering support. Other examples would be a Slack alternative that compiles to both the browser and regular platforms like Linux, Windows and Mac. I would imagine that Wasm would also be available on smartphones like iOS and Android through WKWebView and the like, so that Slack client could perhaps also be able to run on those platforms using the Wasm code.

It's worth mentioning though that Wasm will not be able to do things that JS currently cannot, like opening a raw TCP socket. We would have to use the alternatives the Web APIs provide like WebSocket and WebRTC.

[drhuffman12]

:+1

[akzhan]

@ysbaddaden what about wasm LLVM target? looks like it's a time to do.

[rhysd]

As long as I know, LLVM 4.0.0 does not support wasm. LLVM HEAD already supports it. So it may be available in 4.0.1 but I'm not sure. I remember LLVM is now in semantic versioning. So I guess it comes with 5.0.0 for stable release.

[ysbaddaden]

Same as other arch's: LLVM does the underlying work, that is generate the binary code (it's awesome), but there is far more to Crystal than just the language: it's library.

Without an official libc (the musl port is unofficial and experimental, advised not to use), we're left with the same state than embedded systems: most of the core and stdlib won't be available, the GC will likely not work, ... not even thinking of the fiber context switches ASM (cold sweat).

WASM may be fun, but it's far off the Crystal roadmap. If someone is courageous (crazy?) enough to work on this, please try, you'll have fun (maybe) :-)

[pannous]

isn't libc provided by emscripten? how else would it run hello_world.c ?

Quote:

Emscripten lets you "link" with JavaScript libraries to do useful things, like render WebGL, run a browser main loop, handle a filesystem, provide bindings to JS so it's easy to call into your compiled code, etc., as well as a full set of compiled libraries like libc and so forth.'

GC can be added later.

[pannous]

Here is how rust does it https://github.com/brson/mir2wasm/

[RX14]

@pannous rust doesn't need a gc.

If someone can demonstrate libgc compiled and working on wasm then this endeavour becomes actually credible.

[pannous]

can't we start without GC? should be fine for demo code which doesn't run for extended periods of time?

[akzhan]

BDWGC on Emscripten (LLVM to WebAssembly backend) now fails on MacOS hosts (but should be OK on others - https://github.com/ivmai/bdwgc/issues/163#issuecomment-304686844).

I suppose it will be fixed with https://github.com/ivmai/bdwgc/issues/163.

[RX14]

So it looks like wasm doesn't have a normal stack at all. I have no idea how we would implement fibers in such an architecture. The note on the GC issue saying that the GC can't walk the stack to find roots is basically a show stopper too. Implementing crystal on wasm looks like it'd require a great deal of thought.

[pannous]

yep, no GC and fibres/threads in first version: http://webassembly.org/docs/future-features/

I'm fine with those crystal features missing in the first adaptation.

[akzhan]

Looks like Crystal cannot be provided until EH, GC and coroutines landed to WebAssembly.

Just wait for these features (they will be added to browsers fast, I suppose).

[RX14]

@pannous I doubt most people would be OK with that or call it a "complete" crystal port so I don't think it's worth the effort at this point in crystal's development. If you or anyone else wishes to work on it i'm sure the core team would be happy to help or merge PRs.

[RX14]

@akzhan I doubt fibers are impossible in webassembly as-is or would get any kind of special support in the future. It's just a matter of finding out how to do it. GC and direct DOM access are things we should wait for.

[catmando]

if anybody else is interested in collaborating on a crystal -> JS transpiler let me know. I think it would have to be a subset without fibers initially. My gut feeling is that we be best to leverage the work down by the opal (ruby -> js) transpiler. I am guessing the major problem is that you have to map crystal class structures, method calls, etc -> JS equivilents, rather than trying to do some lower level emulation. This is what opal does.

[vendethiel]

if anybody else is interested in collaborating on a crystal -> JS transpiler let me know.

wouldn't it be easier to find a way to get a GC/wait for GC?

My gut feeling is that we be best to leverage the work down by the opal (ruby -> js) transpiler.

maybe parts of the parser can be reused, but i don't see anything else

[badlee]

👍 +1

[ylluminate]

Not sure if you guys have seen this thread yet: https://github.com/WebAssembly/binaryen/issues/1312

I believe we can get through the GC issue now with the help as noted there.

Note that by using emscripten we will be able to still run Crystal on non-WASM supporting browsers that only use pure JS.

[RX14]

I'll reopen this for now since wasm has a full spec and implementations in browsers.

[larubujo]

jack of all trades, master of none

[RX14]

@larubujo that's a great reason for not implementing or focussing on implementing wasm now or before 1.0, but I see no harm in keeping this issue open as a place to discuss possibilities. I've made it clear I'm not going to put any effort into wasm personally.

[catmando]

I don't get it... sorry.
WASM provides a way to get crystal running in the browser Crystal running in the browser would greatly speed adoption by making it a full-stack language.
Crystal in browser would allow for a crystal React, DSL and a system similar to ruby-hyperloop to exist.

Understand that you can't do everything at once, but this does seem like it would greatly speed adoption.

What am I missing

[straight-shoota]

Widespread adaption is not an immediate goal (for the core team) - getting a stable 1.0 release of the language is. And even that is conceiveably not really immediate.

As I understand it, WASM support is not considered a critical feature for 1.0, and that seems logical. In the end, it is just another backend Crystal can compile to.

Is WASM really going to be that huge deal? I don't know. Windows support seems certainly much more important and is a requirement for 1.0.

Now, this should not hold anyone off from working on WASM support. If you manage to accomplish something in that direction it will certainly be appreciated. And if it get's done before 1.0 release all the better.

But it's not a high priority for core developers currently. There are more than enough things to work on.

[pannous]

"Windows support seems certainly much more important" An external observer (and fan of the language) dares to strongly disagree here.

• if you compile to wasm, crystal programs can automatically run on all platforms including windows. without installation!
• this is huge! wasm is going to be huge. prediction: it will replace the JVM as the most popular 'portable' platform.
• there are already hundreds of languages running on windows
• right now there is almost no language compiling to wasm

Don't miss the opportunity guys.

If you target wasm you get Windows for free. Also with Ubuntu subsytem you can already compile crystal on Windows? So why waste your valuable time on something almost obsolete, when you can change the world?

[ylluminarious]

Can't agree with @pannous enough. It's really sad to see how many open-source projects dwindle because of a lack of proper marketing. Crystal would be very astute to take advantage of wasm and not just scoff at it as a far-off milestone.

Crystal certainly is at a point where it could take programming languages by storm. It's even higher than Rust and several others on the TIOBE index right now. That could all change in an instant, though. And I bet it will have something to do with whether Crystal runs on wasm.

[ylluminarious]

@catmando is also very on-point and I share his confusion at the resistance to this issue.

[ylluminate]

So we're going to get this done. We have several commitments now from people. If you'd like to get involved, shoot me a note at my username at gmail and we'll get the collaboration going.

[RX14]

The problem I forsee is the standard library. We haven't got most of the stdlib portable to non-posix OSes, let alone the browser. The browser isn't a traditional OS, it has no filesystem and wasm's memory model is pretty wacky compared to the tradtional one.

It'll be extremely hard but if we can port crystal to wasm it'll mean that we know that crystal is extremely portable.

[ylluminate]

@RX14 this is precisely where I see empscripten coming in strong since it provides those for us. Then it will just be a matter of testing and benchmarking to see if we need to do something different, etc.

[RX14]

Oh, emscripten provides a libc I guess. Fair enough, thats probably easier than I thought.

[ghost]

kotlin (native) has wasm support and has garbage collection as well, maybe their approach helps

[catmando]

Stable 1.0 release vs. Wide Spread Adoption.

These are not contradictory. Wide Spread Adoption means you have more people interested in contributing, you have more people using in "anger", you get more realistic use cases, and find those edge case flaws you didn't think of until you get wider usage. This is all builds momentum towards the 1.0 release.

Huge advantage of crystal that everybody talks about is SPEED. Well if you can offload computation to the client's browser, that is really going to give you speed.

[theodorton]

Code sharing between client and server application is IMO a huge argument for targeting wasm. It seems like game development is one of the big applications on WebAssembly and from my little experience with Crystal it seems like it would be a great platform for developing games.

I did some proof-of-concept work on a Crystal to Javascript transpiler last year. My original roadmap was to have a transpiler as 1.0 and convert to tooling for wasm with a 2.0 release as the only goal of the project was to write Crystal code that could be executed one way or another in a browser environment. I'd be happy to help with any work needed. Although my experience with low-level compiler stuff is very limited I have a lot of experience with web applications and requirements for code sharing between server and client.

[ylluminate]

Great stuff @theodorton. I was also previously playing with the direct to JS route a la Opal, which I quite like the idea, BUT it really does seem that by piping things through Emscripten we can do this a lot more quickly and with markedly less effort. You might want to pop over here to chime in and chat with us.

[catmando]

I have a longer conversation here: https://gitter.im/crystal_wasm/implementation The more I think about it, the more I think @theodorton 's approach (Crow?) is correct. Why? Because at the moment WASM is just not mature enough. I.e. lacks GC, and as a result (and more critically) objects can't be shared with JS land. This means that even if you can get a full crystal compiler going, you have to rewrite every JS lib in crystal. React for example.

I think if a bunch of us help what @theodorton has done so far, it might be quicker.

I am just playing devils advocate here, and can be convinced that I am wrong of course!

[andriytyurnikov]

It looks like few blockchain projects considering WASM as their primary or secondary VM, so browsers are not only runtime for WASM.

[asterite]

But blockchains are a waste of energy and resources... let's not hurry all together to destroy this planet faster, please?

[andriytyurnikov]

@asterite not those blockchains :) Blockchains with wasm are environment friendly, and they need your support :)

[faustinoaq]

Q: Don't you need features like garbage collection and threading added to WebAssembly to make this work?

No, WebAssembly in its current state is sufficient. The .NET runtime handles its own garbage collection and threading concerns.

^ Interesting :sweat_smile:

^ https://github.com/aspnet/Blazor/wiki/FAQ#q-dont-you-need-features-like-garbage-collection-and-threading-added-to-webassembly-to-make-this-work