Bucklescript in browser code execution

[thangngoc89]

Hey. So we are talking about developing bucklescript in-browser code execution. I've been investigating this space for a while but didn't have a chance to put everything all together

There are several attempts already and most have limitation that I want to address:

1. https://reasonml.github.io/try . Everyone should know this one. It uses browserify for resolving all the require statements generated by bsb, bundling and executing it. Everything must be resolved at compile time. Which mean that if you want to support more libraries, you need to recompile part of the js artifacts. Also, it doesn't support multi-files project

2. codesandbox.io supports ReasonML with multi-files project. Some issues:

• Codesandbox uses something called sandpack that has a similar api to webpack. So it has the same problem as bs-loader. Sandpack's model is that with each input file you need to output at most 1 file. @-jaredly worked around this issue by packing all modules into a single file that will be compiled with bucklescript at once. The format looks like this:

module Foo = {
#1 Foo

// actual content of Foo module
}
module Bar = {
#1 Bar

// actual content of Bar module
}

OCaml compiler's magics will output the correct error path for each module. But this way, you can't have a clean error location report if you have a project with mixed re and ml files. ml/re must to converted to the same code format before compiling.

• And the same issue with try, you can't dynamically load new libraries into this sandbox

[thangngoc89]

My proposal for a working bs in-browser code execution module with be doing what ReasonML's team has been encouraged: Separating the compilation and execution state of ReasonML.

1. Compilation

I've figured this out nicely, just a little bit of poblishing and we're good to go

2. Execution

This is the hardest part that I haven't figure it out yet. Given the compiled js file from bs, we need to resolve all require statements, external packages and finally executing the code.

I'm a huge fan of Demoboard for its UX and snappiness of the UI. Unfortunately it isn't open source but the core resolver is.

Ideally, we'll rebuild demoboard with polestar, uses unpkg.com and pika.dev for importing packages.

Though I'm open to ideas and faster way to achieve this

[jchavarri]

Hey @thangngoc89 thanks for all the details.

I was thinking that running Webpack in the browser might be a good idea, but I found this article by codesandbox creator where he outlines why it's hard to do so.

I have some questions:

• Is it a possibility to run bspack on the browser to bundle all modules, using jsoo to be able to run it in the browser? I bet this is tricky, but it would remove a lot of the complexity from JS bundlers. I guess it'd make hard (impossible?) to dynamically load new libraries, as you mentioned.
• If bspack is a bad idea, and JS bundlers are the way to go, what's the minimal proof of concept that would help move the project forward? As you have compilation already figured out, would this PoC be something like taking 2 strings that would represent 2 js files and try to run them through polestar to obtain a single string / js file?

[thangngoc89]

@jchavarri bspack is for packaging all ml/re modules into a single module. This is exactly @-jaredly's solution for Codesandbox. I've outlined some pros/cons about this approach above.

But still, this would contain Javascript require statements that needs to be resolved before actual code execution.

Regardings webpack api, sandpack is built upon webpack's api. I've spent several days trying to get sandpack works as a standalone package and came to the conclusion that it's tightly coupled with codesandbox's infrastructure and isn't meant to be reused elsewhere

[jchavarri]

and came to the conclusion that it's tightly coupled with codesandbox's infrastructure and isn't meant to be reused elsewhere

@thangngoc89 What exact issues did you find? I'm just curious, as I think there are many upsides on leveraging a packager used in a service as popular as codesandbox :)

About sandpack, I found this issue on GitLab that has some interesting details about sandpack architecture: https://gitlab.com/gitlab-org/gitlab-ce/issues/47268#note_81824483.

Pasting here some info, just in case. I think smooshpack is synonym of sandpack right?

For reference this is my understanding of the different components.

• smooshpack runs in the browser in the scope of Monaco and sends the source code to the bundler which runs in the iframe used for the preview.
• bundler runs in the browser in the scope of the iframe used for the live preview. The bundler generates a list of dependencies and requests them from the packager
• packager is a service that packages up the dependencies to run in the browser. This is the difficult component to run in a self-hosted GitLab instance, and in this first iteration would hope to use the Codesandbox.io service for.

[thangngoc89]

@jchavarri all links are hard coded to codesandbox.io links. and yeah smooshpack is a synonym of sandpack.

[jchavarri]

Ah, too bad. I saw this kind of code and thought it'd be ready to work without codesandbox.io.

I wonder though, if GitLab made it work (video), I guess there must be a way to "unplug" it from codesandbox? Or maybe they are running it through codesandbox? I'd find that surprising. I might try to set up a small repo just to learn more about how it works 🤔

[thangngoc89]

@jchavarri If you can make it work, then problem solved :)

[jchavarri]

I have managed to follow the react-sandpack example shown in the codesandbox-client repo.

I also realized thanks to this comment that all the calls to sandpack-xxx.codesandbox.io are just being proxied to https://unpkg.com/smooshpack@0.0.xx/sandpack/. So once I realized that, and with the knowledge that CODESANDBOX_ENV set to development will try to load sandpack from localhost, I have managed to take requests to external servers down to 11:

I created a repo with instructions in https://github.com/jchavarri/sandpack-example in case it's helpful. Next steps would be to figure out where those 11 requests are originating (one of them at least is from the uuid package being required in the example), and / or if it'd be possible to get rid of them.

[thangngoc89]

@jchavarri that's some progress :) . And this is why I need fresh eyes. I've been in this rabbit hole for too long and couldn't see any lights

[jchavarri]

@thangngoc89 Some more info about where the requests to codesandbox are coming from.

One (babel-runtime@6.26.0) is coming from the actual codesandbox client code where it seems babel is always loaded, no matter what.

Unrelated, I saw parcel is planning to have browser support at some point: https://github.com/parcel-bundler/parcel/issues/1253. I found a very recent PR that abstracts over the filesystem so one can use it without node: https://github.com/parcel-bundler/parcel/pull/3244.