Bucklescript Compiler daemon mode

[OvermindDL1]

For long running build systems, such as brunch being able to recompile and serve files on the fly as they are changed, including compiling multiple files simultaneously, it would be convenient to have a '--daemon' or so mode for BSC that could take commands on stdin and give output on stdout. Right now it works like this:

1. Start up the build engine
2. The build engine noticed 3 files need to be rebuilt and calls the compile callback for the appropriate module for the file (OCaml/bucklescript in this case)
3. That module gets called 3 times rapidly and concurrently, and it calls the bsc compiler for the file it was given, this 3 instances of BSC are now running.
4. If the compile fails, say due to dependencies not being built yet then it tries to recompile the file along with its dependencies, however these dependencies can be being compiled by another bsc instance at the same time, this all happens in a fraction of a second, but it is enough to occasionally have files output over each other, corrupting them (of which a recompile fixes).

A preferred way, from the perspective from the build system integration, would be for a bsc compiler (or a wrapper there-of that calls bsc) to do this:

1. The build system plugin is initialized, which would load up the bsc-daemon thing.
2. It gives the daemon a list of files (and potentially include directories and so forth, if not specified on the command-line) to compile and load in to memory.
3. The stdout gives, say, (following ocamlmerlin's style) json of the output of the compiles, simple success messages for most, warnings and errors for others as appropriate.
4. As files are changed by the IDE then a command is sent to the daemon to recompile that file, of which the daemon can look for it on disk or the contents of the file itself could be passed directly to the daemon (saving the file lookup from disk cost if it is already loaded).
5. Just keep repeating that type of thing, sending commands to the daemon, it sends results back, it keeps everything that it can in memory for development speed.

Commands to the daemon, along with example responses could be something like:

Daemon STDIN	Daemon STDOUT (eventually, whenever it completes, it should not be synchronous if possible, you can rapid-fire compile commands for example)
{"cmd":"compile","file":"main.ml"}\n	{"cmd":"compile","file":"main.ml","result":"Successful"}\n
{"cmd":"compile","file":"main.ml","source":"let a = 42"}\n	{"cmd":"compile","file":"main.ml","result":"Successful"}\n
{"cmd":"compile","file":"main.ml","jsOutputDir":"/path/to/output/dir","source":"let a = 42"}\n	{"cmd":"compile","file":"main.ml","result":"Successful"}\n
{"cmd":"compile","file":"main.ml","jsOutputReturn":true,,"source":"let a = 42"}\n	{"cmd":"compile","file":"main.ml","result":"Successful","js":"var a = 42;"}\n
{"cmd":"compile","file":"fib.ml"}\n	{"cmd":"compile","file":"fib.ml","result":"Successful","warnings":[{"type":26,"message":"unused variable a."]}\n

Among many others such as getting dependencies, getting dependents based on the files that are currently loaded, intermediate file output directories (cmi/cmj), etc... etc...

This is based on a conversation from this thread: https://github.com/bloomberg/bucklescript/issues/561#issuecomment-249948755

[nebuta]

An interesting idea, but why don't you invoke bsc every time the file changes? Does that cause serious performance issues?

[OvermindDL1]

An interesting idea, but why don't you invoke bsc every time the file changes? Does that cause serious performance issues?

I already do, and that is how it works now, but that is what is causing the problem.

Say 4 files change at once, suddenly 4 bsc's will be spawned to handle each file, however lets say one of the files depends on one of the other files that is compiling so it is included in its compile-set as well, thus lets say one bsc is compiling file a.ml and another bsc at the same time is compiling file a.ml and b.ml since b.ml depends on a.ml, then multiple copies of cmj/cmi get written for the same file, which occasionally causes a corruption of that file (at least on Windows, not experienced that yet on linux).

Or let's say it is a freshly git cloned project, need to npm run build in my case, thus suddenly 30 bsc's are spawned, one for each file, most of them die pretty quickly because the files they depend on are not compiled yet, thus that makes a failing build. So right now I have it set up so that if a bsc compile fails then it retries compiling with not only its file but all files that this file depends on (via -bs-files), which then causes likewise stomping on other things again.

Right now I kind of work around some of these issues by compiling again a few times until it works or I am satisfied it is dying for another reason (such as the code not compiling), which it is still a nasty race condition that it does not always catch, thus causing issues like this pretty commonly:

File "vdom.ml", line 1:
Error: The files web_node.cmi and web_window.cmi
       make inconsistent assumptions over interface Web_node

Which is funny because all the cmi files were wiped before I just ran that build. Recompiling the listed files gets rid of the error

The issue I am trying to solve is having the npm brunch build system compile with bsc. How it works is this:

1. On initial run scan the filesystem for all files that it wants to watch (in my case *.ml files).
2. For each file it will open every file and load it in to memory, passing the contents of the files to the plugin that handles that file type (mine for *.ml files) by calling the 'getDependencies' function, this function should return a list of files that should cause 'this' file to be recompiled any time any of 'those' are recompiled. This is not used to calculate dependency ordering, only 'triggers' for when this file should be recompiled.
3. For each file that is wanted by one of the plugins (like mine for bucklescript) it will now call a compile function on the plugin once for each file, concurrently, thus every file is now spawning a bsc, all simultaneously, of which many will fail because their dependencies are not compiled first, so then they will retry again while trying to compile the dependencies.
4. From this point on any time a watched file (*.ml in the case for my plugin) is touched, say by saving in VIM/Emacs then brunch will immediately read in the new file data, pass the file path and file contents to my compile function (of which I then discard the already read-in file data as I have no clue how to pass it to bsc in stdin at this time), call up bsc, recompile the file, bsc writes out the JS to a temp directory I specify, I then read in the javascript file (yay more needless file system writing, bad for SSD's) then pass it back to brunch, which then combines it with everything else in-memory, optimizes and minifies it then writes a single packed and optimized (in my case named) app.js file (as well as auto-refreshing my test/dev page in the browser if I have it loaded, it can also do per-module hot-swapping too without page reloads but I rarely do that due to function bindings).

The whole process, even with the repeated compilations, can process all 15 ml files along with all dependency checking, optimizing, minimizing, etc... in the slow case of 1.3 seconds, thus speed is not the issue, rather it is the files getting corrupted is the issue.

The way most brunch plugins work, say for typescript as an example, is it calls into the typescript javascript compiler interface (which BSC does not have one, that would be nice though, but not necessary :-) ), passes the file contents to it, it compiles it and passes it back, never writing anything to the system (intermediate files being written and re-written repeatedly are very bad on SSD's). They handle dependencies by waiting for brunch to give them those dependencies and only compiling once the in-memory dependency code is compiled first, of which it then calls the passed in callback with the success/failure of the compilation of each file as it finishes them. An example:

1. The brunch compiler plugin receives a file in compile (say "b.ml", order is arbitrary and random, and concurrent so there is no real order), it passes the file contents and a callback to the compiler.
2. The compiler received the contents of "b.ml", however it depends on "a.ml" to be in-memory first, so it waits without doing anything or until a timeout happens.
3. The brunch compiler plugin receives another file in compile (say "a.ml"), so it passes the contents of it to the compiler.
4. The compiler receives the contents of "a.ml", which has no dependencies (or are already fulfilled), so it then compiles "a.ml", calls the callback it was given with the result of the compile (the compiled javascript and success/error message and sourcemap and such), but then notices that "b.ml" now has its dependencies fulfilled so it compiles its source that it had been holding on to then calls its callback with the result.

I've been trying to emulate the above steps in my plugin but I still am not getting some things correct apparently as some dependency issues still keep popping up (I don't want to upload this state to github yet until it is working better as it is currently barely better than before).

As well as this still does not fix the writing intermediate files of cmi/cmj/js files to disk, which again, is bad on the SSD here at work. ^.^

[bobzhang]

hi @OvermindDL1 , I am in the middle of something this week, I will have a look at your code this weekend

[OvermindDL1]

All good, I went ahead and uploaded my latest version of the brunch plugin anyway, still has a few corner cases that bug me that I need to figure out, but it works well enough I think. :-)

[bobzhang]

@OvermindDL1 did you try -bs-files, it will finish very very quickly for your 15 files, I think it might not be the bottle-neck, there is a large chance your JS build system may be the bottleneck. I have around 300 test files (by using facebook watchman), I never feel any delay, amost instantly

About daemon mode, I think it is a desirable feature, would make it even faster (by caching cmi files), any reference design to have a look? btw, json decoding is not cheap, we might need design a better protocol since the design goal is to make the build lightning fast.

Edit: we will also provide bsdep soon, which should be much faster than ocamldep

[OvermindDL1]

did you try -bs-files, it will finish very very quickly for your 15 files

Indeed that is what I was originally using. :-) I was having it compile all files simultaneously every time any changed, and it was indeed faster, but on initial builds or when two files changed at the same time then two compilers would be called up to compile everything and would routinely corrupt the cmi/cmj files thus spitting out errors and failed compilations.

The issue I have is that it tells me one-at-a-time which files have changed at the time that they change. Right now (if you tried my bucklescript-testing github project compilation) you will see it is pretty slow, right now that is because I call ocamldep for every compile for every file to see what they depend on and if the others have not finished compiling yet then it waits until they do finish compiling then re-runs ocamldep (in case dependencies changed at the last compile, which happens sometimes hence why I had to remove my cache for these), and that is by far the slowest part of the current compiling process (and why compiling 15 files fresh takes about 8 seconds because it calls ocamldep over and over and over in case dependencies change). If bsdep is faster than that would help by far. :-)

While editing files when you hit save the recompilation is pretty fast, though about 300ms to 800ms now because it has to call ocamldep to verify no new dependencies again that need to be compiled first before it compiles the file itself.

I have around 300 test files (by using facebook watchman), I never feel any delay, amost instantly

Yep, if I were able to get a list of files from brunch that would work, except it compiles on demand and only compiles files that are needed by the project (which it receives concurrently but not in the same call). But yep when I tested with that it did indeed was fast, just running it over all the files multiple times caused a lot of corrupted cmj/cmi files. ^.^

btw, json decoding is not cheap, we might need design a better protocol since the design goal is to make the build lightning fast.

Indeed it is not, I was just thinking how merlin does it. S-expressions are probably something I'd pick (or a low-level binary interface if that were even remotely easy to do in javascript...).

Edit: we will also provide bsdep soon, which should be much faster than ocamldep

Whoo!

[bobzhang]

hi @OvermindDL1 , I wrote some docs about the build system, would you check see if it is helpful? http://bloomberg.github.io/bucklescript/Manual.html#_build_system_support

[OvermindDL1]

would you check see if it is helpful?

Sure. :-)

What it is doing is what an early version of my bucklescript-brunch plugin was doing, problem it caused was that when brunch asked the files to be built that the equivalent of the 'build.sh' was called, say, 25 times (I'm up to 25 files), each instance compiling all files, which (at least on Windows, did not experience the corruption on my limited linux testing before I changed how it worked) caused file corruption in the intermediate files thus breaking building of some of the instances. I do still have the option in my plugin to work like that (by setting compileAllAtOnce: true) and this is the error it gave me when I built all:

Compile Issues of tea_cmd.ml :
Fatal error: exception End_of_file

Compile Issues of web.ml :
Fatal error: exception End_of_file

Compile Issues of main_form.ml :
Fatal error: exception End_of_file

Even though the output javascript did fully come out, I could not grab the javascript because my pipeline for the individual files was killed due to the compiler returning a non-successful error code. The files that it gets that error from is random on each run as well, just by hitting up->enter it gave these this time:

Compile Issues of main_todo_optimized.ml :
Fatal error: exception End_of_file

Compile Issues of tea_program.ml :
Fatal error: exception End_of_file

Compile Issues of web_node.ml :
Fatal error: exception End_of_file

If I create a variable to lock it to only allow compile if another 'complete' compile it not going then it randomly breaks on-the-fly compiling as when a file changes while a compiling happens then it will not get recompiled, not a rare case as I often hit :w<enter> then notice something that I know is wrong in another file and fix it and save again real quick before the compile completes, in that case it would not compile my new change as it saw a compile already in progress.

If instead I always have it recompile again after an existing compile finishes if another file got a recompile request then that causes full-compiles to take the amount of time of a single compile * the number of files that do not get queued while a compile is in progress, which will be artificially high but might still work having a minimum time of the time of two full compiles. I might try this method and see if I can live with it...

It is just a big race condition, always a fun thing. ^.^

[bobzhang]

@OvermindDL1 did you see the line include .depend, make will compile its dependency first which is the right way. it seems brunch does not respect build dependency before building the lib?

[OvermindDL1]

@OvermindDL1 did you see the line include .depend, make will compile its dependency first which is the right way. it seems brunch does not respect build dependency before building the lib?

No Makefile support on Windows though. ^.^

And it expects the build system to do that, that is for notifications so it can rebuild files that depend on other files, not build order, the build order is supposed to be handled by the compiler by waiting for the necessary parts before it returns the compiled information. Many compilers can compile even if the dependencies of a file are not, but they do need the files recompiled if the dependencies are recompiled. This is very common for Javascript build systems, hence why it follows this (admittedly odd) model, but it is the fastest out for it hence why it is well used in many areas. :-)

[OvermindDL1]

Oh, and that Makefile, even if I used something like it, still would not resolve the concurrency issue, which is the main issue on this, the ocaml/bucklescript compiler does not handle multiple instances of it compiling the exact same set of files at the exact same time very well. :-)

[bobzhang]

Many compilers can compile even if the dependencies of a file are not,

It's true, but that's not the case of OCaml, the build tool has to handle that.

Oh, and that Makefile, even if I used something like it, still would not resolve the concurrency issue,

It will, since every time files changed, make all will build dependencies first, it is very fast.

No Makefile support on Windows though. ^.^

This is an issue, I am thinking of writing a [https://ninja-build.org/](Ninja generator) which works well on Windows, then we have the fastest build system with a blazing fast compiler.

the ocaml/bucklescript compiler does not handle multiple instances of it compiling the exact same set of files at the exact same time very well

It is true, but most compilers would not handle this issue either. Do you do any benchmark to see if such concurrency is really needed(instead, using the sequential model)? Suppose you have a dynamic dependency graph in brunch, every time your file changed, its dependent got recompiled and update the dynamic dependency graph, I think that's how most build system works

[OvermindDL1]

Oh I am a fan of ninja, I swapped to it a few years ago for my C++ work and it cut my build time almost in half, over a half hour savings on many projects! But still, that is another dependency that I do not want users to require. Optimally a simple npm install <blah> would be all that is needed, most users only know the JS system (where-as I am quite new to it, used to do systems programming for my work, but everything is moving to the web...).

It is true, but most compilers would not handle this issue either. Do you do any benchmark to see if such concurrency is really needed(instead, using the sequential model)? Suppose you have a dynamic dependency graph in brunch, every time your file changed, its dependent got recompiled and update the dynamic dependency graph, I think that's how most build system works

I've not found how to to prevent brunch from operating concurrently, and the file list I get only one at a time with no indicator that it is 'done', it is very much built for the JS ecosystem.

I am thinking that just forcing only a single compilation at a time (a lock) might be best, with 'another' recompilation when it is done if any other files were given to me while the compilation was happening. The single compilation just have it compile 'everything' simultaneously as I do in compileAllAtOnce: true mode, probably will play with that today.

And yes, this does feel very much like trying to fit C++ into a javascript build system, it does not 'fit' very well (although files can be compiled individually there without any others having been, just not linked until all are done).

[bobzhang]

ninja can dump compilation commands IIRC, so that you will not add a build dependency to your user (only dev dependency). JavaScript can work it that way because there is no binary signature checking cross modules. and c++ solves the problem by using includes which slows down each compilation unit

[OvermindDL1]

And initial testing is promising, it is still not as fast as the initial builds usually are, 2.3s for 52 files but individual rebuilds are faster since it only compiles once for those (unless you happen to save multiple fast enough), and I've yet to be able to cause a corruption error, so it seems my lock is holding, this might work for a release then, I might just release this on to npm today. Thanks for the idea of the lock (indirectly you did ^.^)! :-)

And yep, javascript is an oddity, it is amazing it works at all (and to be honest, it doesn't, with how many exceptions I see thrown on random websites ^.^).

[bobzhang]

@OvermindDL1 I don't have to time to read your projects yet, here is a minor tip to help improve callback performance:

let rec fold_left f accu l =
  match l with
    [] -> accu
  | a::l -> fold_left f (f accu a) l

let rec fold_left f accu l =
  match l with
    [] -> accu
  | a::l -> fold_left f (f accu a [@bs]) l

The second one is significantly faster

[OvermindDL1]

I've been profiling my 'Tea' module set in Chrome and the vdom is registering as the second highest call, the only function that takes any noticable time on the profiler is patchVNodesOnElems, which is a recursive function that gets inlined properly into a while loop in javascript with all calls working as direct calls without currying. It works that way because I have to support iterating two lists of variable lengths where one or the other may not always increment either. The only Curry calls there are where it is calling a user thunk, which rightfully might actually be curried and is a very rare call.

The major cost according to the profiler is actually in the user view function callbacks, those end up generating a lot of code that looks like this:

function view(model) {
  return Tea_html.div(/* None */0, /* None */0, /* [] */0, /* :: */[
              Tea_html.span(/* None */0, /* None */0, /* :: */[
                    Vdom.style("text-weight", "bold"),
                    /* [] */0
                  ], /* :: */[
                    /* Text */Block.__(0, ["" + model]),
                    /* [] */0
                  ]),
              /* :: */[
                Tea_html.br(/* [] */0),
                /* :: */[
                  view_button("Increment", /* Increment */0),
                  /* :: */[
                    Tea_html.br(/* [] */0),
                    /* :: */[
                      view_button("Decrement", /* Decrement */1),
                      /* :: */[
                        Tea_html.br(/* [] */0),
                        /* :: */[
                          view_button("Set to 42", /* Set */[42]),
                          /* :: */[
                            Tea_html.br(/* [] */0),
                            /* :: */[
                              model !== 0 ? view_button("Reset", /* Reset */2) : Tea_html.noNode,
                              /* [] */0
                            ]
                          ]
                        ]
                      ]
                    ]
                  ]
                ]
              ]
            ]);
}

Where JS array creation and Block.__ calls take the most time (on average, this specific short example does not have many Block.__ calls so it is dominated by array generation time). Things like Tea_html.div are this in Javascript:

function div($staropt$star, $staropt$star$1, props, nodes) {
  var key = $staropt$star ? $staropt$star[0] : "";
  var unique = $staropt$star$1 ? $staropt$star$1[0] : "";
  return Vdom.fullnode("", "div", key, unique, props, nodes);
}

So that incurs two conditionals first of all (which the JIT 'should' optimize out well since they are usually always or never taken, though I am planning to test it by breaking it up into multiple function calls), and the Vdom.fullnode is:

function fullnode(namespace, tagName, key, unique, props, vdoms) {
  return /* Node */Block.__(1, [
            namespace,
            tagName,
            key,
            unique,
            props,
            vdoms
          ]);
}

So currently I am trying to think of a design to minimize the TEA-like html generation without breaking TEA-code-style compatibility (it is mostly copy-paste from elm -> ocaml right now with a few minor fixups) as the array generation time and Block.__ calls are dominating just due to their sheer number.

There were a couple often-called areas early on that I saw Curry.* being called that I put [@bs] on that did indeed help quite a bit. it is quite a useful annotation. :-)

Speaking of, ton of changes the past day, need to re-profile...

[bobzhang]

Any idea to have some structure sharing to avoid too much allocation?

[OvermindDL1]

Any idea to have some structure sharing to avoid too much allocation?

I thought of that but could not figure out a way to finagle it into the TEA-style coding, it is very specific in how things should be built and done. :-(

[bobzhang]

It seems list does too much allocation, how about using immutable array

[OvermindDL1]

It seems list does too much allocation, how about using immutable array

I was originally using [| nodes here |] syntax, but it was an odd jump for TEA-users so I reverted it back to [ nodes here ]. If only I could operate on the AST or so from my functions. ^.^

If you have any idea that allows the 'users' to use this syntax to form their views (as this is identical to how Elm does it, sans the Union/Sum type not having curry'able constructors, which bugs, but I do not want to force a ppx usage), that would be wonderful:

let viewInput task =
  header
    [ class' "header" ]
    [ h1 [] [ text "todos" ]
    ; input
        [ class' "new-todo"
        ; placeholder "What needs to be done?"
        ; autofocus true
        ; value task
        ; name "newTodo"
        ; onInput (fun str -> (UpdateField str))  (* Oh how I wish I could just do `onInput UpdateField` here... *)
        ; onEnter Add
        ]
        []
    ]

EDIT: Do note, as it stands it is about on-par with Elm in speed (slightly above, but within a margin of error), so it is not a 'big' deal, but it is one of those inefficiencies that would be trivial to solve with a bit of lisp'y macros or so. :-) (Mine is OAK-TEA).

[bobzhang]

Do you have some numbers using array? Let's see if it's worth it

[OvermindDL1]

At this point it would take a fairly major restructuring of the entirety of the VDom to test so I cannot say whether it would be or not. The iteration pattern I use in the VDom is definitely geared to lists though. If you think it might be worth it then I can try though?

[bobzhang]

if allocation is the bottleneck, I would say a better data structure is worth exploration. maybe you can create a separate module Seq to wrap around list type to see which operations you need and replace the data structure with a better one in the future

reply@reply.github.com At: 10/05/16 10:31:33" data-digest="From: reply@reply.github.com At: 10/05/16 10:31:33" style=""> From: reply@reply.github.com At: 10/05/16 10:31:33 To: bucklescript@noreply.github.com Cc: HONGBO ZHANG (BLOOMBERG/ 731 LEX), comment@noreply.github.com Subject: Re: [bloomberg/bucklescript] Bucklescript Compiler daemon mode (#811)

At this point it would take a fairly major restructuring of the entirety of the VDom to test so I cannot say whether it would be or not. The iteration pattern I use in the VDom is definitely geared to lists though. If you think it might be worth it then I can try though?
— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[OvermindDL1]

I went ahead and made a change in a branch, it changed the complicated 'todo' view test in this way:

Run#	List	Array
1	102.8ms	95.9ms
2	103.0ms	92.2ms
3	102.1ms	90.9ms

So it is noticable, but not sure how much it matters, it is still making hundreds of arrays, but not thousands now. The generated code is shorter though, but there are still a lot of Block.__ being called just because of the union types being used. Regardless, the generated code definitely looks better:

function array_viewInput(task) {
  return Vdom.arraynode("", "header", "", "", /* array */[/* RawProp */Block.__(0, [
                  "className",
                  "header"
                ])], /* array */[
              Vdom.arraynode("", "h1", "", "", /* array */[], /* array */[/* Text */Block.__(0, ["todos"])]),
              Vdom.arraynode("", "input", "", "", /* array */[
                    /* RawProp */Block.__(0, [
                        "className",
                        "new-todo"
                      ]),
                    /* RawProp */Block.__(0, [
                        "placeholder",
                        "What needs to be done?"
                      ]),
                    Tea_html.autofocus(/* true */1),
                    /* RawProp */Block.__(0, [
                        "value",
                        task
                      ]),
                    /* RawProp */Block.__(0, [
                        "name",
                        "newTodo"
                      ]),
                    Tea_html.onInput(/* None */0, function (str) {
                          return /* UpdateField */Block.__(0, [str]);
                        }),
                    onEnter(/* None */0, /* Add */1)
                  ], /* array */[])
            ]);
}

I'm just not sure that the small speed difference matters though considering the usability decrease ([||] is not an elm syntax), it is already on par with elm now (faster or slower depending on an individual case).

[bobzhang]

looks interesting, do you have lots of duplication like [ "className", "header" ] maybe we can optmize it (lift it outside)

[OvermindDL1]

That would be user generated code. The Todo application is supposed to be as identical to the Elm version as possible to make direct conversions easier and to make the test as identical as possible between them (Elm has a benchmark tool, was slightly broken and causing times to read the same for almost everything but I fixed it in my own copy (do not try running the ember/angular/react tests as those all require node to be installed on the server and I do not have node installed on the server, they just will crash on the client because of that, but elm and oak-tea tests run fine)). I keep playing with it and testing new versions so what you benchmark of mine might not always be representative of my usual tests but it is usually pretty close.

However, the [ "className", "header" ] bits come from the source code of the original function of:

let array_viewInput task =
  let open Vdom in
  arraynode "" "header" "" ""
    [| class' "header" |]
    [| arraynode "" "h1" "" "" [||] [| text "todos" |]
    ; arraynode "" "input" "" ""
        [| class' "new-todo"
        ; placeholder "What needs to be done?"
        ; autofocus true
        ; value task
        ; name "newTodo"
        ; onInput (fun str -> (UpdateField str))
        ; onEnter Add
        |]
        [||]
    |]

Or the non-array version you can see up in a prior comment on this thread. However, those things are user defined and will be application unique, and in this case that is the only version of that as well.

And a side-note, I really hate how the ocaml Array module does not have filter and such... ;-)

/me is tempted to include janestreet's core or batteries in the Tea app... wonder what their license is like...

And for comparison, if you are curious, here is the Elm version of that same function (of even the same code copy/pasted):

var _evancz$elm_todomvc$Todo$viewInput = function (task) {
    return A2(
        _elm_lang$html$Html$header,
        _elm_lang$core$Native_List.fromArray(
            [
                _elm_lang$html$Html_Attributes$class('header')
            ]),
        _elm_lang$core$Native_List.fromArray(
            [
                A2(
                _elm_lang$html$Html$h1,
                _elm_lang$core$Native_List.fromArray(
                    []),
                _elm_lang$core$Native_List.fromArray(
                    [
                        _elm_lang$html$Html$text('todos')
                    ])),
                A2(
                _elm_lang$html$Html$input,
                _elm_lang$core$Native_List.fromArray(
                    [
                        _elm_lang$html$Html_Attributes$class('new-todo'),
                        _elm_lang$html$Html_Attributes$placeholder('What needs to be done?'),
                        _elm_lang$html$Html_Attributes$autofocus(true),
                        _elm_lang$html$Html_Attributes$value(task),
                        _elm_lang$html$Html_Attributes$name('newTodo'),
                        _elm_lang$html$Html_Events$onInput(_evancz$elm_todomvc$Todo$UpdateField),
                        _evancz$elm_todomvc$Todo$onEnter(_evancz$elm_todomvc$Todo$Add)
                    ]),
                _elm_lang$core$Native_List.fromArray(
                    []))
            ]));
};

And _elm_lang$core$Native_List.fromArray is defined as:

var Nil = { ctor: '[]' };

function Cons(hd, tl)
{
    return { ctor: '::', _0: hd, _1: tl };
}

function fromArray(arr)
{
    var out = Nil;
    for (var i = arr.length; i--; )
    {
        out = Cons(arr[i], out);
    }
    return out;
}

So it ends up doing the same thing, but even slower since it has to pass over an array each time.

[bobzhang]

indeed, let's focus on basic part first, there is plenty of places for optimization later. btw, it is quite easy to optimize Array.of_list [a;b;c] into [|a;b;c|], if that helps

[OvermindDL1]

Yep, I'm trying to focus on the api to be as identical to the known one as possible so it is easy to pick up, the internals are a bit ugly in many/most places, but it works decently fast as-is. :-)

[bobzhang]

@OvermindDL1 hi, I got ninja works very well for buckle -- it takes 5s to build around 1200 targets, and almost 0 for a no-op. combined with a file watch service (either using watchman or nodejs), this will give you great dev experience. Are you interested in trying it out when it's ready? what is the other advantages of using brunch?

[OvermindDL1]

Brunch is mostly due to its default integration into the phoenix pipeline and multi-platformness without requiring anything beyond elixir and node for compiling (and even then both of which are not needed for releases as standalone servers can be generated). If you can get something that does not require any binary dependencies beyond node then I could integrate it, but requiring something like make or ninja (even though I already use both, not everyone does, and very few elixir devs do) is not useful for many phoenix devs. :-)

My bucklescript-brunch ended up having an issue in full compile mode on linux-only that caused near every cmi/cmj file to get corrupted, need to flesh that out before I release it... >.>

[bobzhang]

@OvermindDL1 ninja is tiny - 388KB bin size under Win, so it might be smaller than jquery. ninja also has a subcommand on ninja -t commands which will dump all commands so you can distribute a build script to avoid deps. I am sold on ninja : )

[OvermindDL1]

Nina is awesome yes, I absolute require it for C++ work due to the substantial time savings during compilation. :-)

I wonder if the license allows my to just redistribute it in bucklescript-brunch, or better yet, if it was distributed in binary form inside bs-platform for all platforms. ^.^

[OvermindDL1]

Found another use for such a daemon mode. Someone is wanting to create a 'build on the fly' plugin (like there are for elm and other to-js languages) where they usually stuff code as it is typed into a waiting compiler that already holds all state. The purpose of this is to be able to display the output javascript in another window in real time to make sure things are made as you need them, however they are having issues doing such a thing with bsc due to similar issues I had with it in brunch.

[bobzhang]

hi @OvermindDL1 , a working build tool will soon be available (probably the end of this month), it is really fast -- change a file and rebuild finishes in around 80 ms, so you can hook it with your file watch service.

I did think about daemon mode, to make it work across different platform is not easy, but bscserver should be easy to make it work across platform (using the standard unix module), are you interested in working on it ?

Note my plan is to keep bsc as small as possible (no deps on unix module), however it is fine to have unix deps for bscserver

[OvermindDL1]

hi @OvermindDL1 , a working build tool will soon be available (probably the end of this month), it is really fast -- change a file and rebuild finishes in around 80 ms, so you can hook it with your file watch service.

Looking forward to this, currently making my own build tool is painful, finding ppx's for example is an utter and outright pain without hard-coding paths as I seem to have to ocamlfind the package to get the path, open the package's 'META' file in that path, parse out the package descriptions, then parse out the ppxopt's, make sure to find other packages it needs, link in other files to the ppx's as they need (wow but that is a pain to get something like ppx_compare working with ppx_deriving and such, ow...), and etc... and so forth all working with bucklescript (which I have still not got working cleanly yet, again ow...). Hardcoding paths makes it easy, but also makes the project not work on other systems. ^.^

I did think about daemon mode, to make it work across different platform is not easy, but bscserver should be easy to make it work across platform (using the standard unix module), are you interested in working on it ?

I would be, would be a fascinating chance to learn more of the innards of the compiler, but my (lack of) time out of work would probably mean that it would take me far too long to work on it just due to the sheer time it would take for me to catch up to a knowledge level where I could embed the compiler like that at all (and my knowledge on things like LLVM do not really assist here beyond the textbook basics). >.>

Note my plan is to keep bsc as small as possible (no deps on unix module), however it is fine to have unix deps for bscserver

Indeed, bsc should be yeah. Honestly I'd be quite happy with some method of using a javascript version of it, even if slower, if only ppx support would not be so much hell... ^.^

[halohalospecial]

Hi, OCaml and BuckleScript newbie here :-) Can we also use bscserver to enable something like this?

In the screenshot, the project is compiled while typing (without saving the files first). Not sure if this is already possible with Merlin + BuckleScript.

Thanks!

[bobzhang]

Hi welcome, Merlin is awesome, it does all that for you.
The bscserver is mainly used to shave compilation time from 80ms to 40ms...

reply@reply.github.com At: 10/18/16 23:08:20" data-digest="From: reply@reply.github.com At: 10/18/16 23:08:20" style=""> From: reply@reply.github.com At: 10/18/16 23:08:20 To: bucklescript@noreply.github.com Cc: HONGBO ZHANG (BLOOMBERG/ 731 LEX), comment@noreply.github.com Subject: Re: [bloomberg/bucklescript] Bucklescript Compiler daemon mode (#811)

Hi, OCaml and BuckleScript newbie here :-) Can we also use bscserver to enable something like this?

In the screenshot, the project is compiled while typing (without saving the files first). Not sure if this is already possible with Merlin + BuckleScript.
Thanks!
— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[bobzhang]

can you be more specific in which part do you need ppx? ppx is probably the most ugly part of the OCaml language, it is nice to avoid them if you can..

reply@reply.github.com At: 10/18/16 17:28:13" data-digest="From: reply@reply.github.com At: 10/18/16 17:28:13" style=""> From: reply@reply.github.com At: 10/18/16 17:28:13 To: bucklescript@noreply.github.com Cc: HONGBO ZHANG (BLOOMBERG/ 731 LEX), comment@noreply.github.com Subject: Re: [bloomberg/bucklescript] Bucklescript Compiler daemon mode (#811)

hi @OvermindDL1 , a working build tool will soon be available (probably the end of this month), it is really fast -- change a file and rebuild finishes in around 80 ms, so you can hook it with your file watch service.

Looking forward to this, currently making my own build tool is painful, finding ppx's for example is an utter and outright pain without hard-coding paths as I seem to have to ocamlfind the package to get the path, open the package's 'META' file in that path, parse out the package descriptions, then parse out the ppxopt's, make sure to find other packages it needs, link in other files to the ppx's as they need (wow but that is a pain to get something like ppx_compare working with ppx_deriving and such, ow...), and etc... and so forth all working with bucklescript (which I have still not got working cleanly yet, again ow...). Hardcoding paths makes it easy, but also makes the project not work on other systems. ^.^

I did think about daemon mode, to make it work across different platform is not easy, but bscserver should be easy to make it work across platform (using the standard unix module), are you interested in working on it ?

I would be, would be a fascinating chance to learn more of the innards of the compiler, but my (lack of) time out of work would probably mean that it would take me far too long to work on it just due to the sheer time it would take for me to catch up to a knowledge level where I could embed the compiler like that at all (and my knowledge on things like LLVM do not really assist here beyond the textbook basics). >.>

Note my plan is to keep bsc as small as possible (no deps on unix module), however it is fine to have unix deps for bscserver

Indeed, bsc should be yeah. Honestly I'd be quite happy with some method of using a javascript version of it, even if slower, if only ppx support would not be so much hell... ^.^
— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[OvermindDL1]

can you be more specific in which part do you need ppx? ppx is probably the most ugly part of the OCaml language, it is nice to avoid them if you can..

I'm trying to emulate the elm syntax for my elm conversion library to simplify converting people as much as possible and part of the syntax of Elm is, given a union/variant type like:

type msg =
  | A of string
  | B of int

Then if you have a function like:

let func f = f 42

Then in elm you can do:

func B

Where in OCaml you have to do:

func (fun i -> B i)

And in Elm if you have a record like:

type model = 
  { i : int
  ; s : string
  }

Then you can access a field on the record via the name syntax, used like:

List.map .s [ { i=42; s="hi" } ]

Where .s is a function like (if this could be defined using this syntax, which it cannot, just pseudo-code:

let .s r = r.s

Where in OCaml you have to do:

List.map (fun r -> r.s) [ { i=42; s="hi" } ]

I personally hate that name syntax (I'd prefer a built-in Lens personally so you could get/set/update/whatever instead of just get, but whatever).

To ease the transition from Elm to Bucklescript I want to support PPX's such as the variant and record field PPX's from jane street that enable syntax that is pretty close to the above (once the union/record is properly decorated, which I can more easily deal with as an instruction step instead of telling people to litter (fun blah -> blah) all over their code):

func b (* Lower case instead of upper case, but close enough *)

List.map s [ { i=42; s="hi" } ] (* It lacks the <dot>, but that is a fine conversion step anyway *)

Also, another note that I do want to clarify. :-) The point of the bscserver is not for speed, it is to remove the need for file-system writing (and reading). Brunch is entirely capable of dynamically creating source, passing it to compilers, passing that output to other compilers, grabbing files remotely and caching them in memory and passing 'those' into compiler while never writing them to disk, etc... I do not always have files to work with is the problem and I do not always want files written to disk (preferably never written to disk), and writing out files to temporary locations just to call an application then read back in files from disk is painful and occasionally concurrently-crashy (yay windows not 'syncing' files to disk quickly so I have to retry on occasion!). Optimally I'd want something like bscserver to be able to be loaded, stay loaded with a stdin/stdout connection, feed commands into it, feed it every file that it will see with it never touching the disk if I tell it to, not writing cmi/cmj/etc... unless I tell it to, and wonderfully I would love if I could load ppx code and have it compile and use that as a ppx for source code that I pass into it (such a 'compile this code' call could just be something like the code string content, a list of already-loaded ppx's to use on it, etc...). And of course a way to flush out cached sources, meta information, and such would be nice though not absolutely necessary. :-)

I know that style does not fit the standard C/C++/OCaml/etc... style compilation, but the javascript world is weird and full of passing functions and calling those functions as callbacks when a task is complete. :-)

And I know that the cmj/cmi and other metadata information would have to be recreated each time, but that is what systems like brunch expect anyway, an initial compile should be a clean slate with incremental recompilations (such as during code editing) just adding on to and replacing parts of the initial compile. This would be a world of slow if C/C++ was done this way, but OCaml compiles fast enough to fit in to it, I just keep running into all sorts of file system issues since I have to keep touching the filesystem (mostly thanks to windows but I actually ran into a concurrency filesystem issue with bsc on linux too...). ^.^

Or a baked in list of ppx's (deriving and addons of deriving is all I think I really optimally need) with bucklescript would be awesome. ^.^

[bobzhang]

closed, issues moved to #864 #865