search

ocaml / dune

A composable build system for OCaml.
https://dune.build/
MIT License
1.6k stars 395 forks source link

support bucklescript #140

Closed jaredly closed 3 years ago

jaredly commented 7 years ago

I tried just aliasing ocamlc to bsc.exe on my PATH, but got an error about cmdliner looking for a config.h file. It would be awesome to be able to use jbuilder with my web projects!

ghost commented 7 years ago

I thought bucklescript already came with a recommended ninja-based build system? /cc @bobzhang

jaredly commented 7 years ago

Bucklescript has a build system, but it's much less flexible than jbuilder. If it's not too much work on your part, I'd really prefer to be able to use jbuilder to target web + native.

ghost commented 7 years ago

I see. I don't have the time to look at bucklescript myself, and can't promise either that rules for it would be integrated into jbuilder, as it is additional maintenance work.

What I would recommend is start by forking jbuilder and change it to use bucklescript rather than the OCaml compiler, then we could see what can be done from there. For instance what could be abstracted in jbuilder to either make it possible to use jbuilder to build bucklescript code, or make it easy to produce a specialized jbuilder for buckescript, etc...

ELLIOTTCABLE commented 6 years ago

:+1: on this. bsb, their build-system, is very young and yet horribly archaic in some ways.

If nothing else, I'm desperate for (inline_tests); but way more usefully, I want to build hybrid projects which only have small Node-facing and opt-facing modules, most of which are platform-agnostic, with only one build-system — not a Frankensteinian mish-mash of multiple build-systems cobbled together by hand for each project!

I have to admit, I'm a little confused: The Introduction says,

Dune is a build system for OCaml and Reason. It is not intended as a completely generic build system that is able to build any given project in any language.

Was the addition of Reason there a mistake? I can find no other mention of Reason or Bucklescript in the docs; only js_of_ocaml …

Anyway. I understand concerns about the maintenance burden, but I strongly feel like the OCaml/Reason community needs some collaboration / cohesiveness. (I mean, there's oasis and dune of course, whatever's going on with bsb itself, esy, bsb-native …) I really think giving such a first-class build tool first-class support for BS could really bridge that gap!

Khady commented 6 years ago

Reason the syntax is supported. Nothing special to do, it just works. Bucklescript is not reason. Bucklescript is not supported.

ghost commented 6 years ago

Personally, I'm not against supporting bucklescript in dune, but I don't want it to be some kind of competition between dune and bsb, because that would be a big waste of everybody's time.

ELLIOTTCABLE commented 6 years ago

Seems like a bit off-topic, but I'd posit that a healthy balance of collaboration and competition between open-source projects for mindshare could only benefit the end-user ¯\_(ツ)_/¯

Anyway: coming back to this, it looks like I mis-read the response to the OP: you're perhaps not willing to support a BuckleScript extension itself, but you're interested in seeing what is required over here to support such an extension, and implementing/maintaining the subsequent extension-or-whatever-system?

Because forking the entire project, and them maintaining an entire fork, porting forward upstream changes, is hella untenable obviously — but a P.O.C. to motivate otherwise-weclome upstream changes to support a small, maintainable extension, is much more palatable! Seems in-line with the Dune 1.0 spirit, too. Do I understand your suggestion correctly?

ghost commented 6 years ago

Yes, basically at this point none of us has any idea what is required to support bucklescript, so we can't really make promises. If someones spends some time and produce a working prototype, even if it is not a generic solution, then we can look at the diff and assess the complexity to decide whether it is something we are happy to support or not in the long run or not.

In any case, just the initial support is not enought: if we support bucklescript, we need at least one person who uses bucklescript regularly in the dune team, so that we can assign bucklescript issues to this person. So someone has to be happy to take this role.

rgrinberg commented 6 years ago

One issue that I'd like to see resolved in bucklescript itself before supporting it is the 4.02.3 issue. If supporting bucklescript means that we must support 4.02.3 forever, then I'm very much against the idea. In my opinion, we should wait and see that bucklescript is able to keep up with the compiler before spending time on this.

rgrinberg commented 5 years ago

@diml can we assign this to @zploskey somehow?

zploskey commented 5 years ago

I plan to start working on this over the weekend with a test project.

rgrinberg commented 5 years ago

Don't hesitate to post your progress here :)

zploskey commented 5 years ago

I'll be trying things out here: https://github.com/zploskey/hello-esy-dune-bsb

zploskey commented 5 years ago

The build.ninja file generated by BuckleScript for a simple "hello world" program may be informative. I haven't used ninja directly before but it reads somewhat similarly to a makefile.

bs_package_flags = -bs-package-name hello-esy-dune-bsb
src_root_dir = /home/zach/src/hello-esy-dune-bsb
bsc = /home/zach/src/hello-esy-dune-bsb/node_modules/bs-platform/lib/bsc.exe
bsdep = /home/zach/src/hello-esy-dune-bsb/node_modules/bs-platform/lib/bsb_helper.exe
warnings = -w -30-40+6+7+27+32..39+44+45+101+a -warn-error +a
bsc_flags = -bs-suffix -nostdlib -I '/home/zach/src/hello-esy-dune-bsb/node_modules/bs-platform/lib/ocaml' -bs-g -g -color always
ppx_flags = 
bs_package_includes = 
bs_package_dev_includes = 
namespace = 
bsb_dir_group = 0
bsc_lib_includes = -I bin
rule build_ast_and_module_sets
  command = ${bsc}  ${pp_flags} ${ppx_flags} ${warnings} ${bsc_flags} -c -o ${out} -bs-syntax-only -bs-binary-ast ${in}
  description = Building ${out}
build bin/hello.mlast : build_ast_and_module_sets $src_root_dir/bin/hello.ml
rule build_deps
  command = ${bsdep} ${namespace} -g ${bsb_dir_group} -MD ${in}
  description = Building ${out}
build bin/hello.mlast.d : build_deps bin/hello.mlast
rule build_cmj_cmi
  command = ${bsc} ${bs_package_flags} -bs-assume-no-mli -bs-no-builtin-ppx-ml -bs-no-implicit-include ${bs_package_includes} ${bsc_lib_includes} ${bsc_extra_includes} ${warnings} ${bsc_flags} -o ${out} -c  ${in} $postbuild
  depfile = ${in}.d
  description = Building ${out}
build bin/hello.cmj |  $src_root_dir/bin/hello.bs.js bin/hello.cmi : build_cmj_cmi bin/hello.mlast
  bs_package_flags = $bs_package_flags  -bs-package-output commonjs:bin
build build.ninja : phony  ||  bin/hello.mlast.d

For our purposes we will likely require using bsc in place of ocamlc and bsb-helper.exe (aliased to bsdep here) in place of ocamldep, or something like that. I'll be looking more closely at what differences there may be.

Thanks to @jaredly for explaining what is going on with the .mlast files. As we thought, these are the binary AST but with some extra data at the beginning expressing the number of dependency files to follow, then those file paths. The code in bsb that reads the AST is here:

https://github.com/BuckleScript/bucklescript/blob/3cf3133a4978df05e0b5fb351a1d2d18dce16c32/jscomp/depends/binary_ast.ml#L33-L43

zploskey commented 5 years ago

I'll have some time to work on this over the weekend today and tomorrow. If anyone more familiar with how Dune is designed could sketch out how you might go about adding the a different back-end you might save me some time. I'm still not clear whether dune will need to learn how to deal with bsconfig.json files (possibly by calling bsb as a library) or if we want to add equivalent configuration to dune files as what BuckleScript adds. The latter seems nicer but a harder transition for people migrating from bsb configs.

We'll probably need to read bsconfigs no matter what so that bsb-only dependencies are supported.

rgrinberg commented 5 years ago

I'm still not clear whether dune will need to learn how to deal with bsconfig.json files (possibly by calling bsb as a library)

I think having dune deal with bsconfig.json would be great. Building existing bucklescript projects with dune will immediately open up a large test suite for us, and dramatically reduce the entry barrier for this feature.

However, depending on bsb as a library doesn't seem so useful however. I'd expect that we'd convert the specification to something that is very similar to dune's stanzas, so it doesn't seem like reusing bsb would work.

or if we want to add equivalent configuration to dune files as what BuckleScript adds

If dune supports bucklescript, I'd expect it to be able to build existing libraries and executables using bucklescript. Of course, it's likely that we'll need a field for bucklescript specific configuration.

jaredly commented 5 years ago

Hrmmm bsb as currently constituted has some differences from dune that will probably just overcomplicate things -- and honestly the reason I'd want dune w/ bucklescript is to get dune's package semantics instead of bsb's. So let's get it working with bsc directly, and then maybe add on bsconfig.json support later.

jaredly commented 5 years ago

Like, what would even be the benefit of having dune call bsb? bsb is a build tool, dune is a build tool... it would be like having dune call jenga. I don't think it makes sense.

zploskey commented 5 years ago

I don't think we should be calling bsb, but we probably need to read bsconfigs eventually. We just need to support the same feature set when calling bsc and be able to build dependencies that use bsb and not dune as a build system. I created a bucklescript branch here and started working on adding a way to set bsc flags in a dune file. Input is definitely appreciated.

rgrinberg commented 5 years ago

dune calling out bsb is definitely ruled out. This would be impractical and wrong on many levels. However, I do think that dune would need a full understanding of bsconfig.json files.

I created a bucklescript branch here and started working on adding a way to set bsc flags in a dune file

Just curious, do we have a good understanding of the various bucklescript flags yet? Understanding these will be useful in making sure that there are no nasty surprises down the road.

zploskey commented 5 years ago

Here's the help output from bsc from latest BS (4.0.6):

$ bsc --help
Usage: bsc <options> <files>
Options are:
  -bs-super-errors  Better error message combined with other tools 
  -bs-re-out  Print compiler output in Reason syntax
  -bs-suffix  Set suffix to .bs.js
  -bs-no-implicit-include  Don't include current dir implicitly
  -bs-assume-has-mli  (internal) Assume mli always exist 
  -bs-assume-no-mli  (internal) Don't lookup whether mli exist or not
  -bs-D  Define conditional variable e.g, -D DEBUG=true
  -bs-list-conditionals  List existing conditional variables
  -bs-binary-ast  Generate binary .mli_ast and ml_ast
  -bs-syntax-only  only check syntax
  -bs-no-bin-annot  disable binary annotations (by default on)
  -bs-eval  (experimental) Set the string to be evaluated, note this flag will be conflicted with -bs-main
  -bs-g  debug mode
  -bs-sort-imports  Sort the imports by lexical order so the output will be more stable (default false)
  -bs-no-sort-imports  No sort (see -bs-sort-imports)
  -bs-package-name  set package name, useful when you want to produce npm packages
  -bs-package-map  set package map, not only set package name but also use it as a namespace
  -bs-no-version-header  Don't print version header
  -bs-package-output  set npm-output-path: [opt_module]:path, for example: 'lib/cjs', 'amdjs:lib/amdjs', 'es6:lib/es6' 
  -bs-no-warn-unimplemented-external  disable warnings on unimplmented c externals
  -bs-no-builtin-ppx-ml disable built-in ppx for ml files (internal use)
  -bs-no-builtin-ppx-mli disable built-in ppx for mli files (internal use)
  -bs-cross-module-opt enable cross module inlining(experimental), default(false)
  -bs-diagnose  More verbose output
  -bs-no-check-div-by-zero  unsafe mode, don't check div by zero and mod by zero
  -bs-noassertfalse  no code for assert false
  -bs-main  set the Main entry module in script mode, for example -bs-main Main
  -bs-I  add source dir search path in script mode
  -bs-files  Provide batch of files, the compiler will sort it before compiling
  -impl <file>  Compile <file> as a .ml file
  -intf <file>  Compile <file> as a .mli file
  - <file>  Treat <file> as a file name (even if it starts with `-')
  -absname  Show absolute filenames in error messages
  -annot  Save information in <filename>.annot
  -bin-annot  Save typedtree in <filename>.cmt
  -c  Compile only (do not link)
  -config  Print configuration values and exit
  -g  Save debugging information
  -i  Print inferred interface
  -I <dir>  Add <dir> to the list of include directories
  -color {auto|always|never}  Enable or disable colors in compiler messages
    The following settings are supported:
      auto    use heuristics to enable colors only if supported
      always  enable colors
      never   disable colors
    The default setting is 'auto', and the current heuristic
    checks that the TERM environment variable exists and is
    not empty or "dumb", and that isatty(stderr) holds.
  -intf-suffix <string>  Suffix for interface files (default: .mli)
  -keep-docs  Keep documentation strings in .cmi files
  -keep-locs  Keep locations in .cmi files
  -labels  Use commuting label mode
  -no-alias-deps  Do not record dependencies for module aliases
  -no-app-funct  Deactivate applicative functors
  -noassert  Do not compile assertion checks
  -nolabels  Ignore non-optional labels in types
  -nostdlib  Do not add default directory to the list of include directories
  -o <file>  Set output file name to <file>
  -open <module>  Opens the module <module> before typing
  -pp <command>  Pipe sources through preprocessor <command>
  -ppx <command>  Pipe abstract syntax trees through preprocessor <command>
  -principal  Check principality of type inference
  -rectypes  Allow arbitrary recursive types
  -safe-string  Make strings immutable
  -short-paths  Shorten paths in types
  -strict-sequence  Left-hand part of a sequence must have type unit
  -strict-formats  Reject invalid formats accepted by legacy implementations
     (Warning: Invalid formats may behave differently from
      previous OCaml versions, and will become always-rejected
      in future OCaml versions. You should use this flag
      to detect and fix invalid formats.)
  -unsafe  Do not compile bounds checking on array and string access
  -v  Print compiler version and location of standard library and exit
  -verbose  Print calls to external commands
  -version  Print version and exit
  -vnum  Print version number and exit
  -w <list>  Enable or disable warnings according to <list>:
        +<spec>   enable warnings in <spec>
        -<spec>   disable warnings in <spec>
        @<spec>   enable warnings in <spec> and treat them as errors
     <spec> can be:
        <num>             a single warning number
        <num1>..<num2>    a range of consecutive warning numbers
        <letter>          a predefined set
     default setting is "+a-4-6-7-9-27-29-32..39-41..42-44-45-48-50-102"
  -warn-error <list>  Enable or disable error status for warnings according
     to <list>.  See option -w for the syntax of <list>.
     Default setting is "-a"
  -warn-help  Show description of warning numbers
  -where  Print location of standard library and exit
  -color {auto|always|never}  Enable or disable colors in compiler messages
    The following settings are supported:
      auto    use heuristics to enable colors only if supported
      always  enable colors
      never   disable colors
    The default setting is 'auto', and the current heuristic
    checks that the TERM environment variable exists and is
    not empty or "dumb", and that isatty(stderr) holds.
  -nopervasives  (undocumented)
  -dsource  (undocumented)
  -dparsetree  (undocumented)
  -dtypedtree  (undocumented)
  -drawlambda  (undocumented)
  -dlambda  (undocumented)
  -help  Display this list of options
  --help  Display this list of options
andreypopp commented 5 years ago

Last time I tried using bsc directly it required bsconfig.json but that could be empty (I didn't look into why it requires though).

zploskey commented 5 years ago

It doesn't seem to need a bsconfig.json to build a simple hello world in ocaml syntax with bsc hello.ml. With Reason syntax I think there is currently not a way to pass the information about it being reason syntax directly to bsc as flags. Running bsc hello.re gives don't know what to do with hello.re. Maybe this is something we can coordinate with BuckleScript on?

zploskey commented 5 years ago

Oh I see. For Reason it applies the extra preprocessors. Well, we can do that too I suppose.

bs_package_flags = -bs-package-name hello-esy-dune-bsb
src_root_dir = /home/zach/src/hello-esy-dune-bsb
bsc = /home/zach/src/hello-esy-dune-bsb/node_modules/bs-platform/lib/bsc.exe
bsdep = /home/zach/src/hello-esy-dune-bsb/node_modules/bs-platform/lib/bsb_helper.exe
warnings = -w -30-40+6+7+27+32..39+44+45+101+a -warn-error +a
bsc_flags = -bs-suffix -nostdlib -I '/home/zach/src/hello-esy-dune-bsb/node_modules/bs-platform/lib/ocaml' -bs-g -g -color always
ppx_flags = 
bs_package_includes = 
bs_package_dev_includes = 
namespace = 
bsb_dir_group = 0
refmt = /home/zach/src/hello-esy-dune-bsb/node_modules/bs-platform/lib/refmt.exe
reason_react_jsx = 
refmt_flags = --print binary
bsc_lib_includes = -I bin
rule build_ast_and_module_sets_from_re
  command = ${bsc} -pp "${refmt} ${refmt_flags}" ${reason_react_jsx}  ${ppx_flags} ${warnings} ${bsc_flags} -c -o ${out} -bs-syntax-only -bs-binary-ast -impl ${in}
  description = Building ${out}
build bin/rehello.mlast : build_ast_and_module_sets_from_re $src_root_dir/bin/rehello.re
rule build_deps
  command = ${bsdep} ${namespace} -g ${bsb_dir_group} -MD ${in}
  description = Building ${out}
build bin/rehello.mlast.d : build_deps bin/rehello.mlast
rule build_cmj_cmi
  command = ${bsc} ${bs_package_flags} -bs-assume-no-mli -bs-no-builtin-ppx-ml -bs-no-implicit-include ${bs_package_includes} ${bsc_lib_includes} ${bsc_extra_includes} ${warnings} ${bsc_flags} -o ${out} -c  ${in} $postbuild
  depfile = ${in}.d
  description = Building ${out}
build bin/rehello.cmj |  $src_root_dir/bin/rehello.bs.js bin/rehello.cmi : build_cmj_cmi bin/rehello.mlast
  bsc_flags = $bsc_flags -bs-re-out -bs-super-errors
  bs_package_flags = $bs_package_flags  -bs-package-output commonjs:bin
build build.ninja : phony  ||  bin/rehello.mlast.d
andreypopp commented 5 years ago

Running bsc hello.re gives don't know what to do with hello.re. Maybe this is something we can coordinate with BuckleScript on?

Dune's preprocessing pipeline should deal with that as it does now for Reason compiling with ocamlopt/ocamlc.

jordwalke commented 5 years ago

This is a cool idea, and I'd be curious to see how this works. One question I have is how / if Dune with bsc would compile dependencies? All bs-platform packages currently use bsb. Bsb sort of requires that everything use bsb in order for one bsb package to work. Building with Dune+bsc seems to put us in the same place. I really do like how right now Dune+jsoo can use arbitrary dependencies regardless of how they were compiled and regardless of which build systems they use. I wonder if it would be possible to use jsoo for dependencies when you can't control which build system they use, and then use Dune+bsc for the top level project.

rgrinberg commented 5 years ago

Full support for bucklescript would include support for interpreting bsconfig.json. To make external packages work with this setup, I think we should simply install external dependencies with the full source and build them on demand. This will only make dune dependencies available through the bucklescript target, but it is acceptable to provide new features that are dune only.

I wonder if it would be possible to use jsoo for dependencies when you can't control which build system they use, and then use Dune+bsc for the top level project.

Which cmi's would the bucklescript project use? I think you still need bucklescript to generate the cmi for w/e library it plans to use.

jaredly commented 5 years ago

@jordwalke jsoo & bsc have incompatible runtime representations, so there would have to be a serialization/interop layer between them if they are to be used together :/

I think "making dune able to build all bucklescript projects (e.g. support bsconfig.json)" is a way bigger task than just "make dune able to use bsc as the compiler instead of jsoo or ocamlc". So maybe it makes sense to tackle the simpler one first?

ghost commented 5 years ago

I think "making dune able to build all bucklescript projects (e.g. support bsconfig.json)" is a way bigger task than just "make dune able to use bsc as the compiler instead of jsoo or ocamlc". So maybe it makes sense to tackle the simpler one first?

That's my impression as well. Being able to build a project using dune files with bsc seems like a very good first goal.

abate commented 5 years ago

any new about this ? the ocaml/dune#bucklescript bucklescript branch seems a bit outdated. How can I help ?

ELLIOTTCABLE commented 5 years ago

:+1: here — I have some free-time at the moment to dedicate to something BuckleScript-related. @abate, if you figure out what's necessary, and want to tag somebody in, catch me in Discord?

ghost commented 5 years ago

I don't really know how the bucklescript compilation pipeline works, but I'm assuming that it is fairly similar to OCaml.

As a result, I suggest to proceed as follow:

By looking at the diff, we should understand better how all this work and how to properly integrate this feature into dune.

abate commented 5 years ago

I played a bit with this idea and followed your plan : First I had to copy the node_modules/lib directory in my _opam directory and add copy the Makefile.config in _opam/lib/ocaml/Makefile.config

Then I linked the bsc file pointing to _opam/lib/node_modules/bs-platform/lib/bsc and on top of my ocamlc. In the end I created a hello_world.ml and dune file and run dune build hello_world.bc --verbose

Running[3]: (cd _build/default && _opam/bin/ocamlc.opt -w @a-4-29-40-41-42-44-45-48-58-59-60-40 -strict-sequence -strict-formats -short-paths -keep-locs -g -bin-annot -I .hello_world.eobjs/byte -no-alias-deps -o .hello_world.eobjs/byte/hello_world.cmo -c -impl hello_world.ml)
Rule failed to generate the following targets:
- .hello_world.eobjs/byte/hello_world.cmo

The hello_world.js file is there, but I get an error as the bsc compiler doesn't know how to generate a cmo file and dune complains. Adding a library stanza to dune yield a similar result with the command : dune build hello.cma --verbose , and the js artifact is correctly generated. I tried to add another module to test the dependency resolution, but this didn't work out of the box.

Judging from these simple experiments, I guess we need to add a new Js mode to dune to make it select the correct compiler and to stop throwing errors for cm? files that bsc is not going to generate.

But I'm not sure what should be changed. Ideally, calling dune with dune build hello.bsc.js I would expect dune to call bsc instead of ocamlc and picking up the correct makefile. I'm not sure how to use ocamldep or bsb .

Something else that bugs me, is that I didn't find a way to simply install bsc using opam. i tried creating an opam switch using the BS ocaml-variants, but then I'm not sure how to proceed to get bsc compiled and installed inside an opam switch. It would be great if bsc could be installed as an opam package.

I'll keep looking at this, if anyone has any comments or idea this would be of great help.

rgrinberg commented 5 years ago

Good work @abate and your initiative is appreciated.

To proceed with the experiment, perhaps we should try to convince dune that .cmo and .cmj files serve the same purpose. One way we can do that is by replacing cmo with cmj in dune. Another way is to add a .cmj -> .cmo copying rule when invoking the compiler from dune.

ghost commented 5 years ago

One way we can do that is by replacing cmo with cmj in dune

That's indeed what I had in mind for the experiment.

abate commented 5 years ago

I started experimenting here : https://github.com/abate/dune/commit/47ff78a4cdf936c256d96a8279191cc0d28d63b0

To set everything up you should : 

yarn add bs-platform --modules-folder $OPAM_SWITCH_PREFIX/lib/node_modules
ln -s $OPAM_SWITCH_PREFIX/lib/node_modules/bs-platform/lib/bsc $OPAM_SWITCH_PREFIX/bin/bsc
cp $OPAM_SWITCH_PREFIX/lib/ocaml/Makefile.config $OPAM_SWITCH_PREFIX/lib/node_modules/bs-platform/lib/ocaml/

I've set this up in a brand new switch ocaml-variants.4.02.3+buckle-master just to be on the safe side.

This way setting up the env variable DUNE_BSC=1, dune will use bsc instead of ocamlc. Now I've tried to attack this cmj -> cmo problem, but it's not that easy.

Maybe somebody can have a look at what I've done and tell me if there is an easier way of doing it ?

rgrinberg commented 5 years ago

Seems like what you're trying to do is to add proper support for bucklescript in dune. That's definitely the right way to go eventually, but I think that initially what we had in mind was far simpler, just sed s/cmo/cmj/g and then see what breaks.

abate commented 5 years ago

I tried, but the very nature of dune, doesn't allow for this kind of monkey-patching. There are many hidden dependencies, and in the end it was just easier for me to hack in the code and to try to figure out how to untangle this bundle. But it seems I'm missing something in particular regarding extensions (I can't make bs.js to work), and I'm not sure if adding a new kind and just say ocamlc == bsc is the right way to go . Maybe poeple with a bit more expertise than me in the dune code base can have a quick look and point me in the right direction ?

rgrinberg commented 5 years ago

There's also a bit of a complication with preprocessing. I believe that we'll need some sort of cross compilation here as I'm not sure if we can make our preprocessor work in bucklescript. To hack around this, we might need to build bucklescript dune with the installed version. This will make sure the native pp is used

ghost commented 5 years ago

@rgrinberg since bucklescript seems to produce a different set of artifacts, I was thinking that we would store them in a separate directory, for instance in .<lib>.objs/bs. Although, that would only work well if the cmi files are compatible between ocamlc and bsc.

jaredly commented 5 years ago

the .cmi files are compatible, so that should work :)

ghost commented 5 years ago

Great :)

Lupus commented 5 years ago

As of bsconfig.json, @andreypopp isn't the game plan to migrate that to esy's package.json and have esy manage the dependencies, leaving only actual building phase to dune?

Lupus commented 5 years ago

Has there been any progress on this? We're thinking of building some sort of abstraction library (in ReasonML) that is supposed to work in both BuckleScript and OCaml native(esy,dune) and so far no clues on how to build that from same codebase.

Interestingly, BuckleScript has an issue to support limited subset of dune files.

@bobzhang @chenglou do you guys have any vision to share on what should be the best way to build cross-platform OCaml/Reason projects targeting BuckleScript on the JS side?

rgrinberg commented 5 years ago

No progress so far. Nobody in the current dune team is qualified to maintain such a project and nobody from the community has stepped up.

ghost commented 5 years ago

Supporting dune files in BuckleScript is a really interesting idea!

FTR, the original goal of Jbuilder was to provide a second implementation of a tool that read jbuild files and interpret them to build OCaml projects. The first implementation was Jenga which wasn't suitable for the opensource world, and in particular the opam world. It succeeded because the jbuild language, which has now evolved into the dune language sits just at the right place: it's simple and high-level enough to make it doable to have several implementations of it, but it is complete enough to describe the majority of projects.

jchavarri commented 4 years ago

I am also looking into how feasible it'd be to run bsc from Dune, based on previous work and recommendations shared in this thread.

First thing I tried is to create a project setup with esy that is reproducible, and doesn't involve manually overwriting binaries like ocamlc, so that hopefully more people from the community can experiment or contribute with less friction: https://github.com/jchavarri/dune-bs.

Running esy from that repo will download both a slightly modified version of Dune that calls bsc (based on @abate fork), and latest version of bsc (7.0.1).

Then esy step1 would reproduce what @diml suggested in a comment above, just build one file. I also replaced appearances of cmo with cmj as @rgrinberg suggested.

This first step is not working yet though. While bsc is able to generate the cmj files (I had to remove flags like -g and -opaque that bsc doesn't support), one thing that doesn't seem to work is the generation of bs.js files.

Something I noticed is that bsc is called without -impl flag when called from bsb/ ninja, while Dune adds this flag. I changed Dune so -impl is not added, and then I can see the generated JavaScript output on the console, but how can Dune know that this output is what should go into the resulting bs.js file?

The diff with master Dune branch is pretty tiny, and can be seen in https://github.com/ocaml/dune/compare/master...jchavarri:dune-bs.

jchavarri commented 4 years ago

I have updated the dune-bs project and related dune branch with a couple of improvements:

Details

The missing part to generate bs.js files was adding more flags to the bsc command:

How to try it

git clone https://github.com/jchavarri/dune-bs 
esy
esy step1

You should see Hello.bs.js and other 2 .bs.js being generated in bin.

The next step I would like to tackle is the don't know what to do with bin/.Hello.eobjs/byte/Add.cmj errors. It seems that Dune calls ocamlc.opt with -o bin/Hello.bc Add.cmj Multiply.cmj Hello.cmj but of course ocamlc doesn't know what to do with this cmj files.

bsc already generates bs.js files for every module, so there is no need to "resolve" this top level target rule. I'm not sure how to disable this rule for the BuckleScript case?

This would be the last step before having a somehow successful build (even if quite hacky for now).

jchavarri commented 4 years ago

I also tried to build files written with Reason syntax, but got an error OCaml and preprocessor have incompatible versions. dune-bs project requires 4.06.1, but this is not enough, the problem is that BuckleScript fork is not compatible with trunk version that Dune uses to refmt .re files. I guess this could be solved by shipping a prebuilt version of refmt that is compiled with BuckleScript fork of OCaml 4.06.1000+BS. All other BuckleScript ppxs are forced to ship precompiled binaries, so that wouldn't be new. cc @jordwalke @ulrikstrid as we've discussed about the "two OCaml compilers problem" in Discord before, maybe there are better ways than using a precompiled binary for refmt?

andreypopp commented 4 years ago

@jchavarri my understanding is that dune uses whatever is in $PATH, so if you put the correct refmt in $PATH then it should use it. Assuming bs fork of the compiler has the same Parsetree as regular compiler of the same version bs fork is based of (4.06.x) then refmt from opam should work.