bamboo.js

[jchavarri]

Thanks @pirapira for your work in Bamboo, the idea of morphing smart contracts is brilliant. Very inspiring project.

I wanted to contribute but my limited knowledge of OCaml led me to try this: compile the Bamboo compiler with BuckleScript to get a JS version of it.

Note: It looks like a large PR but in reality the relevant changes are just a few hundred lines, most of the changes come from automatically Menhir and ocamllex generated source files, see more in "In detail" below.

Tooling

• The main tool is bsb-native: a fork of BuckleScript that allows to compile using the native OCaml compiler, or BuckleScript, depending on a -backend flag. There are plans to merge bsb-native into the original BuckleScript at some point in the future.

Strategy / goals

• Make sure it was possible without large changes to the code –as most of the BuckleScript tooling is still quite new.
• Touch as few files / lines as possible, in order to reuse the max % of code between both implementations
• Create thin wrappers (located in the /cross-platform folder) that abstract each platform specifics. The amount of wrappers required turned out to be quite low (6 modules).

In detail

• BuckleScript was forked from OCaml 4.02.3 and hasn't been updated yet. This requires to stop using some recent APIs like Unix.sleepf
• OCaml Batteries Included is not compatible with BuckleScript as it contains C code. I wrapped all the used functions in cross-platform modules (there is a BuckleScript standard API in the works, it could be interesting for this case)
• bsb-native allows to conditionally compile using the pragma #if BSB_BACKEND = "js" then...
• BuckleScript requires a couple of configuration files: package.json and bsconfig.json. The schema of the second is here
• Stopped tracking the .merlin file as BuckleScript generates it automatically
• BuckleScript handles generators like Menhir or ocamllex by adding some configuration in bsconfig.json. I checked in the generated parser.ml and lexer.ml files to git to follow BuckleScript recommendations (see "Note" here).
• Big_int is written in C and thus not compatible with BS, used some bindings I wrote for the JS library bn.js and wrote a wrapper to unify APIs.
• Cryptokit is not compatible either, I inlined some bindings to keccak JS library
• Same for Rope, but in this case I copy/pasted the implementation to remove some native dependencies. I'm not sure what to do about this one.
• I had to move some files in order to make the different compilations pass. In particular, compileFile.ml(i)are placed in exec now
• There's a new folder exec-js that contains the main entry point for the JS compiler: bambooJs.ml

Running it

The next steps suppose that the required packages has been installed with opam. I think they get installed just by doing opam install bamboo but i'm not familiar enough with opam:

1. Run npm install from the root folder
2. Compile to JS with ./node_modules/.bin/bsb -make-world -backend js
3. Run JS with node ./lib/js/src/exec-js/bambooJs.js < src/parse/examples/026_abc.bbo (you will need node installed)
4. Compile to native with ./node_modules/.bin/bsb -make-world -backend native
5. Extra: I added a small test script in __tests__ to check that the output of both compilers when running them with all contracts in examples is the same: cd __tests__ && node compare-js-native.js (heads up: the output is always the same 🎉 )

@pirapira looking forward to your feedback :)

[pirapira]

Thank you very much for the efforts, and thanks for figuring out what it takes to do this.

I think I should keep successful Travis builds during the changes.

During the transition, I don't mind having cross-platform modules which are not really used. So, one way would be to add those pieces in several PRs. The other way is to do a beautiful big PR that just merges clean.

[pirapira]

Instead of Rope, we could just use lists of strings or something like that. Maybe I should extend rlp lib to take strings in addition to rope?

[jchavarri]

@pirapira Thanks for the review.

I am realizing now that the reason why the build has broken is that the #if BSB_BACKEND = "js" pragma can't be process by ocamlbuild (bsb has it's own preprocess step to deal with it).

I'm going to investigate if this could be fixed by adding a ppx to the existing oasis configuration. But if you have any ideas on how this could be solved, please let me know.

[jchavarri]

In particular, I'm looking at https://github.com/bsansouci/matchenv/.

[pirapira]

I have no experiences with BuckleScript. If any library is a blocker, I can try to remove it.

[jchavarri]

Thanks, I think it's more a problem with making the two build systems (bsb and oasis) work at the same time. I wonder what your thoughts would be about a potential replacement of Oasis for a package management tool that is better suited for BuckleScript (like esy)? I'm exploring the alternatives, but this exploration can take different directions depending on what your preferences are :)

[pirapira]

@jchavarri I have no emotional attachments with Oasis. I'm fine if opam install works.

[jchavarri]

@pirapira I managed to fix the tests. It's taken some time as I was exploring the addition of esy as a package manager (see https://github.com/esy/esy/issues/135) but there are some issues with the different versions of OCaml supported between the native and the JS flavours of BuckleScript, so that addition might need to be done in a new PR.

Things don't change much wrt what existed already except the whole build process is handled now by bsb-native. I also had to pin batteries to a custom fork (https://github.com/bsansouci/batteries-included) to support OCaml 4.02.3 for BuckleScript to compile.

If you are ok with all that, the last steps that I'd propose to do in this PR are:

• Move the installation steps (opam install, npm install etc) from .travis.yml to its own shell script (setup.sh for example)
• Update the README to detail the installation steps
• Remove Oasis related files and references.

Please let me know if you see any other blockers. Thanks!

[pirapira]

No blockers. Meanwhile, I'll try the branch locally.

[jchavarri]

Cool, I have no time right now to add the required files but if you want to test locally you'll need to follow the steps in travis.yml:

opam switch 4.02.3+buckle-master
eval `opam config env`
# It's important to install batteries first, so the proper version of rpc can be installed afterwards
git clone https://github.com/bsansouci/batteries-included
cd ./batteries-included && opam pin add -y batteries . && cd ..
opam install -y ocamlfind menhir rope zarith ppx_deriving rpc=1.9.52 cryptokit hex
npm install

(of course you can place batteries-included fork anywhere else)

Then you can run npm run build and all the binaries will be placed in lib/bs/native/. Or alternatively npm run build-js for the JS version.

[jchavarri]

@pirapira I think this is ready.

Some tasks that we've mentioned previously or that I noted while i went, that could be nice to revisit in future PRs:

• Run the JS compiler on the browser (not only nodeJS)
• Rope replacement with rlp
• Add __tests__/compare-js-native.js to CI
• Remove dependency on custom fork of batteries-included
• Add support for the abi option to the JS compiler