MLton / mlton

The MLton repository
http://mlton.org
Other
960 stars 127 forks source link

Some benchmark stats and a possible regression in unstable #537

Closed hummy123 closed 12 months ago

hummy123 commented 12 months ago

Hi there. I'm sorry for the lack of brevity in this message. Have a few things to mention that might be of interest (and a request), and brevity isn't my strong point.

I've been curious about Standard ML and MLton after using OCaml and F#, and I'm very impressed by it after porting a Rope (data structure) implementation to SML, which I had previously implemented in both the aforementioned languages, and measuring running time with a build generated from MLton.

Firest, here are the benchmark stats, with data sets on the left column and language/compiler on the top row:

Running the data sets 1 time: MLton (nanoseconds)
Sveltecomponent 15671
Rustcode 26003
Sephblog 50511
Automerge 93118
Running the data sets on rope 100 times: MLton (ms) OCaml (ms)
Sveltecomponent < 0 6
Rustcode 1 11
Sephblog 3 14
Automerge 8 23
Running the data sets 500 times: MLton (ms)
Sveltecomponent 2
Rustcode 6
Sephblog 19
Automerge 44

The data sets are edit traces (sequences of commands to insert and delete text at different positions in the rope), from this repository. Those data sets were recorded over the course of many hours I would guess (one of them is an academic paper) so it's all very fast. I thought they might be of interest because of the benchmark page on MLton (comparing with different SML implementations).

OCaml's performance is pretty stable (hardly changes), which I think is thanks to its GC. MLton's GC seems to slow it down at some point, but I think the slowdown in these benchmarks shows an unrealistic situation (here is the repetition code) so not something to worry about.

These were on the latest stable release (MLton 20210117), using the C back-end because the LLVM one doesn't work on my Mac M2 Mini.

To expand, the mlton -codegen llvm proj.mlb command fails with the following message MLton 20210117 raised: Fail: call to system failed with Fail: exit status 1: opt -mem2reg -O2 -o /var/folders/kw/0s7kg7914l5dcd7pzjlqlxdw0000gn/T/filetvmYzH.opt.bc /var/folders/kw/0s7kg7914l5dcd7pzjlqlxdw0000gn/T/filencatj5.as.bc

Which I think is fine/reasonable, given that arm64-darwin is only preliminarily supported at this stage. (I did get llvm codegen working through that terminal command after cloning and building one of the recent Git commits of MLton, but got a stack overflow error when running a build of the same source code that works fine in the latest stable. Maybe a failure to eliminate tail calls was introduced?)

I'm sorry for how long my message has been so far. I would be happy to switch to Standard ML and MLton for my current project (which is planning to depend on C libraries rather than be reliant on a language's ecosystem), but the one thing holding me back is the lack of functional record update because I often have to interact with records that have 6+ fields. So currently trying to get a grasp on the page on functional record updates, and will hopefully manage fine.

MatthewFluet commented 12 months ago

Thanks for the note, but I'm not sure if there is anything actionable here. You mentioned a compilation failure, but not enough details to reproduce. Perhaps open another issue with a clear description of the bug?