(v2.0) REPL benchmark stats

[ncave]

Some preliminary baseline stats:

------------------------------------------------
Fable 2.0.0 alpha, FCS 24.0.1
------------------------------------------------

Medium test file (10 kB):

------------------------------------------------
REPL benchmark - DotNet Core 2.1.300
------------------------------------------------
InteractiveChecker created in 1315 ms
iteration 1, FCS time: 2418 ms, Fable time: 598 ms
iteration 2, FCS time: 78 ms, Fable time: 42 ms
iteration 3, FCS time: 67 ms, Fable time: 41 ms
iteration 4, FCS time: 74 ms, Fable time: 41 ms
iteration 5, FCS time: 67 ms, Fable time: 44 ms
iteration 6, FCS time: 66 ms, Fable time: 62 ms
iteration 7, FCS time: 71 ms, Fable time: 104 ms
iteration 8, FCS time: 74 ms, Fable time: 52 ms
iteration 9, FCS time: 67 ms, Fable time: 43 ms
iteration 10, FCS time: 88 ms, Fable time: 41 ms

------------------------------------------------
REPL benchmark - Fable 1.3.7, Node.js 10.5.0
------------------------------------------------
InteractiveChecker created in 5232 ms
iteration 1, FCS time: 1866 ms, Fable time: 736 ms
iteration 2, FCS time: 1077 ms, Fable time: 402 ms
iteration 3, FCS time: 969 ms, Fable time: 348 ms
iteration 4, FCS time: 846 ms, Fable time: 328 ms
iteration 5, FCS time: 834 ms, Fable time: 322 ms
iteration 6, FCS time: 764 ms, Fable time: 304 ms
iteration 7, FCS time: 745 ms, Fable time: 293 ms
iteration 8, FCS time: 744 ms, Fable time: 286 ms
iteration 9, FCS time: 729 ms, Fable time: 289 ms
iteration 10, FCS time: 728 ms, Fable time: 287 ms

------------------------------------------------
REPL benchmark - Fable 2.0.0 alpha, Node.js 10.5.0
------------------------------------------------
InteractiveChecker created in 2840 ms
iteration 1, FCS time: 1018 ms, Fable time: 235 ms
iteration 2, FCS time: 681 ms, Fable time: 131 ms
iteration 3, FCS time: 559 ms, Fable time: 69 ms
iteration 4, FCS time: 524 ms, Fable time: 145 ms
iteration 5, FCS time: 522 ms, Fable time: 63 ms
iteration 6, FCS time: 519 ms, Fable time: 65 ms
iteration 7, FCS time: 497 ms, Fable time: 60 ms
iteration 8, FCS time: 491 ms, Fable time: 58 ms
iteration 9, FCS time: 473 ms, Fable time: 54 ms
iteration 10, FCS time: 453 ms, Fable time: 48 ms

Looks like there is some improvement over Fable 1.3.x :) The Fable times are close to dotnet, hopefully FCS javascript can be optimized further. Update: See bottom of thread for up-to-date stats.

[ncave]

Binary sizes (uncompressed):

------------------------------------------------
Fable 2.0.0 alpha, FCS 24.0.1
------------------------------------------------

------------------------------------------------
REPL benchmark - DotNet Core 2.1.300 (Release)
------------------------------------------------
bench.dll     :     14,848 bytes
Fable.JS.dll  :  1,613,824 bytes
fcs-fable.dll : 11,409,920 bytes 
------------- : ----------------
Total 3 Files : 13,038,592 bytes

------------------------------------------------
REPL benchmark - Fable 1.3.7, JavaScript
------------------------------------------------
CommonJS   :  10,784,296 bytes
ES2015     :   9,916,968 bytes
rollup     :   9,311,190 bytes
webpack -p :   4,776,623 bytes

------------------------------------------------
REPL benchmark - Fable 2.0.0 alpha, JavaScript
------------------------------------------------
CommonJS   :  14,197,490 bytes
ES2015     :  13,402,026 bytes
rollup     :  11,118,337 bytes
webpack -p :   3,679,020 bytes

Looks like the unminified size is 30% bigger than Fable 1.3.x, but that goes away after minification. Good job @alfonsogarciacaro!

[ncave]

Large test file (50 kB):

------------------------------------------------
REPL benchmark - DotNet Core 2.1.300
------------------------------------------------
InteractiveChecker created in 1374 ms
iteration 1, FCS time: 2837 ms, Fable time: 864 ms
iteration 2, FCS time: 417 ms, Fable time: 263 ms
iteration 3, FCS time: 373 ms, Fable time: 303 ms
iteration 4, FCS time: 495 ms, Fable time: 236 ms
iteration 5, FCS time: 363 ms, Fable time: 229 ms
iteration 6, FCS time: 462 ms, Fable time: 233 ms
iteration 7, FCS time: 433 ms, Fable time: 222 ms
iteration 8, FCS time: 369 ms, Fable time: 297 ms
iteration 9, FCS time: 378 ms, Fable time: 228 ms
iteration 10, FCS time: 461 ms, Fable time: 226 ms

------------------------------------------------
REPL benchmark - Fable 2.0.0 alpha, Node.js 10.5.0
------------------------------------------------
InteractiveChecker created in 2914 ms
iteration 1, FCS time: 7527 ms, Fable time: 531 ms
iteration 2, FCS time: 8852 ms, Fable time: 275 ms
iteration 3, FCS time: 8392 ms, Fable time: 257 ms
iteration 4, FCS time: 8266 ms, Fable time: 241 ms
iteration 5, FCS time: 8308 ms, Fable time: 212 ms
iteration 6, FCS time: 8123 ms, Fable time: 349 ms
iteration 7, FCS time: 8336 ms, Fable time: 209 ms
iteration 8, FCS time: 8305 ms, Fable time: 207 ms
iteration 9, FCS time: 8214 ms, Fable time: 202 ms
iteration 10, FCS time: 8207 ms, Fable time: 201 ms

Looks like the FCS javascript does not scale linearly with size. Update: See bottom of thread for up-to-date stats.

[alfonsogarciacaro]

This awesome @ncave! Thanks a lot for this data, it will make it much easier to check if future developments go in the right direction :+1: The results are very interesting and promising indeed. It's really encouraging to see the Fable part is as fast as .NET (although it may be because we're skipping the optimization phase) and I'm sure we can make the FCS part faster too.

It was expected the unminified bundle was somewhat bigger (because of name mangling) but also that the result would work better with JS minifiers :) I wonder if the improvement when creating the Interactive checker is also due to the WebAssembly bits in Long.js, as the int64 type is used a lot when reading the assemblies.

[ncave]

@alfonsogarciacaro Good news everyone! After switching the FCS lexer from char to uint16, the bench times improved from 20x to 2x (vs dotnet core) on large files. Update: See bottom of thread for up-to-date stats.

Large test file (50 kB):

------------------------------------------------
REPL benchmark - DotNet Core 2.1.300
------------------------------------------------
InteractiveChecker created in 1244 ms
iteration 1, FCS time: 2672 ms, Fable time: 786 ms
iteration 2, FCS time: 438 ms, Fable time: 213 ms
iteration 3, FCS time: 343 ms, Fable time: 212 ms
iteration 4, FCS time: 349 ms, Fable time: 211 ms
iteration 5, FCS time: 360 ms, Fable time: 213 ms
iteration 6, FCS time: 435 ms, Fable time: 217 ms
iteration 7, FCS time: 436 ms, Fable time: 211 ms
iteration 8, FCS time: 352 ms, Fable time: 214 ms
iteration 9, FCS time: 354 ms, Fable time: 217 ms
iteration 10, FCS time: 348 ms, Fable time: 222 ms

------------------------------------------------
REPL benchmark - Fable 2.0.0 alpha, Node.js 10.5.0
------------------------------------------------
InteractiveChecker created in 2885 ms
iteration 1, FCS time: 2209 ms, Fable time: 531 ms
iteration 2, FCS time: 1259 ms, Fable time: 287 ms
iteration 3, FCS time: 923 ms, Fable time: 263 ms
iteration 4, FCS time: 838 ms, Fable time: 229 ms
iteration 5, FCS time: 898 ms, Fable time: 249 ms
iteration 6, FCS time: 787 ms, Fable time: 234 ms
iteration 7, FCS time: 734 ms, Fable time: 223 ms
iteration 8, FCS time: 701 ms, Fable time: 214 ms
iteration 9, FCS time: 688 ms, Fable time: 204 ms
iteration 10, FCS time: 712 ms, Fable time: 200 ms

[ncave]

On medium size files, down from 7x to 3x (vs dotnet core): Update: See bottom of thread for up-to-date stats.

Medium test file (10 kB):

------------------------------------------------
REPL benchmark - DotNet Core 2.1.300
------------------------------------------------
InteractiveChecker created in 1251 ms
iteration 1, FCS time: 2401 ms, Fable time: 614 ms
iteration 2, FCS time: 73 ms, Fable time: 41 ms
iteration 3, FCS time: 67 ms, Fable time: 41 ms
iteration 4, FCS time: 94 ms, Fable time: 40 ms
iteration 5, FCS time: 87 ms, Fable time: 41 ms
iteration 6, FCS time: 87 ms, Fable time: 42 ms
iteration 7, FCS time: 69 ms, Fable time: 41 ms
iteration 8, FCS time: 66 ms, Fable time: 42 ms
iteration 9, FCS time: 66 ms, Fable time: 41 ms
iteration 10, FCS time: 70 ms, Fable time: 41 ms

------------------------------------------------
REPL benchmark - Fable 2.0.0 alpha, Node.js 10.5.0
------------------------------------------------
InteractiveChecker created in 2868 ms
iteration 1, FCS time: 896 ms, Fable time: 239 ms
iteration 2, FCS time: 365 ms, Fable time: 128 ms
iteration 3, FCS time: 346 ms, Fable time: 74 ms
iteration 4, FCS time: 242 ms, Fable time: 63 ms
iteration 5, FCS time: 215 ms, Fable time: 70 ms
iteration 6, FCS time: 199 ms, Fable time: 58 ms
iteration 7, FCS time: 212 ms, Fable time: 55 ms
iteration 8, FCS time: 213 ms, Fable time: 58 ms
iteration 9, FCS time: 200 ms, Fable time: 56 ms
iteration 10, FCS time: 183 ms, Fable time: 52 ms

[alfonsogarciacaro]

This is great @ncave, awesome job!!! :clap: :clap: :clap:

[MangelMaxime]

Awesome work @ncave :)

Thank you for what you are doing 👏 👏

[alfonsogarciacaro]

I'm very happy that we finally have a good example of the performance boost of using typed arrays thanks to you @ncave :) I also wonder if we should consider compiling char as JS number (and char[] as UInt16Array). IIRC the main reason to use JS string was for string enumeration. But we now have to make special conversion from string to char seq anyway to fix #1279. So we could use the opportunity to change char representation. What do you think @ncave?

[ncave]

@alfonsogarciacaro Sure, let's try it, since we can measure perf now.

[alfonsogarciacaro]

@ncave I've started some work to compile chars as numbers in this branch. I managed to make the tests pass, but I'm concerned there may be situations where string is casted to untyped IEnumerable or a generic seq<'T> and things can break because we cannot detect the conversions and when enumerating we'll get strings instead of numbers. What do you think?

[ncave]

@alfonsogarciacaro Do you have an example?

[ncave]

@alfonsogarciacaro I mean, is it different than casting char (or int or float) to an object and passing it around, where you lose type?

[alfonsogarciacaro]

A bit contrived, but something like this:

open System.Collections

let chunkTyped size (xs: 'T seq) =
    Seq.chunkBySize size xs

let chunkUntyped size (xs: IEnumerable) =
    xs |> Seq.cast<obj> |> Seq.chunkBySize size

testCase "Casting string to IEnumerable" <| fun () ->
    let xs1 = "fffff" |> chunkTyped 3
    let xs2 = "fffff" |> chunkUntyped 3
    Seq.toList xs1 |> equal [[|'f';'f';'f'|]; [|'f';'f'|]]
    Seq.toList xs2 |> equal [[|'f';'f';'f'|]; [|'f';'f'|]]

The second assert will fail in Fable dev2.0-chars (it passes in .NET F#). Although now that I think about it, it may not be something very important because we're already relying in knowing the generics at compile time for many collection methods, so casting to untyped IEnumerable is bound to cause problems in many situations. Can we live with that? Is compiling chars as numbers an improvement?

Another issue I just noticed though is when printing chars (as in printfn "%A" 'a').

[ncave]

@alfonsogarciacaro I don't know if compiling chars as numbers will be an improvement, trying to perf test the dev2.0-chars but it's failing with this: Fable/src/dotnet/Fable.Core/Fable.Core.fs(1,1): error FABLE: Classes without a primary constructor cannot be inherited: System.Attribute was there a recent change in constructors in that branch?

[alfonsogarciacaro]

Hmm, I did remove some defensive code to simplify things, but it's weird that error didn't happen before. For now, I just added a dummy Attribute type so it's possible to inherit from it and then rebased dev2.0-chars branch.

[ncave]

@alfonsogarciacaro I don't see any difference in the bench performance of dev2.0 and dev2.0-chars, so it's up to you which way you want to go, whatever works best. Possibly JS interop needs to be taken into consideration, but I'm no expert there so I don't have an opinion. If you decide to replace the chars with ints, perhaps the printfn "%A" issue can be fixed by examining the argument types and converting the chars before calling the print javascript, but I really have no preference either way.

[alfonsogarciacaro]

Perfect, thanks a lot for checking @ncave. Ok, I will just submit a WIP PR for now, and we can revisit it later. BTW, I want to start building the REPL again in Appveyor (I keep having problems when building the REPL locally on macOS). Are you just using the REPL Fake target for that or doing anything else?

[ncave]

@alfonsogarciacaro I was just building the bench for now (cd src/dotnet/Fable.JS/bench && yarn build). Just FYI, there is still an outstanding issue with the Fable 2.0-generated FCS javascript that I'm struggling with, it's not emitting any object members aside from the constructor into the F# AST. That actually might be skewing the stats above, as there is less work to process afterwards. But there shouldn't be any problem building it to CI.

[alfonsogarciacaro]

@ncave I've made some changes in the bench app, I hope you're OK with them. I separated the REPL from the bench (so it's not necessary to build the REPL every time we want to make changes in the bench) and also created two different projects for each platform (JS and .NET). These are the results in my machine, after unziping the fable-repl AppVeyor artifact to Fable.JS/out dir (you can also get it by running yarn build in Fable.JS dir), and then running sh local_test.sh in the bench dir (note I've also updated the metadata files in the repo):

------------------------------------------------
REPL benchmark - DotNet Core 2.1.200 (Release)
------------------------------------------------
iteration 1, FCS time: 11090 ms, Fable time: 5615 ms
iteration 2, FCS time: 1358 ms, Fable time: 747 ms
iteration 3, FCS time: 1693 ms, Fable time: 592 ms
iteration 4, FCS time: 1148 ms, Fable time: 487 ms
iteration 5, FCS time: 803 ms, Fable time: 739 ms
iteration 6, FCS time: 1378 ms, Fable time: 669 ms
iteration 7, FCS time: 785 ms, Fable time: 492 ms
iteration 8, FCS time: 937 ms, Fable time: 522 ms
iteration 9, FCS time: 946 ms, Fable time: 487 ms
iteration 10, FCS time: 783 ms, Fable time: 579 ms

------------------------------------------------
REPL benchmark - Fable 2.0.0 alpha, Node.js 8.9.4
------------------------------------------------
InteractiveChecker created in 3893 ms
iteration 1, FCS time: 3866 ms, Fable time: 1270 ms
iteration 2, FCS time: 2955 ms, Fable time: 768 ms
iteration 3, FCS time: 2277 ms, Fable time: 781 ms
iteration 4, FCS time: 2327 ms, Fable time: 719 ms
iteration 5, FCS time: 2017 ms, Fable time: 616 ms
iteration 6, FCS time: 2048 ms, Fable time: 624 ms
iteration 7, FCS time: 2134 ms, Fable time: 584 ms
iteration 8, FCS time: 1945 ms, Fable time: 616 ms
iteration 9, FCS time: 2038 ms, Fable time: 1009 ms
iteration 10, FCS time: 1996 ms, Fable time: 692 ms

BTW, I removed the #if FABLE_COMPILER conditionals in FableTransforms.fs and JS compilation is still working! (As expected the Fable part is doing some more work now.) My computer is slower than yours but the comparative results seem to be similar :)

We probably can announce the beta now and prepare and online version of REPL 2 :tada:

[ncave]

@alfonsogarciacaro That's fine, although my preference would be to keep the bench as a single project. Its whole purpose is it to run the same exact code on different platforms and compare results, which will be a bit harder to maintain if said code is in two different projects.

[alfonsogarciacaro]

You're right, but it was a bit difficult to keep a single project after separating the REPL (the JS version doesn't have a direct reference to Fable.JS). However, the main file is still the same for both projects (Fable.JS/bench/app.fs) so hopefully this will prevent that the code diverges. Maybe we can try to keep them like this for now and go back to a single project if we see it's not working?

[ncave]

1487 fixes the REPL skipping some bindings issue. As suspected, the real numbers are not as spectacular as before since the compiler has to do more, here are the latest more realistic numbers:

------------------------------------------------
Large test file (50 kB):
------------------------------------------------

------------------------------------------------
REPL benchmark - DotNet Core 2.1.301
------------------------------------------------
InteractiveChecker created in 1273 ms
iteration 1, FCS time: 2876 ms, Fable time: 814 ms
iteration 2, FCS time: 436 ms, Fable time: 213 ms
iteration 3, FCS time: 372 ms, Fable time: 211 ms
iteration 4, FCS time: 369 ms, Fable time: 216 ms
iteration 5, FCS time: 374 ms, Fable time: 214 ms
iteration 6, FCS time: 370 ms, Fable time: 215 ms
iteration 7, FCS time: 374 ms, Fable time: 214 ms
iteration 8, FCS time: 441 ms, Fable time: 212 ms
iteration 9, FCS time: 429 ms, Fable time: 220 ms
iteration 10, FCS time: 373 ms, Fable time: 214 ms

------------------------------------------------
REPL benchmark - Fable 2.0.0 alpha, Node.js 10.6.0
------------------------------------------------
InteractiveChecker created in 1830 ms
iteration 1, FCS time: 3434 ms, Fable time: 1490 ms
iteration 2, FCS time: 2131 ms, Fable time: 956 ms
iteration 3, FCS time: 1724 ms, Fable time: 948 ms
iteration 4, FCS time: 1521 ms, Fable time: 893 ms
iteration 5, FCS time: 1464 ms, Fable time: 831 ms
iteration 6, FCS time: 1561 ms, Fable time: 908 ms
iteration 7, FCS time: 1402 ms, Fable time: 841 ms
iteration 8, FCS time: 1341 ms, Fable time: 830 ms
iteration 9, FCS time: 1468 ms, Fable time: 894 ms
iteration 10, FCS time: 1370 ms, Fable time: 847 ms

------------------------------------------------
Medium test file (10 kB):
------------------------------------------------

------------------------------------------------
REPL benchmark - DotNet Core 2.1.301
------------------------------------------------
InteractiveChecker created in 1286 ms
iteration 1, FCS time: 2571 ms, Fable time: 654 ms
iteration 2, FCS time: 74 ms, Fable time: 43 ms
iteration 3, FCS time: 75 ms, Fable time: 43 ms
iteration 4, FCS time: 78 ms, Fable time: 46 ms
iteration 5, FCS time: 74 ms, Fable time: 43 ms
iteration 6, FCS time: 73 ms, Fable time: 42 ms
iteration 7, FCS time: 75 ms, Fable time: 42 ms
iteration 8, FCS time: 78 ms, Fable time: 42 ms
iteration 9, FCS time: 73 ms, Fable time: 42 ms
iteration 10, FCS time: 74 ms, Fable time: 49 ms

------------------------------------------------
REPL benchmark - Fable 2.0.0 alpha, Node.js 10.6.0
------------------------------------------------
InteractiveChecker created in 1849 ms
iteration 1, FCS time: 1283 ms, Fable time: 597 ms
iteration 2, FCS time: 704 ms, Fable time: 269 ms
iteration 3, FCS time: 518 ms, Fable time: 242 ms
iteration 4, FCS time: 473 ms, Fable time: 217 ms
iteration 5, FCS time: 435 ms, Fable time: 232 ms
iteration 6, FCS time: 390 ms, Fable time: 210 ms
iteration 7, FCS time: 364 ms, Fable time: 197 ms
iteration 8, FCS time: 385 ms, Fable time: 200 ms
iteration 9, FCS time: 369 ms, Fable time: 190 ms
iteration 10, FCS time: 355 ms, Fable time: 180 ms

Still, only about 4x-5x slower (vs the highly optimized .NET Core) is not too bad as a start. I don't think we have explored all the opportunities for optimization yet.

[alfonsogarciacaro]

@ncave Latest Fable 2 online REPL puts the FCS/Fable compiler in a worker so it doesn't freeze the UI. I also added an option to enable FCS optimization, logging for some performance stats (visible in the browser console, similar to the ones you created for the bench) as well as the sudoku and ray trace samples. Hopefully this makes it easier to play with the REPL to find more optimization opportunities :)

BTW, Fable 1.37 was around 10x slower than .NET, right? So we still have a performance improvement even if the times are not as close as we hoped to .NET.

[ncave]

@alfonsogarciacaro Yes, Fable 2.0 seems about twice as fast as Fable 1.3.x.

[alfonsogarciacaro]

@ncave I'm mixing here the Fable 1 stats from the first comment with Fable 2 data from this one. Does this look right?

Medium test file (10 kB):

------------------------------------------------
REPL benchmark - DotNet Core 2.1.300
------------------------------------------------
InteractiveChecker created in 1315 ms
iteration 1, FCS time: 2418 ms, Fable time: 598 ms
iteration 2, FCS time: 78 ms, Fable time: 42 ms
iteration 3, FCS time: 67 ms, Fable time: 41 ms
iteration 4, FCS time: 74 ms, Fable time: 41 ms
iteration 5, FCS time: 67 ms, Fable time: 44 ms
iteration 6, FCS time: 66 ms, Fable time: 62 ms
iteration 7, FCS time: 71 ms, Fable time: 104 ms
iteration 8, FCS time: 74 ms, Fable time: 52 ms
iteration 9, FCS time: 67 ms, Fable time: 43 ms
iteration 10, FCS time: 88 ms, Fable time: 41 ms

------------------------------------------------
REPL benchmark - Fable 1.3.7, Node.js 10.5.0
------------------------------------------------
InteractiveChecker created in 5232 ms
iteration 1, FCS time: 1866 ms, Fable time: 736 ms
iteration 2, FCS time: 1077 ms, Fable time: 402 ms
iteration 3, FCS time: 969 ms, Fable time: 348 ms
iteration 4, FCS time: 846 ms, Fable time: 328 ms
iteration 5, FCS time: 834 ms, Fable time: 322 ms
iteration 6, FCS time: 764 ms, Fable time: 304 ms
iteration 7, FCS time: 745 ms, Fable time: 293 ms
iteration 8, FCS time: 744 ms, Fable time: 286 ms
iteration 9, FCS time: 729 ms, Fable time: 289 ms
iteration 10, FCS time: 728 ms, Fable time: 287 ms

------------------------------------------------
REPL benchmark - Fable 2.0.0 alpha, Node.js 10.6.0
------------------------------------------------
InteractiveChecker created in 1849 ms
iteration 1, FCS time: 1283 ms, Fable time: 597 ms
iteration 2, FCS time: 704 ms, Fable time: 269 ms
iteration 3, FCS time: 518 ms, Fable time: 242 ms
iteration 4, FCS time: 473 ms, Fable time: 217 ms
iteration 5, FCS time: 435 ms, Fable time: 232 ms
iteration 6, FCS time: 390 ms, Fable time: 210 ms
iteration 7, FCS time: 364 ms, Fable time: 197 ms
iteration 8, FCS time: 385 ms, Fable time: 200 ms
iteration 9, FCS time: 369 ms, Fable time: 190 ms
iteration 10, FCS time: 355 ms, Fable time: 180 ms

[ncave]

@alfonsogarciacaro More or less, yes. Not sure what you're trying to do, up-to-date measurements or reposting from top of thread? REPL performance didn't change much after switching back the chars impl, which is good. We can re-measure after the properties branch settles.

[alfonsogarciacaro]

Thanks @ncave. I just wanted to have the proper comparison between Fable 1 and 2 to show in blow posts and talks :)

[ncave]

@alfonsogarciacaro To be fair, some of the Fable2 improvement on the REPL workload comes from switching the FCS-Fable lexer from char to uint16, which would have benefitted Fable1 too, so I'm not sure how useful those numbers are for comparison. Still, the REPL2 is faster, that's a fact.

[alfonsogarciacaro]

Yeah, but nobody knows that :wink: Just kidding, it'd be nice to try to run the benchmark with the uint16 arrays in Fable 1, but anyways I'll mention the benchmark contains some performance tweaks in Fable 2.

[zpodlovics]

Could you please specify what version of FCS/Fable needed to build this project? Is there any tutorial available (as it will probably require some trick to build it).

Right now it fails with:

Skipping project "/tmp/FSharp.Compiler.Service_fable/fcs/fcs-fable/fcs-fable.fsproj" because it was not found.

[ncave]

@zpodlovics If you just want to run the REPL benchmark locally:

git clone -b fable2 https://github.com/ncave/FSharp.Compiler.Service.git FSharp.Compiler.Service_fable
cd FSharp.Compiler.Service_fable
fcs\build CodeGen.Fable

cd ../Fable
build
yarn build-bench
yarn start-bench

If you by any chance run any perf analysis, please share, thanks in advance.

[zpodlovics]

@ncave Thank you! Almost there. First it missed some kind of resource file FSComp. I generated it running dotnet restore in FSharp.Compiler.Service_fable/fcs/fcs-fable/codegen

However when I try to build the bench project after lot's of compilation it fails with:

fable: Compiled ../../../../../FSharp.Compiler.Service_fable/src/fsharp/ErrorResolutionHints.fs
fable: Compiled ../../../../../FSharp.Compiler.Service_fable/src/utils/EditDistance.fs
ERROR: File /tmp/FSharp.Compiler.Service_fable/fcs/fcs-fable/codegen/pars.fs doesn't belong to parsed project (/tmp/FSharp.Compiler.Service_fable/fcs/fcs-fable/codegen/pars.fs)

Any idea? What files should I have in FSharp.Compiler.Service_fable/fcs/fcs-fable/codegen ?

[ncave]

@zpodlovics I apologize, here is the missing codegen step:

cd FSharp.Compiler.Service_fable
fcs\build CodeGen.Fable

I have updated the steps above as well.

[zpodlovics]

@ncave Thank you for your help! I was able to build the repl. I was also able to do some profiling. The profiling done with perf + nodejs --perf-basic-prof option.

1) Speedscope visualization (you have to install speedscope from https://github.com/jlfwong/speedscope)

fable2-repl-2018-08_31_1-speedscope.zip Node ran multiple threads, the first thread is not really insteresting, you can change between the threads by clicking to the up (^) /down (v) icons on the top middle part of the visualization screen (^ perf tid v). The most interesting thread is the 6.

2) SVG visualization, zoomable (by clicking one of the element) (Warning: it will take really-really long time to render an ~50MB SVG):

fable2-repl-2018-08_31_1-svg.zip

[ncave]

Up-to-date stats based on the main FCS-Fable PR branch, showcasing the performance impact from switching the LexBuffer type to uint16:

------------------------------------------------
REPL benchmark
Fable v2.0.0-beta-005, FCS v25.0.1,
.NET Core v2.1.402, Node.js v10.11.0
(best times out of several runs)
------------------------------------------------

------------------------------------------------
Medium test file (10 kB), LexBuffer<char> (Node.js):
------------------------------------------------
InteractiveChecker created in 1780 ms
iteration 1, FCS time: 1398 ms, Fable time: 551 ms
iteration 2, FCS time: 965 ms, Fable time: 299 ms
iteration 3, FCS time: 866 ms, Fable time: 214 ms
iteration 4, FCS time: 803 ms, Fable time: 212 ms
iteration 5, FCS time: 721 ms, Fable time: 187 ms
iteration 6, FCS time: 693 ms, Fable time: 173 ms
iteration 7, FCS time: 655 ms, Fable time: 176 ms
iteration 8, FCS time: 665 ms, Fable time: 158 ms
iteration 9, FCS time: 630 ms, Fable time: 159 ms
iteration 10, FCS time: 613 ms, Fable time: 154 ms
Done in 13.36s.

------------------------------------------------
Medium test file (10 kB), LexBuffer<uint16> (Node.js):
------------------------------------------------
InteractiveChecker created in 1830 ms
iteration 1, FCS time: 1166 ms, Fable time: 561 ms
iteration 2, FCS time: 692 ms, Fable time: 295 ms
iteration 3, FCS time: 515 ms, Fable time: 275 ms
iteration 4, FCS time: 483 ms, Fable time: 189 ms
iteration 5, FCS time: 404 ms, Fable time: 185 ms
iteration 6, FCS time: 406 ms, Fable time: 179 ms
iteration 7, FCS time: 347 ms, Fable time: 187 ms
iteration 8, FCS time: 366 ms, Fable time: 159 ms
iteration 9, FCS time: 347 ms, Fable time: 162 ms
iteration 10, FCS time: 335 ms, Fable time: 154 ms
Done in 10.49s.

------------------------------------------------
Medium test file (10 kB), LexBuffer<uint16> (.NET Core):
------------------------------------------------
InteractiveChecker created in 1321 ms
iteration 1, FCS time: 2269 ms, Fable time: 615 ms
iteration 2, FCS time: 77 ms, Fable time: 41 ms
iteration 3, FCS time: 75 ms, Fable time: 42 ms
iteration 4, FCS time: 89 ms, Fable time: 44 ms
iteration 5, FCS time: 71 ms, Fable time: 45 ms
iteration 6, FCS time: 87 ms, Fable time: 44 ms
iteration 7, FCS time: 77 ms, Fable time: 49 ms
iteration 8, FCS time: 74 ms, Fable time: 44 ms
iteration 9, FCS time: 83 ms, Fable time: 44 ms
iteration 10, FCS time: 75 ms, Fable time: 42 ms

------------------------------------------------
Large test file (50 kB), LexBuffer<char> (Node.js):
------------------------------------------------
InteractiveChecker created in 1822 ms
iteration 1, FCS time: 9243 ms, Fable time: 1462 ms
iteration 2, FCS time: 9145 ms, Fable time: 828 ms
iteration 3, FCS time: 8803 ms, Fable time: 754 ms
iteration 4, FCS time: 8983 ms, Fable time: 785 ms
iteration 5, FCS time: 8634 ms, Fable time: 734 ms
iteration 6, FCS time: 8622 ms, Fable time: 688 ms
iteration 7, FCS time: 8679 ms, Fable time: 754 ms
iteration 8, FCS time: 8473 ms, Fable time: 700 ms
iteration 9, FCS time: 8714 ms, Fable time: 689 ms
iteration 10, FCS time: 8798 ms, Fable time: 746 ms
Done in 99.37s.

------------------------------------------------
Large test file (50 kB), LexBuffer<uint16> (Node.js):
------------------------------------------------
InteractiveChecker created in 1826 ms
iteration 1, FCS time: 3456 ms, Fable time: 1415 ms
iteration 2, FCS time: 2082 ms, Fable time: 845 ms
iteration 3, FCS time: 1512 ms, Fable time: 761 ms
iteration 4, FCS time: 1536 ms, Fable time: 793 ms
iteration 5, FCS time: 1356 ms, Fable time: 712 ms
iteration 6, FCS time: 1295 ms, Fable time: 697 ms
iteration 7, FCS time: 1407 ms, Fable time: 769 ms
iteration 8, FCS time: 1245 ms, Fable time: 699 ms
iteration 9, FCS time: 1220 ms, Fable time: 673 ms
iteration 10, FCS time: 1366 ms, Fable time: 752 ms
Done in 27.70s.

------------------------------------------------
Large test file (50 kB), LexBuffer<uint16> (.NET Core):
------------------------------------------------
InteractiveChecker created in 1391 ms
iteration 1, FCS time: 2596 ms, Fable time: 794 ms
iteration 2, FCS time: 446 ms, Fable time: 228 ms
iteration 3, FCS time: 356 ms, Fable time: 232 ms
iteration 4, FCS time: 426 ms, Fable time: 218 ms
iteration 5, FCS time: 364 ms, Fable time: 218 ms
iteration 6, FCS time: 359 ms, Fable time: 226 ms
iteration 7, FCS time: 356 ms, Fable time: 227 ms
iteration 8, FCS time: 454 ms, Fable time: 220 ms
iteration 9, FCS time: 354 ms, Fable time: 224 ms
iteration 10, FCS time: 357 ms, Fable time: 222 ms

To recap, using LexBuffer<uint16> in FCS-Fable (Node.js) is approximately 3-4x faster than LexBuffer<char> on large files (50k), and 1.5-2x faster on medium size files (10k). (on .NET Core there is no difference between the two).