search

gluon-lang / gluon

A static, type inferred and embeddable language written in Rust.
https://gluon-lang.org
MIT License
3.2k stars 145 forks source link

Add some mention of performance to documentation #561

Open lilyball opened 6 years ago

lilyball commented 6 years ago

I love how comprehensive the README is for this project, but I was surprised to see zero mention at all of the performance of Gluon. I'm not expecting detailed performance measurements, but it might be nice to know what the expected performance ballpark is as compared to other embeddable languages.

Marwes commented 6 years ago

I'd definitely like to have some performance comparisons. It takes quite a bit of effort to set everything up and create fair benchmarks however which is why I haven't gone through the effort yet.

naturallymitchell commented 5 years ago

Fibonacci looks like the best practice for this, going by these resources:

With just a benchmark of Gluon and and another lang or 2 on someone's machine, we could get a good sense.

Marwes commented 5 years ago

Did a quick and dirty comparison of factorial compared to lua, it is not that great atm since vm optimization has been less of a priority than language features.

factorial/gluon         time:   [13.705 us 13.812 us 13.932 us]
                        change: [+23.875% +25.569% +27.223%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 7 outliers among 100 measurements (7.00%)
  4 (4.00%) high mild
  3 (3.00%) high severe
factorial/lua           time:   [1.4681 us 1.4778 us 1.4895 us]
                        change: [-4.4998% -1.9454% +1.1977%] (p = 0.17 > 0.05)
                        No change in performance detected.
Found 18 outliers among 100 measurements (18.00%)
  3 (3.00%) high mild
  15 (15.00%) high severe

Part of the overhead is certainly because of bounds checking, some of that could be fixed without major changes without resorting wildly unsafe code (ala C) but some would need larger changes such as a bytecode validator. The function call preparation in gluon is also a bit more expensive in gluon than it should be which should be improved.

Most egregiously, I had to modify the gluon benchmark to directly use the primitive operators #Int*. Using the normally, overload + etc implies two(!) extra function calls per operation. Without those changes gluon performs much worse still but once function inlining (which is the works, but blocked on https://github.com/salsa-rs/salsa/pull/147) that change won't be necessary.

https://github.com/Marwes/gluon/tree/lua_bench

Marwes commented 5 years ago

With https://github.com/gluon-lang/gluon/pull/764 the gluon times are roughly halved.

factorial/gluon/1       time:   [346.49 ns 348.52 ns 350.77 ns]                              
                        change: [-14.644% -13.254% -11.667%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
  5 (5.00%) high mild
  2 (2.00%) high severe
factorial/gluon/10      time:   [3.4676 us 3.5005 us 3.5388 us]                                
                        change: [-15.823% -14.227% -12.660%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 6 outliers among 100 measurements (6.00%)
  2 (2.00%) high mild
  4 (4.00%) high severe
factorial/gluon/20      time:   [7.0148 us 7.0680 us 7.1256 us]                                
                        change: [-12.142% -10.964% -9.8744%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 4 outliers among 100 measurements (4.00%)
  3 (3.00%) high mild
  1 (1.00%) high severe
factorial/lua/1         time:   [311.94 ns 315.15 ns 319.15 ns]                            
                        change: [-11.530% -10.437% -9.3352%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 11 outliers among 100 measurements (11.00%)
  1 (1.00%) low mild
  5 (5.00%) high mild
  5 (5.00%) high severe
factorial/lua/10        time:   [857.27 ns 861.57 ns 866.42 ns]                              
                        change: [-15.759% -14.308% -12.986%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 15 outliers among 100 measurements (15.00%)
  2 (2.00%) low severe
  1 (1.00%) low mild
  6 (6.00%) high mild
  6 (6.00%) high severe
factorial/lua/20        time:   [1.4435 us 1.4504 us 1.4588 us]                              
                        change: [-15.585% -14.067% -12.673%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 11 outliers among 100 measurements (11.00%)
  1 (1.00%) low severe
  2 (2.00%) low mild
  3 (3.00%) high mild
  5 (5.00%) high severe