search

nuprl / gradual-typing-performance

10 stars 5 forks source link

GC time increasing? #100

Closed bennn closed 9 years ago

bennn commented 9 years ago

We currently throw away GC time results in the benchmarks. Consider re-running some to correlate GC changes to CPU time changes.

bennn commented 9 years ago

It's not increasing. GC time in our programs ranges from 0.5% (suffixtree) to 30% (quad). The worst case GC time is within a standard deviation of the untyped GC time for each benchmark.

bennn commented 9 years ago

Full results:

gregor
======

Untyped :
- Avg. Runtime 240.40 (5.31)
- Avg. GC time 3.80 (4.42)
- Avg. R-GC    236.60 (7.95)
- Avg. %-diff  1.59 (1.85)

Raw Data (cpu time . gc time) :
((244 . 4) (240 . 13) (250 . 0) (237 . 11) (240 . 3) (233 . 3) (243 . 0) (247 . 0) (234 . 4) (236 . 0))

Worst-Case :
- Avg. Runtime 239.60 (6.59)
- Avg. GC time 5.80 (3.37)
- Avg. R-GC    233.80 (6.31)
- Avg. %-diff  2.41 (1.37)

Raw Data (cpu time . gc time) :
((234 . 4) (234 . 7) (240 . 3) (234 . 3) (247 . 7) (233 . 9) (244 . 11) (234 . 4) (253 . 10) (243 . 0))

kcfa
====

Untyped :
- Avg. Runtime 19407.70 (763.61)
- Avg. GC time 1476.70 (134.42)
- Avg. R-GC    17931.00 (844.27)
- Avg. %-diff  7.63 (0.86)

Raw Data (cpu time . gc time) :
((21660 . 1276) (19524 . 1393) (19083 . 1323) (19213 . 1518) (19227 . 1459) (19103 . 1732) (19003 . 1527) (19030 . 1635) (19104 . 1369) (19130 . 1535))

Worst-Case :
- Avg. Runtime 19485.70 (665.52)
- Avg. GC time 1405.70 (140.71)
- Avg. R-GC    18080.00 (651.58)
- Avg. %-diff  7.22 (0.72)

Raw Data (cpu time . gc time) :
((19090 . 1328) (19370 . 1308) (19890 . 1233) (19874 . 1428) (19113 . 1161) (19120 . 1607) (21247 . 1567) (19010 . 1532) (19090 . 1503) (19053 . 1390))

lnm
===

Untyped :
- Avg. Runtime 215.10 (3.11)
- Avg. GC time 37.80 (4.42)
- Avg. R-GC    177.30 (5.93)
- Avg. %-diff  17.58 (2.13)

Raw Data (cpu time . gc time) :
((220 . 33) (213 . 43) (210 . 40) (213 . 36) (213 . 32) (217 . 44) (217 . 37) (214 . 44) (220 . 36) (214 . 33))

Worst-Case :
- Avg. Runtime 216.80 (2.09)
- Avg. GC time 38.60 (4.20)
- Avg. R-GC    178.20 (5.27)
- Avg. %-diff  17.81 (2.01)

Raw Data (cpu time . gc time) :
((217 . 41) (213 . 37) (217 . 34) (214 . 47) (217 . 40) (216 . 33) (220 . 39) (217 . 41) (220 . 33) (217 . 41))

mbta
====

Untyped :
- Avg. Runtime 628.90 (4.48)
- Avg. GC time 62.30 (3.69)
- Avg. R-GC    566.60 (4.45)
- Avg. %-diff  9.90 (0.56)

Raw Data (cpu time . gc time) :
((630 . 63) (634 . 66) (634 . 59) (633 . 70) (623 . 63) (633 . 65) (623 . 60) (626 . 60) (623 . 60) (630 . 57))

Worst-Case :
- Avg. Runtime 632.90 (9.51)
- Avg. GC time 62.30 (5.68)
- Avg. R-GC    570.60 (6.62)
- Avg. %-diff  9.84 (0.79)

Raw Data (cpu time . gc time) :
((640 . 60) (627 . 57) (627 . 64) (644 . 70) (654 . 70) (634 . 70) (623 . 56) (627 . 63) (627 . 57) (626 . 56))

morse-code
==========

Untyped :
- Avg. Runtime 214.80 (1.66)
- Avg. GC time 3.80 (2.48)
- Avg. R-GC    211.00 (3.03)
- Avg. %-diff  1.77 (1.15)

Raw Data (cpu time . gc time) :
((214 . 0) (217 . 0) (216 . 3) (213 . 4) (217 . 7) (216 . 7) (213 . 3) (213 . 7) (213 . 4) (216 . 3))

Worst-Case :
- Avg. Runtime 214.90 (3.11)
- Avg. GC time 5.60 (2.58)
- Avg. R-GC    209.30 (4.03)
- Avg. %-diff  2.61 (1.21)

Raw Data (cpu time . gc time) :
((213 . 10) (210 . 7) (213 . 4) (220 . 6) (213 . 0) (217 . 3) (216 . 6) (214 . 7) (213 . 6) (220 . 7))

quad
====

Untyped :
- Avg. Runtime 170.60 (4.36)
- Avg. GC time 52.30 (3.95)
- Avg. R-GC    118.30 (2.33)
- Avg. %-diff  30.63 (1.66)

Raw Data (cpu time . gc time) :
((170 . 53) (167 . 50) (170 . 50) (176 . 60) (170 . 50) (170 . 50) (170 . 50) (180 . 60) (163 . 50) (170 . 50))

Worst-Case :
- Avg. Runtime 171.10 (4.64)
- Avg. GC time 51.90 (3.30)
- Avg. R-GC    119.20 (2.52)
- Avg. %-diff  30.31 (1.27)

Raw Data (cpu time . gc time) :
((167 . 53) (170 . 50) (170 . 50) (170 . 50) (170 . 50) (167 . 50) (167 . 50) (180 . 60) (170 . 50) (180 . 56))

sieve
=====

Untyped :
- Avg. Runtime 8608.00 (19.76)
- Avg. GC time 1995.50 (226.02)
- Avg. R-GC    6612.50 (219.99)
- Avg. %-diff  23.18 (2.61)

Raw Data (cpu time . gc time) :
((8587 . 2176) (8587 . 1852) (8587 . 2276) (8637 . 2240) (8620 . 1822) (8614 . 1949) (8607 . 1769) (8584 . 1565) (8620 . 2134) (8637 . 2172))

Worst-Case :
- Avg. Runtime 8598.00 (35.03)
- Avg. GC time 2138.00 (164.22)
- Avg. R-GC    6460.00 (172.23)
- Avg. %-diff  24.87 (1.92)

Raw Data (cpu time . gc time) :
((8614 . 2301) (8614 . 2043) (8657 . 2355) (8640 . 1852) (8594 . 2000) (8607 . 1949) (8530 . 2125) (8584 . 2168) (8566 . 2271) (8574 . 2316))

snake
=====

Untyped :
- Avg. Runtime 337.00 (1.55)
- Avg. GC time 19.40 (8.38)
- Avg. R-GC    317.60 (8.78)
- Avg. %-diff  5.76 (2.49)

Raw Data (cpu time . gc time) :
((336 . 27) (336 . 17) (340 . 13) (336 . 17) (337 . 42) (340 . 16) (336 . 14) (336 . 17) (336 . 17) (337 . 14))

Worst-Case :
- Avg. Runtime 343.00 (6.16)
- Avg. GC time 21.60 (5.04)
- Avg. R-GC    321.40 (6.07)
- Avg. %-diff  6.29 (1.43)

Raw Data (cpu time . gc time) :
((343 . 23) (337 . 20) (337 . 13) (340 . 17) (337 . 19) (353 . 21) (336 . 24) (350 . 21) (350 . 25) (347 . 33))

suffixtree
==========

Untyped :
- Avg. Runtime 2587.70 (19.37)
- Avg. GC time 11.10 (12.68)
- Avg. R-GC    2576.60 (26.28)
- Avg. %-diff  0.43 (0.49)

Raw Data (cpu time . gc time) :
((2590 . 9) (2593 . 3) (2574 . 40) (2640 . 4) (2580 . 7) (2573 . 31) (2570 . 0) (2580 . 3) (2597 . 10) (2580 . 4))

Worst-Case :
- Avg. Runtime 2597.20 (16.02)
- Avg. GC time 9.60 (6.00)
- Avg. R-GC    2587.60 (17.34)
- Avg. %-diff  0.37 (0.23)

Raw Data (cpu time . gc time) :
((2620 . 13) (2620 . 3) (2577 . 7) (2580 . 3) (2620 . 7) (2590 . 7) (2597 . 7) (2586 . 9) (2586 . 17) (2596 . 23))

synth
=====

Untyped :
- Avg. Runtime 128.50 (1.50)
- Avg. GC time 25.00 (4.63)
- Avg. R-GC    103.50 (4.20)
- Avg. %-diff  19.44 (3.52)

Raw Data (cpu time . gc time) :
((127 . 19) (130 . 23) (130 . 29) (127 . 19) (127 . 29) (130 . 30) (127 . 19) (127 . 29) (130 . 23) (130 . 30))

Worst-Case :
- Avg. Runtime 129.20 (2.14)
- Avg. GC time 25.20 (2.64)
- Avg. R-GC    104.00 (3.32)
- Avg. %-diff  19.51 (2.04)

Raw Data (cpu time . gc time) :
((127 . 24) (134 . 23) (130 . 21) (130 . 27) (130 . 26) (130 . 28) (127 . 26) (127 . 25) (127 . 22) (130 . 30))

tetris
======

Untyped :
- Avg. Runtime 488.50 (2.80)
- Avg. GC time 8.20 (1.66)
- Avg. R-GC    480.30 (2.97)
- Avg. %-diff  1.68 (0.34)

Raw Data (cpu time . gc time) :
((487 . 9) (487 . 6) (490 . 7) (483 . 7) (490 . 10) (490 . 10) (494 . 7) (490 . 10) (487 . 6) (487 . 10))

Worst-Case :
- Avg. Runtime 488.80 (4.38)
- Avg. GC time 7.40 (3.20)
- Avg. R-GC    481.40 (2.84)
- Avg. %-diff  1.51 (0.63)

Raw Data (cpu time . gc time) :
((484 . 6) (487 . 6) (487 . 6) (486 . 4) (487 . 10) (487 . 7) (500 . 16) (493 . 6) (490 . 6) (487 . 7))

zordoz
======

Untyped :
- Avg. Runtime 217.90 (2.21)
- Avg. GC time 17.30 (3.23)
- Avg. R-GC    200.60 (3.83)
- Avg. %-diff  7.94 (1.47)

Raw Data (cpu time . gc time) :
((214 . 16) (220 . 23) (220 . 10) (220 . 17) (217 . 16) (220 . 17) (216 . 20) (216 . 17) (220 . 20) (216 . 17))

Worst-Case :
- Avg. Runtime 218.70 (1.62)
- Avg. GC time 16.70 (1.62)
- Avg. R-GC    202.00 (2.14)
- Avg. %-diff  7.64 (0.74)

Raw Data (cpu time . gc time) :
((217 . 20) (220 . 17) (217 . 13) (216 . 16) (220 . 17) (220 . 16) (220 . 17) (217 . 17) (220 . 17) (220 . 17))