search

getkyo / kyo

Toolkit for Scala Development
https://getkyo.io
Apache License 2.0
481 stars 38 forks source link

Add benchmarks for TypeMap vs ZEnvironment #500

Closed hearnadam closed 6 days ago

hearnadam commented 1 week ago

The results may surprise you 😄

I removed addGetRepeat at @kyri-petrou 's suggestion - though it's perhaps notable that Kyo's map performed quite poorly at it.

Allocation free tags seem to make a big difference.

[info] Benchmark                                                 Mode  Cnt         Score         Error   Units
[info] TypeMapBench.addGetMultiKyo                              thrpt    5   9020961.970 ±  653351.180   ops/s
[info] TypeMapBench.addGetMultiKyo:gc.alloc.rate                thrpt    5      2749.299 ±     222.767  MB/sec
[info] TypeMapBench.addGetMultiKyo:gc.alloc.rate.norm           thrpt    5       320.001 ±       0.003    B/op
[info] TypeMapBench.addGetMultiKyo:gc.count                     thrpt    5        63.000                counts
[info] TypeMapBench.addGetMultiKyo:gc.time                      thrpt    5        36.000                    ms
[info] TypeMapBench.addGetMultiZIO                              thrpt    5    351174.979 ±   24298.631   ops/s
[info] TypeMapBench.addGetMultiZIO:gc.alloc.rate                thrpt    5      2278.277 ±     153.343  MB/sec
[info] TypeMapBench.addGetMultiZIO:gc.alloc.rate.norm           thrpt    5      6808.027 ±       0.059    B/op
[info] TypeMapBench.addGetMultiZIO:gc.count                     thrpt    5        53.000                counts
[info] TypeMapBench.addGetMultiZIO:gc.time                      thrpt    5        35.000                    ms
[info] TypeMapBench.addGetOneKyo                                thrpt    5  13145328.268 ±  660482.350   ops/s
[info] TypeMapBench.addGetOneKyo:gc.alloc.rate                  thrpt    5      4007.848 ±     200.190  MB/sec
[info] TypeMapBench.addGetOneKyo:gc.alloc.rate.norm             thrpt    5       320.001 ±       0.002    B/op
[info] TypeMapBench.addGetOneKyo:gc.count                       thrpt    5        77.000                counts
[info] TypeMapBench.addGetOneKyo:gc.time                        thrpt    5        53.000                    ms
[info] TypeMapBench.addGetOneZIO                                thrpt    5   9614828.249 ±  425321.554   ops/s
[info] TypeMapBench.addGetOneZIO:gc.alloc.rate                  thrpt    5      8428.980 ±     354.832  MB/sec
[info] TypeMapBench.addGetOneZIO:gc.alloc.rate.norm             thrpt    5       920.001 ±       0.003    B/op
[info] TypeMapBench.addGetOneZIO:gc.count                       thrpt    5       143.000                counts
[info] TypeMapBench.addGetOneZIO:gc.time                        thrpt    5        82.000                    ms
[info] TypeMapBench.addGetRepeatBaselineKyo                     thrpt    5   4289568.348 ±  612670.596   ops/s
[info] TypeMapBench.addGetRepeatBaselineKyo:gc.alloc.rate       thrpt    5      9745.220 ±    1394.919  MB/sec
[info] TypeMapBench.addGetRepeatBaselineKyo:gc.alloc.rate.norm  thrpt    5      2384.002 ±       0.007    B/op
[info] TypeMapBench.addGetRepeatBaselineKyo:gc.count            thrpt    5       136.000                counts
[info] TypeMapBench.addGetRepeatBaselineKyo:gc.time             thrpt    5        99.000                    ms
[info] TypeMapBench.addGetRepeatBaselineZIO                     thrpt    5    397137.371 ±  292312.355   ops/s
[info] TypeMapBench.addGetRepeatBaselineZIO:gc.alloc.rate       thrpt    5      5151.952 ±    3779.662  MB/sec
[info] TypeMapBench.addGetRepeatBaselineZIO:gc.alloc.rate.norm  thrpt    5     13616.023 ±       0.042    B/op
[info] TypeMapBench.addGetRepeatBaselineZIO:gc.count            thrpt    5        61.000                counts
[info] TypeMapBench.addGetRepeatBaselineZIO:gc.time             thrpt    5        78.000                    ms
[info] TypeMapBench.addKyo                                      thrpt    5   4691753.284 ± 1707113.868   ops/s
[info] TypeMapBench.addKyo:gc.alloc.rate                        thrpt    5      6453.682 ±    2256.514  MB/sec
[info] TypeMapBench.addKyo:gc.alloc.rate.norm                   thrpt    5      1448.002 ±       0.005    B/op
[info] TypeMapBench.addKyo:gc.count                             thrpt    5        65.000                counts
[info] TypeMapBench.addKyo:gc.time                              thrpt    5       107.000                    ms
[info] TypeMapBench.addZIO                                      thrpt    5   2143050.785 ±  194019.312   ops/s
[info] TypeMapBench.addZIO:gc.alloc.rate                        thrpt    5      9541.951 ±    1060.574  MB/sec
[info] TypeMapBench.addZIO:gc.alloc.rate.norm                   thrpt    5      4688.005 ±       0.014    B/op
[info] TypeMapBench.addZIO:gc.count                             thrpt    5        93.000                counts
[info] TypeMapBench.addZIO:gc.time                              thrpt    5        76.000                    ms
[info] TypeMapBench.getKyo                                      thrpt    5  18351452.568 ±  385186.443   ops/s
[info] TypeMapBench.getKyo:gc.alloc.rate                        thrpt    5         0.010 ±       0.028  MB/sec
[info] TypeMapBench.getKyo:gc.alloc.rate.norm                   thrpt    5         0.001 ±       0.002    B/op
[info] TypeMapBench.getKyo:gc.count                             thrpt    5           ≈ 0                counts
[info] TypeMapBench.getZIO                                      thrpt    5   5991920.115 ±  240944.695   ops/s
[info] TypeMapBench.getZIO:gc.alloc.rate                        thrpt    5      8680.592 ±     340.956  MB/sec
[info] TypeMapBench.getZIO:gc.alloc.rate.norm                   thrpt    5      1520.002 ±       0.004    B/op
[info] TypeMapBench.getZIO:gc.count                             thrpt    5        90.000                counts
[info] TypeMapBench.getZIO:gc.time                              thrpt    5        60.000                    ms
[info] TypeMapBench.pruneKyo                                    thrpt    5    637665.103 ±  219479.964   ops/s
[info] TypeMapBench.pruneKyo:gc.alloc.rate                      thrpt    5       228.233 ±      77.691  MB/sec
[info] TypeMapBench.pruneKyo:gc.alloc.rate.norm                 thrpt    5       376.016 ±       0.040    B/op
[info] TypeMapBench.pruneKyo:gc.count                           thrpt    5         8.000                counts
[info] TypeMapBench.pruneKyo:gc.time                            thrpt    5         6.000                    ms
[info] TypeMapBench.pruneZIO                                    thrpt    5    517195.941 ±   17549.498   ops/s
[info] TypeMapBench.pruneZIO:gc.alloc.rate                      thrpt    5      1084.113 ±      33.922  MB/sec
[info] TypeMapBench.pruneZIO:gc.alloc.rate.norm                 thrpt    5      2200.018 ±       0.041    B/op
[info] TypeMapBench.pruneZIO:gc.count                           thrpt    5        29.000                counts
[info] TypeMapBench.pruneZIO:gc.time                            thrpt    5        17.000                    ms
[info] TypeMapBench.unionKyo                                    thrpt    5   1318901.957 ±   24143.725   ops/s
[info] TypeMapBench.unionKyo:gc.alloc.rate                      thrpt    5      1345.899 ±      23.891  MB/sec
[info] TypeMapBench.unionKyo:gc.alloc.rate.norm                 thrpt    5      1072.008 ±       0.022    B/op
[info] TypeMapBench.unionKyo:gc.count                           thrpt    5        45.000                counts
[info] TypeMapBench.unionKyo:gc.time                            thrpt    5        20.000                    ms
[info] TypeMapBench.unionZIO                                    thrpt    5   2972171.845 ±   99808.034   ops/s
[info] TypeMapBench.unionZIO:gc.alloc.rate                      thrpt    5      8000.840 ±     273.273  MB/sec
[info] TypeMapBench.unionZIO:gc.alloc.rate.norm                 thrpt    5      2824.004 ±       0.010    B/op
[info] TypeMapBench.unionZIO:gc.count                           thrpt    5       123.000                counts
[info] TypeMapBench.unionZIO:gc.time                            thrpt    5        70.000                    ms

Union appears to be a strong point for ZEnvironment.

fwbrasil commented 1 week ago

I removed addGetRepeat at @kyri-petrou 's suggestion - though it's perhaps notable that Kyo's map performed quite poorly at it.

Not sure I understood, is addGetRepeatBaseline* a different version? It'd be nice to keep it, especially if it's not performing well since it can uncover good optimization opportunities.

Allocation free tags seem to make a big difference.

It'll get even more lightweight when intersection and union types are in the tag string as well!

hearnadam commented 1 week ago

@fwbrasil you can see addGetRepeat in ZIO's codebase:

https://github.com/zio/zio/blob/series/2.x/benchmarks/src/main/scala/zio/ZEnvironmentBenchmark.scala