Benchmarks

[rtfeldman]

Someone on Hacker News brought up benchmarks. It would be interesting to see how seamless-immutable compares to other immutable libraries in js-hashtrie-benchmark (which seems like a fair comparison given API similarities, even though it does not happen to use a hashtrie implementation under the hood), particularly once Prod Mode exists.

[crudh]

I created seamless versions of all the tests (except the sum test that uses reduce on an object, not sure what an eqiuvalent would be in seamless) and put it here. Please let me know if I can improve any of them.

The Put all and Remove all tests puts/removes all properties one by one. So I did that with the seamless implementations too. But I also created Batch-versions of them that calls merge and without once with an array instead. Seems like what you would do in a real application.

And it would be interesting to see the tests with a Prod Mode.

These are the results:

Get nth
hashtrie(10)                  :     10988624.32 +/- 0.89% op/s
hamt(10)                      :     11662756.60 +/- 0.92% op/s
hamt_plus(10)                 :     10689406.32 +/- 0.89% op/s
persistent-hash-trie(10)      :      9868732.28 +/- 0.97% op/s
mori hash_map(10)             :      2337573.70 +/- 0.85% op/s
immutable(10)                 :      9873107.21 +/- 0.70% op/s
seamlessImmutable(10)         :     11640944.63 +/- 0.59% op/s

put nth
hashtrie(10)                  :      1851101.94 +/- 0.52% op/s
hamt(10)                      :      1396350.78 +/- 0.75% op/s
hamt_plus(10)                 :      1359048.46 +/- 0.82% op/s
persistent-hash-trie(10)      :       333296.20 +/- 0.68% op/s
mori hash_map(10)             :      2069530.07 +/- 0.82% op/s
immutable(10)                 :      1987496.81 +/- 0.57% op/s
seamlessImmutable(10)         :       103734.16 +/- 0.63% op/s

Put All
hashtrie(10)                  :       298151.58 +/- 0.58% op/s
hamt(10)                      :       245611.96 +/- 0.59% op/s
hamt_plus(10)                 :       228329.11 +/- 0.56% op/s
persistent-hash-trie(10)      :        46293.96 +/- 0.79% op/s
mori hash_map(10)             :        98024.61 +/- 0.58% op/s
immutable(10)                 :       164094.88 +/- 0.57% op/s
seamlessImmutable(10)         :         9432.17 +/- 0.56% op/s
seamlessImmutableBatch(10)    :        50610.03 +/- 0.65% op/s

Put All (transient)
hamt(10)                      :       187368.43 +/- 0.66% op/s
hamt_plus(10)                 :       367960.47 +/- 0.66% op/s
mori hash_map(10)             :       130079.97 +/- 0.70% op/s
immutable(10)                 :       220634.84 +/- 0.60% op/s
seamlessImmutable(10)         :        59298.56 +/- 0.67% op/s

remove nth
hashtrie(10)                  :      2192759.16 +/- 0.63% op/s
hamt(10)                      :      2022976.44 +/- 0.60% op/s
hamt_plus(10)                 :      1557230.87 +/- 0.71% op/s
persistent-hash-trie(10)      :       171527.96 +/- 0.79% op/s
mori hash_map(10)             :      1402983.09 +/- 0.67% op/s
immutable(10)                 :      2168393.04 +/- 0.66% op/s
seamlessImmutable(10)         :        70445.80 +/- 0.72% op/s

Remove All
hashtrie(10)                  :       281865.17 +/- 0.69% op/s
hamt(10)                      :       245017.59 +/- 0.78% op/s
hamt_plus(10)                 :       190793.16 +/- 0.66% op/s
persistent-hash-trie(10)      :        29254.94 +/- 0.73% op/s
mori hash_map(10)             :       184961.04 +/- 0.63% op/s
immutable(10)                 :       268382.34 +/- 0.78% op/s
seamlessImmutable(10)         :         7364.39 +/- 1.14% op/s
seamlessImmutableBatch(10)    :       115008.51 +/- 1.26% op/s

Remove All (transient)
hamt(10)                      :       205820.55 +/- 0.57% op/s
hamt_plus(10)                 :       334430.04 +/- 0.58% op/s
mori hash_map(10)             :        97248.60 +/- 0.94% op/s
immutable(10)                 :       533887.03 +/- 0.63% op/s
seamlessImmutable(10)         :        79780.72 +/- 0.53% op/s

Count
hashtrie(10)                  :       471568.80 +/- 0.46% op/s
hamt(10)                      :     14029433.73 +/- 0.47% op/s
hamt_plus(10)                 :     13548777.96 +/- 0.62% op/s
persistent-hash-trie(10)      :       186148.73 +/- 0.76% op/s
mori hash_map(10)             :     75418087.81 +/- 0.70% op/s
immutable(10)                 :     78292673.71 +/- 1.07% op/s
seamlessImmutable(10)         :      1038235.14 +/- 0.62% op/s

Sum
hashtrie(10)                  :       459645.40 +/- 0.55% op/s
hamt(10)                      :      9921932.32 +/- 0.43% op/s
hamt_plus(10)                 :      9604783.60 +/- 0.67% op/s
persistent-hash-trie(10)      :       279183.30 +/- 0.69% op/s
mori hash_map(10)             :      1263520.34 +/- 0.54% op/s
immutable(10)                 :      2336308.25 +/- 0.71% op/s

Keys
hashtrie(10)                  :       275352.68 +/- 0.60% op/s
hamt(10)                      :      4441970.69 +/- 0.83% op/s
hamt_plus(10)                 :      5865476.21 +/- 0.60% op/s
persistent-hash-trie(10)      :       217769.70 +/- 0.53% op/s
mori hash_map(10)             :       451105.71 +/- 0.69% op/s
immutable(10)                 :            0.00 +/- 0.00% op/s
seamlessImmutable(10)         :      1032501.82 +/- 0.57% op/s

[crudh]

I commented out everything in makeImmutable except for addImmutabilityTag and ran the tests again. Seems like a 50% increase for most tests, and up to 100% for some.

Get nth
seamlessImmutable(10)         :     13306890.75 +/- 0.81% op/s

put nth
seamlessImmutable(10)         :       156080.34 +/- 0.87% op/s

Put All
seamlessImmutable(10)         :        14119.24 +/- 1.01% op/s
seamlessImmutableBatch(10)    :        60817.56 +/- 0.91% op/s

Put All (transient)
seamlessImmutable(10)         :        79070.08 +/- 1.06% op/s

remove nth
seamlessImmutable(10)         :        99218.78 +/- 1.04% op/s

Remove All
seamlessImmutable(10)         :        10465.64 +/- 1.20% op/s
seamlessImmutableBatch(10)    :       157722.65 +/- 1.31% op/s

Remove All (transient)
seamlessImmutable(10)         :        97212.40 +/- 1.20% op/s

Count
seamlessImmutable(10)         :      1849054.44 +/- 1.25% op/s

Keys
seamlessImmutable(10)         :      2132544.74 +/- 0.52% op/s

[rtfeldman]

Wow, this is awesome! Thanks so much for doing this. Makes it pretty clear that prodMode would make a big performance difference.

[wmertens]

@crudh could you put that in a PR for that benchmark (maybe along with regular JS objects)? I'd be interested in trying it out myself...

[crudh]

@wmertens: not sure they are interested since seamless isn't hashtrie based? I guess that's why they haven't got a vanilla JS entry either.

But if you can check with them and get an ok I will clean up my branch, add regular JS and submit a PR.

[wmertens]

@crudh turns out they make the benchmark too slow, see referenced issue :-)

[crudh]

@wmertens: hehe, ok :) yeah, it's an unfair comparison. Even for simple things like getting a count (the count test) then for a vanilla/seamless object you need Object.keys which is o(N) where the more complex libraries can hold their own count.

[rtfeldman]

Thanks for your awesome work on this @crudh! I'm going to close this issue now that we have some benchmarks, but will tag #6 to remember this is worth revisiting once Prod Mode exists.

[jeffbski]

Has the benchmark been redone to use prod mode yet? It would be interesting to see how things compare.