search

isi-vista / immutablecollections

A library for immutable collections, in the spirit of Guava's Immutable Collections.
MIT License
3 stars 2 forks source link

Simple creation benchmarks for ImmutableSet and ImmutableList #22

Closed gabbard closed 5 years ago

gabbard commented 5 years ago

Partially addresses #15 , #14

gabbard commented 5 years ago 
------------------------------------------------------------------------------------------ benchmark 'Creating from big frozenset': 3 tests -----------------------------------------------------------------------------------------
Name (time in ns)                Min                       Max                      Mean                  StdDev                    Median                     IQR            Outliers              OPS            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
frozenset                    41.9580 (1.0)            440.5594 (1.0)             47.1240 (1.0)           13.5645 (1.0)             41.9580 (1.0)            6.9930 (1.0)     3463;3463  21,220,615.7540 (1.0)      100000         143
set                      51,000.0000 (>1000.0)    409,000.0000 (928.37)      56,053.7589 (>1000.0)   13,807.9584 (>1000.0)     52,000.0000 (>1000.0)    1,000.0000 (143.00)   800;2142      17,840.0168 (0.00)       9524           1
ImmutableSet          1,047,000.0000 (>1000.0)  2,001,000.0000 (>1000.0)  1,171,621.0996 (>1000.0)  135,563.1902 (>1000.0)  1,127,000.0000 (>1000.0)  163,250.0000 (>1000.0)    113;16         853.5183 (0.00)        673           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------ benchmark 'Creating from big list': 6 tests ------------------------------------------------------------------------------
Name (time in us)            Min                   Max                  Mean              StdDev                Median                 IQR            Outliers          OPS            Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
list                     26.0000 (1.0)        342.0000 (1.0)         29.3941 (1.0)        8.3899 (1.0)         28.0000 (1.0)        2.0000 (2.00)    1385;1640  34,020.4471 (1.0)       21277           1
tuple                    27.0000 (1.04)       964.0000 (2.82)        30.4866 (1.04)      11.8144 (1.41)        28.0000 (1.00)       1.0000 (1.0)     1297;3650  32,801.3134 (0.96)      23256           1
ImmutableList            61.0000 (2.35)       788.0000 (2.30)        73.9014 (2.51)      26.8039 (3.19)        64.0000 (2.29)       8.0000 (8.00)    1010;1830  13,531.5434 (0.40)      10000           1
frozenset               191.0000 (7.35)     1,343.0000 (3.93)       226.6852 (7.71)      59.9255 (7.14)       200.0000 (7.14)      43.0000 (43.00)     450;371   4,411.4041 (0.13)       4066           1
set                     191.0000 (7.35)     1,873.0000 (5.48)       232.9883 (7.93)      94.1754 (11.22)      199.0000 (7.11)      40.0000 (40.00)     327;479   4,292.0610 (0.13)       4017           1
ImmutableSet          1,193.0000 (45.88)    5,607.0000 (16.39)    1,540.8000 (52.42)    476.2356 (56.76)    1,363.5000 (48.70)    318.0000 (318.00)      85;82     649.0135 (0.02)        750           1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------- benchmark 'Creating from big set': 3 tests ------------------------------------------------------------------------------
Name (time in us)            Min                   Max                  Mean              StdDev                Median                 IQR            Outliers          OPS            Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
frozenset                49.0000 (1.0)        405.0000 (1.32)        60.5523 (1.0)       17.0886 (1.0)         52.0000 (1.0)       11.0000 (1.0)     1742;1639  16,514.6527 (1.0)       13158           1
set                      50.0000 (1.02)       307.0000 (1.0)         63.3455 (1.05)      19.5377 (1.14)        55.0000 (1.06)      19.0000 (1.73)      343;140  15,786.4505 (0.96)       2501           1
ImmutableSet          1,049.0000 (21.41)    2,389.0000 (7.78)     1,305.2515 (21.56)    273.7017 (16.02)    1,196.0000 (23.00)    311.0000 (28.27)       84;27     766.1359 (0.05)        497           1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------------------- benchmark 'Creating from big tuple': 6 tests --------------------------------------------------------------------------------------------
Name (time in ns)                Min                        Max                      Mean                  StdDev                    Median                     IQR            Outliers              OPS            Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
tuple                        55.8659 (1.0)           2,256.9832 (1.0)             64.6637 (1.0)           17.0767 (1.0)             61.4525 (1.0)            5.5866 (1.0)     4722;4987  15,464,616.6978 (1.0)      100000         179
list                     26,000.0000 (465.40)      396,000.0000 (175.46)      28,336.0852 (438.21)     9,242.3130 (541.22)      27,000.0000 (439.36)     2,000.0000 (358.00)    500;645      35,290.6900 (0.00)      10417           1
ImmutableList            61,000.0000 (>1000.0)     786,000.0000 (348.25)      68,057.6265 (>1000.0)   21,018.2130 (>1000.0)     63,000.0000 (>1000.0)    4,000.0000 (716.00)    340;648      14,693.4304 (0.00)       5848           1
frozenset               190,000.0000 (>1000.0)     996,000.0000 (441.30)     211,638.3372 (>1000.0)   41,120.5700 (>1000.0)    194,000.0000 (>1000.0)   22,000.0000 (>1000.0)   505;536       4,725.0418 (0.00)       4330           1
set                     190,000.0000 (>1000.0)     536,000.0000 (237.49)     205,133.5006 (>1000.0)   29,371.3884 (>1000.0)    193,000.0000 (>1000.0)    8,000.0000 (>1000.0)   395;880       4,874.8742 (0.00)       3985           1
ImmutableSet          1,175,000.0000 (>1000.0)  11,467,000.0000 (>1000.0)  1,563,293.2227 (>1000.0)  662,502.3094 (>1000.0)  1,335,000.0000 (>1000.0)  417,750.0000 (>1000.0)     69;64         639.6753 (0.00)        723           1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------ benchmark 'Creating from empty frozenset': 3 tests -----------------------------------------------------------------------------
Name (time in ns)            Min                    Max                  Mean                StdDev                Median                   IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
frozenset                37.6569 (1.0)         439.3305 (1.0)         46.2467 (1.0)         14.5165 (1.0)         41.8410 (1.0)          4.1841 (1.0)     7662;8888   21,623.1798 (1.0)       90910         239
set                      64.7482 (1.72)      2,697.8417 (6.14)        88.7194 (1.92)        35.8345 (2.47)        71.9424 (1.72)        14.3885 (3.44)   17046;22395   11,271.4918 (0.52)     100001         139
ImmutableSet          6,000.0000 (159.33)   88,000.0000 (200.30)   8,164.0147 (176.53)   4,265.4555 (293.83)   7,000.0000 (167.30)   1,000.0000 (239.00)   951;1435      122.4888 (0.01)      14706           1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------- benchmark 'Creating from empty list': 6 tests ---------------------------------------------------------------------------------
Name (time in ns)            Min                     Max                  Mean                StdDev                Median                   IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
tuple                    60.0000 (1.0)          830.0000 (1.12)        72.6964 (1.0)         20.2048 (1.0)         70.0000 (1.0)         10.0000 (1.0)     2314;2314   13,755.8341 (1.0)       76924         100
list                     80.0000 (1.33)         740.0000 (1.0)         94.6145 (1.30)        23.7537 (1.18)        90.0000 (1.29)        10.0000 (1.00)    3915;3915   10,569.2045 (0.77)     100000         100
frozenset                90.0000 (1.50)       1,190.0000 (1.61)       116.9546 (1.61)        42.7023 (2.11)       100.0000 (1.43)        20.0000 (2.00)   10384;11852    8,550.3231 (0.62)      58824         100
set                      90.0000 (1.50)       4,790.0000 (6.47)       122.5634 (1.69)        47.7687 (2.36)       100.0000 (1.43)        60.0000 (6.00)   14988;2155    8,159.0443 (0.59)     100001         100
ImmutableList         4,000.0000 (66.67)     82,000.0000 (110.81)   5,577.7142 (76.73)    2,464.0386 (121.95)   5,000.0000 (71.43)    1,000.0000 (100.00)    583;705      179.2849 (0.01)      18182           1
ImmutableSet          6,000.0000 (100.00)   130,000.0000 (175.68)   8,149.2080 (112.10)   4,010.1985 (198.48)   7,000.0000 (100.00)   1,000.0000 (100.00)   891;1827      122.7113 (0.01)      13699           1
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

--------------------------------------------------------------------------------- benchmark 'Creating from empty set': 3 tests --------------------------------------------------------------------------------
Name (time in ns)            Min                    Max                  Mean                StdDev                Median                   IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
frozenset                62.9371 (1.0)       1,216.7832 (1.0)         77.0403 (1.03)        22.8540 (1.06)        69.9301 (1.0)          6.9930 (1.0)     5920;6708   12,980.2111 (0.97)     100000         143
set                      64.7482 (1.03)      1,690.6475 (1.39)        74.5329 (1.0)         21.6329 (1.0)         71.9424 (1.03)         7.1942 (1.03)    4646;6274   13,416.9024 (1.0)      100001         139
ImmutableSet          6,000.0000 (95.33)    96,000.0000 (78.90)    8,367.8286 (112.27)   4,510.1651 (208.49)   7,000.0000 (100.10)   1,000.0000 (143.00)   896;1400      119.5053 (0.01)      11905           1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------- benchmark 'Creating from empty tuple': 6 tests --------------------------------------------------------------------------------
Name (time in ns)            Min                     Max                  Mean                StdDev                Median                   IQR            Outliers  OPS (Kops/s)            Rounds  Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
tuple                    53.8922 (1.0)          425.1497 (1.0)         66.0407 (1.0)         16.2152 (1.0)         59.8802 (1.0)          5.9880 (1.0)     4689;6361   15,142.1764 (1.0)       90910         167
list                     80.0000 (1.48)         500.0000 (1.18)       105.6030 (1.60)        27.6105 (1.70)       100.0000 (1.67)        20.0000 (3.34)      537;537    9,469.4236 (0.63)       9709         100
frozenset                90.0000 (1.67)         830.0000 (1.95)       104.9845 (1.59)        33.5786 (2.07)       100.0000 (1.67)        10.0000 (1.67)    3538;4144    9,525.2156 (0.63)      50001         100
set                      90.0000 (1.67)       1,210.0000 (2.85)       110.5641 (1.67)        39.7027 (2.45)       100.0000 (1.67)        20.0000 (3.34)   11815;14122    9,044.5276 (0.60)     100001         100
ImmutableList         4,000.0000 (74.22)     85,000.0000 (199.93)   5,503.6959 (83.34)    2,414.3008 (148.89)   5,000.0000 (83.50)    1,000.0000 (167.00)  1044;1044      181.6961 (0.01)      30304           1
ImmutableSet          6,000.0000 (111.33)   164,000.0000 (385.75)   8,201.8215 (124.19)   4,415.8680 (272.33)   7,000.0000 (116.90)   1,000.0000 (167.00)   931;2040      121.9241 (0.01)      15152           1
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

--------------------------------------------------------------------------------- benchmark 'Creating from small frozenset': 3 tests --------------------------------------------------------------------------------
Name (time in ns)             Min                     Max                   Mean                StdDev                 Median                   IQR              Outliers  OPS (Kops/s)            Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
frozenset                 80.3571 (1.0)          785.7143 (1.0)          95.2259 (1.0)         29.4567 (1.0)          89.2857 (1.0)          8.9286 (1.0)      8731;11716   10,501.3455 (1.0)      100000         112
set                       83.3333 (1.04)       1,116.6667 (1.42)        107.2887 (1.13)        42.1724 (1.43)        100.0000 (1.12)        16.6667 (1.87)    11691;13442    9,320.6452 (0.89)      90910          60
ImmutableSet          13,000.0000 (161.78)   210,000.0000 (267.27)   16,594.3953 (174.26)   6,900.8404 (234.27)   15,000.0000 (168.00)   2,000.0000 (224.00)    1285;1731       60.2613 (0.01)      16950           1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------- benchmark 'Creating from small list': 6 tests -----------------------------------------------------------------------------------
Name (time in ns)             Min                     Max                   Mean                StdDev                 Median                   IQR             Outliers  OPS (Kops/s)            Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
tuple                     72.0000 (1.0)        1,072.0000 (1.20)         86.6342 (1.0)         25.1236 (1.0)          80.0000 (1.0)          8.0000 (1.0)      5933;6232   11,542.7902 (1.0)       90910         125
list                     100.0000 (1.39)         990.0000 (1.11)        114.2461 (1.32)        28.4773 (1.13)        110.0000 (1.38)        10.0000 (1.25)     4760;6132    8,753.0365 (0.76)      90910         100
frozenset                110.0000 (1.53)         890.0000 (1.0)         139.8081 (1.61)        45.4886 (1.81)        120.0000 (1.50)        10.0000 (1.25)   10852;14194    7,152.6603 (0.62)      83334         100
set                      110.0000 (1.53)      11,360.0000 (12.76)       133.5510 (1.54)        64.4241 (2.56)        120.0000 (1.50)        10.0000 (1.25)     3553;4685    7,487.7746 (0.65)      47620         100
ImmutableList          5,000.0000 (69.44)    114,000.0000 (128.09)    7,478.4640 (86.32)    3,742.3074 (148.96)    7,000.0000 (87.50)    1,000.0000 (125.00)   1659;2457      133.7173 (0.01)      31250           1
ImmutableSet          13,000.0000 (180.56)   173,000.0000 (194.38)   17,962.3630 (207.34)   9,218.3517 (366.92)   15,000.0000 (187.50)   2,000.0000 (250.00)   1663;2603       55.6720 (0.00)      18519           1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------- benchmark 'Creating from small set': 3 tests -----------------------------------------------------------------------------------
Name (time in ns)             Min                     Max                   Mean                StdDev                 Median                   IQR             Outliers  OPS (Kops/s)            Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
frozenset                 80.0000 (1.0)        1,230.0000 (1.64)         91.4462 (1.01)        24.1262 (1.0)          90.0000 (1.0)         10.0000 (1.0)      4255;4958   10,935.3915 (0.99)     100000         100
set                       80.0000 (1.0)          750.0000 (1.0)          90.8813 (1.0)         26.9643 (1.12)         90.0000 (1.0)         10.0000 (1.0)      1870;2172   11,003.3670 (1.0)       41667         100
ImmutableSet          13,000.0000 (162.50)   145,000.0000 (193.33)   17,360.5310 (191.02)   8,260.1507 (342.37)   15,000.0000 (166.67)   2,000.0000 (200.00)   1687;2385       57.6019 (0.01)      16950           1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------- benchmark 'Creating from small tuple': 6 tests ----------------------------------------------------------------------------------
Name (time in ns)             Min                     Max                   Mean                StdDev                 Median                   IQR             Outliers  OPS (Kops/s)            Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
tuple                     57.3248 (1.0)          394.9045 (1.0)          67.7698 (1.0)         18.0125 (1.0)          63.6943 (1.0)         12.7389 (1.27)     7259;7248   14,755.8333 (1.0)      100000         157
list                     100.0000 (1.74)         920.0000 (2.33)        113.0101 (1.67)        25.9548 (1.44)        110.0000 (1.73)        10.0000 (1.0)      3489;5205    8,848.7651 (0.60)      90910         100
frozenset                110.0000 (1.92)       1,160.0000 (2.94)        132.2904 (1.95)        39.7962 (2.21)        120.0000 (1.88)        10.0000 (1.00)     7638;9008    7,559.1263 (0.51)      83334         100
set                      110.0000 (1.92)       1,250.0000 (3.17)        129.8343 (1.92)        39.3428 (2.18)        120.0000 (1.88)        20.0000 (2.00)     6227;6589    7,702.1278 (0.52)      76924         100
ImmutableList          5,000.0000 (87.22)     94,000.0000 (238.03)    6,991.2502 (103.16)   2,641.3345 (146.64)    7,000.0000 (109.90)   1,000.0000 (100.00)   1013;1116      143.0359 (0.01)      28572           1
ImmutableSet          13,000.0000 (226.78)   168,000.0000 (425.42)   17,815.3248 (262.88)   8,571.2081 (475.85)   15,000.0000 (235.50)   2,000.0000 (200.00)   1255;2007       56.1314 (0.00)      12346           1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Legend:
  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
  OPS: Operations Per Second, computed as 1 / Mean
=======================================================
codecov[bot] commented 5 years ago

Codecov Report

Merging #22 into master will increase coverage by 0.15%. The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #22      +/-   ##
==========================================
+ Coverage   90.97%   91.13%   +0.15%     
==========================================
  Files           7        7              
  Lines         576      575       -1     
==========================================
  Hits          524      524              
+ Misses         52       51       -1
Impacted Files Coverage Δ
immutablecollections/immutableset.py 87.04% <ø> (+0.44%) :arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update a5984ce...a94a4fa. Read the comment docs.

gabbard commented 5 years ago

Ready for re-review, be sure to note the last commit. I don't know why mypy didn't catch this before.