Initial cut at benchmarking

MichaelTiemannOSC commented 10 months ago

Add some benchmark tests to compare PintArray performance against NumPy arrays of quantities. Over time we should be able to show specific patterns where PintArrays confer substantial performance advantages over naive use of Quantities with Pandas.

The Check dependency specification is failing because I don't know how to get the unit test bits of pyproject.toml to respect the dependencies of the test declaration. The pytest install is not installing any of the components I've asked for.

[ ] Closes # (insert issue number)
[ ] Executed pre-commit run --all-files with no errors
[ ] The change is fully covered by automated unit tests
[ ] Documented in docs/ as appropriate
[ ] Added an entry to the CHANGES file

MichaelTiemannOSC commented 10 months ago

Progress report:

There was redundancy/confusion as the benchmarks attempted to mix and match not only the sensible left-hand/right-hand parameters of a particular data type/data length, but also across species of tests. I've separated that out to:

pint: tests Quantity arrays (e.g, Q([1, 2, 3], 'meter')) and also PintArrays (e.g., PA([1, 2, 3], 'meter')).
numpy: tests ndarrays filled with quantities. These ndarrays have "object" dtype.
pandas: tests Pandas Series filled with quantities. These Series have "object" dtype.
pint-pandas: tests Pandas Series made with PintArrays.

Compare:

(Pdb) data['short_ndarray']
array([1., 2., 3.])
(Pdb) data['short_Qarray_meter']
<Quantity([1. 2. 3.], 'meter')>
(Pdb) data['short_PintArray_meter']
<PintArray>
[1.0, 2.0, 3.0]
Length: 3, dtype: pint[meter]
(Pdb) data['short_ndarray_meter']
array([<Quantity(1.0, 'meter')>, <Quantity(2.0, 'meter')>,
       <Quantity(3.0, 'meter')>], dtype=object)
(Pdb) data['short_Series_meter']
0    1.0 meter
1    2.0 meter
2    3.0 meter
dtype: object
(Pdb) data['short_Series_PA_meter']
0    1.0
1    2.0
2    3.0
dtype: pint[meter]

The numbers below show how extraordinarily better performance is with Pint-ish arrays than simply fobbing off quantities into ndarrays and Series without any help. Here's the subset that deals with "meters OP kilometers":

================================================================================================================================= test session starts =================================================================================================================================
platform darwin -- Python 3.9.18, pytest-7.3.2, pluggy-1.3.0
Matplotlib: 3.7.2
Freetype: 2.6.1
benchmark: 4.0.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /Users/michael/Documents/GitHub/MichaelTiemannOSC/pint-pandas
configfile: pyproject.toml
plugins: dash-2.11.1, subtests-0.11.0, hypothesis-6.82.7, cython-0.2.1, nbval-0.10.0, cov-4.1.0, asyncio-0.21.1, mpl-0.16.1, anyio-3.7.1, xdist-3.3.1, benchmark-4.0.0
asyncio: mode=strict
collected 205 items / 180 deselected / 25 selected                                                                                                                                                                                                                                    

pint_pandas/testsuite/benchmarks/test_30_numpy.py .........................                                                                                                                                                                                                     [100%]

----------------------------------------------------------------------------------------------------------- benchmark: 25 tests -----------------------------------------------------------------------------------------------------------
Name (time in us)                                     Min                     Max                   Mean                 StdDev                 Median                   IQR            Outliers          OPS            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_op2_pint[mul-keys5]                          15.2060 (1.0)          176.9790 (1.30)         16.9094 (1.0)           2.8152 (1.0)          16.0460 (1.0)          0.8780 (1.05)    1154;3126  59,138.5921 (1.0)       15089           1
test_op2_pint[truediv-keys5]                      15.9450 (1.05)         136.4110 (1.0)          17.1432 (1.01)          3.1589 (1.12)         16.5800 (1.03)         0.8370 (1.0)       319;708  58,332.2530 (0.99)      16657           1
test_op2_pint[eq-keys5]                           28.5230 (1.88)         159.2180 (1.17)         31.7822 (1.88)          5.4851 (1.95)         29.5100 (1.84)         1.9550 (2.34)     501;1174  31,464.1910 (0.53)       5251           1
test_op2_pint[lt-keys5]                           41.7850 (2.75)         481.7010 (3.53)         46.7901 (2.77)          9.5785 (3.40)         43.2230 (2.69)        10.0985 (12.07)      264;66  21,372.0361 (0.36)       5451           1
test_op2_pint[add-keys5]                          42.8290 (2.82)         210.1070 (1.54)         46.2670 (2.74)          6.1372 (2.18)         44.0020 (2.74)         2.2390 (2.68)      387;390  21,613.6997 (0.37)       2562           1
test_op2_PintArray[eq-pa_keys5]                   59.6400 (3.92)         209.2680 (1.53)         65.1067 (3.85)          8.2151 (2.92)         61.1200 (3.81)         3.5800 (4.28)      888;912  15,359.4042 (0.26)       4369           1
test_op2_PintArray[mul-pa_keys5]                  71.8160 (4.72)         293.8310 (2.15)         80.3090 (4.75)         11.9202 (4.23)         73.8355 (4.60)        17.8285 (21.30)      492;26  12,451.9017 (0.21)       4196           1
test_op2_PintArray[lt-pa_keys5]                   72.5860 (4.77)         303.8950 (2.23)         80.8647 (4.78)         10.6172 (3.77)         75.9405 (4.73)        17.5960 (21.02)     1165;16  12,366.3393 (0.21)       4446           1
test_op2_PintArray[truediv-pa_keys5]              74.0210 (4.87)         292.2900 (2.14)         79.3044 (4.69)          9.3599 (3.32)         75.6650 (4.72)         6.1385 (7.33)      399;285  12,609.6439 (0.21)       4660           1
test_op2_PintArray[add-pa_keys5]                 106.2930 (6.99)         558.0470 (4.09)        119.4598 (7.06)         18.0572 (6.41)        111.1345 (6.93)        26.1855 (31.28)      365;14   8,371.0203 (0.14)       2852           1
test_op2_pint_pandas[eq-pd_pa_keys5]             116.8350 (7.68)         322.0990 (2.36)        126.3630 (7.47)         13.8153 (4.91)        119.8755 (7.47)         6.7115 (8.02)      478;542   7,913.7102 (0.13)       3124           1
test_op2_pint_pandas[lt-pd_pa_keys5]             130.6460 (8.59)         448.3420 (3.29)        143.4476 (8.48)         19.8231 (7.04)        134.1680 (8.36)         9.1065 (10.88)     521;555   6,971.1856 (0.12)       2983           1
test_op2_pint_pandas[mul-pd_pa_keys5]            135.3970 (8.90)         326.8720 (2.40)        150.7436 (8.91)         17.5624 (6.24)        139.0810 (8.67)        33.1670 (39.63)       801;4   6,633.7798 (0.11)       2754           1
test_op2_pint_pandas[truediv-pd_pa_keys5]        136.6040 (8.98)         384.4900 (2.82)        148.4773 (8.78)         14.9308 (5.30)        146.2980 (9.12)        12.1945 (14.57)     235;106   6,735.0353 (0.11)       2467           1
test_op2_pint_pandas[add-pd_pa_keys5]            183.4380 (12.06)        468.7320 (3.44)        203.4195 (12.03)        23.0310 (8.18)        193.6135 (12.07)       28.5885 (34.16)      558;42   4,915.9484 (0.08)       2404           1
test_op2_numpy[multiply-np_keys5]             12,097.7410 (795.59)    86,812.4030 (636.40)   13,719.9214 (811.38)    8,416.5507 (>1000.0)  12,544.2345 (781.77)     472.5050 (564.52)       1;11      72.8867 (0.00)         78           1
test_op2_pandas[mul-pd_keys5]                 12,301.2980 (808.98)    88,861.2480 (651.42)   13,862.6987 (819.82)    8,784.5490 (>1000.0)  12,739.7490 (793.95)     485.8690 (580.49)        1;3      72.1360 (0.00)         75           1
test_op2_numpy[divide-np_keys5]               12,699.4180 (835.16)    87,682.4130 (642.78)   14,216.6670 (840.75)    8,721.9007 (>1000.0)  13,181.4270 (821.48)     305.1073 (364.52)        1;3      70.3400 (0.00)         73           1
test_op2_pandas[truediv-pd_keys5]             12,737.8420 (837.69)    89,511.1440 (656.19)   14,225.3962 (841.27)    8,877.0167 (>1000.0)  13,132.7200 (818.44)     365.1270 (436.23)        1;2      70.2968 (0.00)         74           1
test_op2_pandas[eq-pd_keys5]                  19,133.7320 (>1000.0)   20,710.9750 (151.83)   19,799.7838 (>1000.0)     393.8900 (139.91)   19,709.2350 (>1000.0)    597.9933 (714.45)       15;0      50.5056 (0.00)         49           1
test_op2_numpy[equal-np_keys5]                19,136.2810 (>1000.0)   20,640.1830 (151.31)   19,717.4706 (>1000.0)     398.9341 (141.71)   19,660.4520 (>1000.0)    537.4562 (642.12)       17;0      50.7164 (0.00)         49           1
test_op2_pandas[lt-pd_keys5]                  34,397.9560 (>1000.0)   36,566.9840 (268.06)   35,500.8638 (>1000.0)     565.7559 (200.96)   35,491.0965 (>1000.0)    787.3280 (940.65)        8;0      28.1683 (0.00)         26           1
test_op2_pandas[add-pd_keys5]                 34,651.4430 (>1000.0)   36,438.7960 (267.13)   35,490.1571 (>1000.0)     515.8815 (183.25)   35,507.6375 (>1000.0)    741.1015 (885.43)       10;0      28.1768 (0.00)         24           1
test_op2_numpy[less-np_keys5]                 34,820.1250 (>1000.0)   37,346.9710 (273.78)   35,742.7535 (>1000.0)     644.0765 (228.78)   35,844.7870 (>1000.0)  1,080.7022 (>1000.0)       9;0      27.9777 (0.00)         25           1
test_op2_numpy[add-np_keys5]                  34,850.4270 (>1000.0)  111,795.4500 (819.55)   38,649.3541 (>1000.0)  15,246.1769 (>1000.0)  35,673.3280 (>1000.0)    778.4795 (930.08)        1;1      25.8737 (0.00)         25           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Legend:
  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
  OPS: Operations Per Second, computed as 1 / Mean
========================================================================================================================= 25 passed, 180 deselected in 21.03s =========================================================================================================================

Interesting that addition takes 2x-3x the work of multiplication.

Also interesting: PintArrays have half the performance of Quantity arrays. Pint Pandas has half the performance of PintArrays. And Numpy and Pandas without PintArrays have 200x the overhead compared with Pandas + PintArrays (and approx 1000x the overhead compared with Quantity arrays).

n.b.: I still don't know how to fix the CI/CD problem of getting pytest-benchmark to install properly. I have installed it manually on my system, but don't see the magic to make pyproject.toml or conftest.py do the right thing.

andrewgsavage commented 10 months ago

I don't think there's much point comparing to Quantity arrays, that will be more about the overheads involved in the elementwise operations on each pint Quantity/on the pandas side dispatching to them. It's faster, as you'd expect, and there's no reason to use arrays of quantities unless you have to.

I do think it would be good to compare a PintArray to a standard pandas array, as it may show operations that could be faster with a better implementation.

andrewgsavage commented 10 months ago

I missed that you'd compared to Q(np.array([1,2]), "m"). That is an interesting comparison!

hgrecco / pint-pandas

Initial cut at benchmarking #198