search

iterative / dvc-bench

Benchmarks for DVC
http://bench.dvc.org/
Apache License 2.0
20 stars 10 forks source link

add import-url/update tests #405

Closed dberenbaum closed 1 year ago

dberenbaum commented 1 year ago

Fix #404

github-actions[bot] commented 1 year ago 

------------------------------------------------------------------------ benchmark 'test_diff-diff': 5 tests ------------------------------------------------------------------------
Name (time in s) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_diff-diff[2.11.0] 1.6391 (1.22) 1.6391 (1.22) 1.6391 (1.22) 0.0000 (1.0) 1.6391 (1.22) 0.0000 (1.0) 0;0 0.6101 (0.82) 1 1
test_diff-diff[2.18.1] 1.3447 (1.00) 1.3447 (1.00) 1.3447 (1.00) 0.0000 (1.0) 1.3447 (1.00) 0.0000 (1.0) 0;0 0.7437 (1.00) 1 1
test_diff-diff[2.24.0] 1.3447 (1.0) 1.3447 (1.0) 1.3447 (1.0) 0.0000 (1.0) 1.3447 (1.0) 0.0000 (1.0) 0;0 0.7437 (1.0) 1 1
test_diff-diff[2.6.3] 3.0665 (2.28) 3.0665 (2.28) 3.0665 (2.28) 0.0000 (1.0) 3.0665 (2.28) 0.0000 (1.0) 0;0 0.3261 (0.44) 1 1
test_diff-diff[main] 1.4027 (1.04) 1.4027 (1.04) 1.4027 (1.04) 0.0000 (1.0) 1.4027 (1.04) 0.0000 (1.0) 0;0 0.7129 (0.96) 1 1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------ benchmark 'test_diff-diff-changed': 5 tests ------------------------------------------------------------------------
Name (time in s) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_diff-diff-changed[2.11.0] 1.1292 (1.80) 1.1292 (1.80) 1.1292 (1.80) 0.0000 (1.0) 1.1292 (1.80) 0.0000 (1.0) 0;0 0.8856 (0.55) 1 1
test_diff-diff-changed[2.18.1] 0.6265 (1.0) 0.6265 (1.0) 0.6265 (1.0) 0.0000 (1.0) 0.6265 (1.0) 0.0000 (1.0) 0;0 1.5962 (1.0) 1 1
test_diff-diff-changed[2.24.0] 0.6320 (1.01) 0.6320 (1.01) 0.6320 (1.01) 0.0000 (1.0) 0.6320 (1.01) 0.0000 (1.0) 0;0 1.5824 (0.99) 1 1
test_diff-diff-changed[2.6.3] 2.3722 (3.79) 2.3722 (3.79) 2.3722 (3.79) 0.0000 (1.0) 2.3722 (3.79) 0.0000 (1.0) 0;0 0.4215 (0.26) 1 1
test_diff-diff-changed[main] 0.6484 (1.04) 0.6484 (1.04) 0.6484 (1.04) 0.0000 (1.0) 0.6484 (1.04) 0.0000 (1.0) 0;0 1.5422 (0.97) 1 1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------- benchmark 'test_diff-diff-changed-noop': 5 tests --------------------------------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_diff-diff-changed-noop[2.11.0] 488.9110 (1.0) 488.9110 (1.0) 488.9110 (1.0) 0.0000 (1.0) 488.9110 (1.0) 0.0000 (1.0) 0;0 2.0454 (1.0) 1 1
test_diff-diff-changed-noop[2.18.1] 626.2493 (1.28) 626.2493 (1.28) 626.2493 (1.28) 0.0000 (1.0) 626.2493 (1.28) 0.0000 (1.0) 0;0 1.5968 (0.78) 1 1
test_diff-diff-changed-noop[2.24.0] 628.7941 (1.29) 628.7941 (1.29) 628.7941 (1.29) 0.0000 (1.0) 628.7941 (1.29) 0.0000 (1.0) 0;0 1.5903 (0.78) 1 1
test_diff-diff-changed-noop[2.6.3] 2,357.1282 (4.82) 2,357.1282 (4.82) 2,357.1282 (4.82) 0.0000 (1.0) 2,357.1282 (4.82) 0.0000 (1.0) 0;0 0.4242 (0.21) 1 1
test_diff-diff-changed-noop[main] 648.9828 (1.33) 648.9828 (1.33) 648.9828 (1.33) 0.0000 (1.0) 648.9828 (1.33) 0.0000 (1.0) 0;0 1.5409 (0.75) 1 1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------- benchmark 'test_diff-diff-noop': 5 tests --------------------------------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_diff-diff-noop[2.11.0] 454.0626 (1.0) 454.0626 (1.0) 454.0626 (1.0) 0.0000 (1.0) 454.0626 (1.0) 0.0000 (1.0) 0;0 2.2023 (1.0) 1 1
test_diff-diff-noop[2.18.1] 580.6235 (1.28) 580.6235 (1.28) 580.6235 (1.28) 0.0000 (1.0) 580.6235 (1.28) 0.0000 (1.0) 0;0 1.7223 (0.78) 1 1
test_diff-diff-noop[2.24.0] 592.6711 (1.31) 592.6711 (1.31) 592.6711 (1.31) 0.0000 (1.0) 592.6711 (1.31) 0.0000 (1.0) 0;0 1.6873 (0.77) 1 1
test_diff-diff-noop[2.6.3] 1,750.2808 (3.85) 1,750.2808 (3.85) 1,750.2808 (3.85) 0.0000 (1.0) 1,750.2808 (3.85) 0.0000 (1.0) 0;0 0.5713 (0.26) 1 1
test_diff-diff-noop[main] 616.1689 (1.36) 616.1689 (1.36) 616.1689 (1.36) 0.0000 (1.0) 616.1689 (1.36) 0.0000 (1.0) 0;0 1.6229 (0.74) 1 1
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Legend:
 Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
 OPS: Operations Per Second, computed as 1 / Mean\n```
github-actions[bot] commented 1 year ago 

------------------------------------------------------------------------ benchmark 'test_diff-diff': 5 tests ------------------------------------------------------------------------
Name (time in s) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_diff-diff[2.11.0] 1.6438 (1.23) 1.6438 (1.23) 1.6438 (1.23) 0.0000 (1.0) 1.6438 (1.23) 0.0000 (1.0) 0;0 0.6084 (0.81) 1 1
test_diff-diff[2.18.1] 1.3326 (1.0) 1.3326 (1.0) 1.3326 (1.0) 0.0000 (1.0) 1.3326 (1.0) 0.0000 (1.0) 0;0 0.7504 (1.0) 1 1
test_diff-diff[2.24.0] 2.0672 (1.55) 2.0672 (1.55) 2.0672 (1.55) 0.0000 (1.0) 2.0672 (1.55) 0.0000 (1.0) 0;0 0.4837 (0.64) 1 1
test_diff-diff[2.6.3] 3.0979 (2.32) 3.0979 (2.32) 3.0979 (2.32) 0.0000 (1.0) 3.0979 (2.32) 0.0000 (1.0) 0;0 0.3228 (0.43) 1 1
test_diff-diff[main] 1.3945 (1.05) 1.3945 (1.05) 1.3945 (1.05) 0.0000 (1.0) 1.3945 (1.05) 0.0000 (1.0) 0;0 0.7171 (0.96) 1 1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------ benchmark 'test_diff-diff-changed': 5 tests ------------------------------------------------------------------------
Name (time in s) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_diff-diff-changed[2.11.0] 1.1412 (1.82) 1.1412 (1.82) 1.1412 (1.82) 0.0000 (1.0) 1.1412 (1.82) 0.0000 (1.0) 0;0 0.8763 (0.55) 1 1
test_diff-diff-changed[2.18.1] 0.6255 (1.0) 0.6255 (1.0) 0.6255 (1.0) 0.0000 (1.0) 0.6255 (1.0) 0.0000 (1.0) 0;0 1.5988 (1.0) 1 1
test_diff-diff-changed[2.24.0] 0.6276 (1.00) 0.6276 (1.00) 0.6276 (1.00) 0.0000 (1.0) 0.6276 (1.00) 0.0000 (1.0) 0;0 1.5933 (1.00) 1 1
test_diff-diff-changed[2.6.3] 4.1376 (6.62) 4.1376 (6.62) 4.1376 (6.62) 0.0000 (1.0) 4.1376 (6.62) 0.0000 (1.0) 0;0 0.2417 (0.15) 1 1
test_diff-diff-changed[main] 0.6547 (1.05) 0.6547 (1.05) 0.6547 (1.05) 0.0000 (1.0) 0.6547 (1.05) 0.0000 (1.0) 0;0 1.5275 (0.96) 1 1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------- benchmark 'test_diff-diff-changed-noop': 5 tests --------------------------------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_diff-diff-changed-noop[2.11.0] 488.4511 (1.0) 488.4511 (1.0) 488.4511 (1.0) 0.0000 (1.0) 488.4511 (1.0) 0.0000 (1.0) 0;0 2.0473 (1.0) 1 1
test_diff-diff-changed-noop[2.18.1] 630.2225 (1.29) 630.2225 (1.29) 630.2225 (1.29) 0.0000 (1.0) 630.2225 (1.29) 0.0000 (1.0) 0;0 1.5867 (0.78) 1 1
test_diff-diff-changed-noop[2.24.0] 631.5591 (1.29) 631.5591 (1.29) 631.5591 (1.29) 0.0000 (1.0) 631.5591 (1.29) 0.0000 (1.0) 0;0 1.5834 (0.77) 1 1
test_diff-diff-changed-noop[2.6.3] 2,368.0538 (4.85) 2,368.0538 (4.85) 2,368.0538 (4.85) 0.0000 (1.0) 2,368.0538 (4.85) 0.0000 (1.0) 0;0 0.4223 (0.21) 1 1
test_diff-diff-changed-noop[main] 653.2052 (1.34) 653.2052 (1.34) 653.2052 (1.34) 0.0000 (1.0) 653.2052 (1.34) 0.0000 (1.0) 0;0 1.5309 (0.75) 1 1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

-------------------------------------------------------------------------------- benchmark 'test_diff-diff-noop': 5 tests --------------------------------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_diff-diff-noop[2.11.0] 451.2774 (1.0) 451.2774 (1.0) 451.2774 (1.0) 0.0000 (1.0) 451.2774 (1.0) 0.0000 (1.0) 0;0 2.2159 (1.0) 1 1
test_diff-diff-noop[2.18.1] 578.3042 (1.28) 578.3042 (1.28) 578.3042 (1.28) 0.0000 (1.0) 578.3042 (1.28) 0.0000 (1.0) 0;0 1.7292 (0.78) 1 1
test_diff-diff-noop[2.24.0] 595.0915 (1.32) 595.0915 (1.32) 595.0915 (1.32) 0.0000 (1.0) 595.0915 (1.32) 0.0000 (1.0) 0;0 1.6804 (0.76) 1 1
test_diff-diff-noop[2.6.3] 1,754.6923 (3.89) 1,754.6923 (3.89) 1,754.6923 (3.89) 0.0000 (1.0) 1,754.6923 (3.89) 0.0000 (1.0) 0;0 0.5699 (0.26) 1 1
test_diff-diff-noop[main] 613.7889 (1.36) 613.7889 (1.36) 613.7889 (1.36) 0.0000 (1.0) 613.7889 (1.36) 0.0000 (1.0) 0;0 1.6292 (0.74) 1 1
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Legend:
 Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
 OPS: Operations Per Second, computed as 1 / Mean\n```