search

iterative / dvc-bench

Benchmarks for DVC
http://bench.dvc.org/
Apache License 2.0
20 stars 10 forks source link

add more ops to test_sharing use case #434

Closed dberenbaum closed 1 year ago

dberenbaum commented 1 year ago

Added some more operations to the test sharing use case to simulate some other scenarios like additions and deletions from a tracked directory and garbage collecting.

If we start using dvc-bench for https://github.com/iterative/dvc/issues/9108, this should be enough to cover the basic scenarios needed.

Fixes #407

github-actions[bot] commented 1 year ago 

---------------------------------------------------------------------------- benchmark 'test_gc-gc': 8 tests ----------------------------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_gc-gc[2.11.0] 518.5391 (1.04) 518.5391 (1.04) 518.5391 (1.04) 0.0000 (1.0) 518.5391 (1.04) 0.0000 (1.0) 0;0 1.9285 (0.96) 1 1
test_gc-gc[2.18.1] 568.8980 (1.14) 568.8980 (1.14) 568.8980 (1.14) 0.0000 (1.0) 568.8980 (1.14) 0.0000 (1.0) 0;0 1.7578 (0.88) 1 1
test_gc-gc[2.39.0] 578.0077 (1.16) 578.0077 (1.16) 578.0077 (1.16) 0.0000 (1.0) 578.0077 (1.16) 0.0000 (1.0) 0;0 1.7301 (0.87) 1 1
test_gc-gc[2.40.0] 589.2863 (1.18) 589.2863 (1.18) 589.2863 (1.18) 0.0000 (1.0) 589.2863 (1.18) 0.0000 (1.0) 0;0 1.6970 (0.85) 1 1
test_gc-gc[2.41.1] 597.6019 (1.19) 597.6019 (1.19) 597.6019 (1.19) 0.0000 (1.0) 597.6019 (1.19) 0.0000 (1.0) 0;0 1.6734 (0.84) 1 1
test_gc-gc[2.45.0] 584.8465 (1.17) 584.8465 (1.17) 584.8465 (1.17) 0.0000 (1.0) 584.8465 (1.17) 0.0000 (1.0) 0;0 1.7099 (0.86) 1 1
test_gc-gc[2.6.3] 500.1550 (1.0) 500.1550 (1.0) 500.1550 (1.0) 0.0000 (1.0) 500.1550 (1.0) 0.0000 (1.0) 0;0 1.9994 (1.0) 1 1
test_gc-gc[main] 632.3890 (1.26) 632.3890 (1.26) 632.3890 (1.26) 0.0000 (1.0) 632.3890 (1.26) 0.0000 (1.0) 0;0 1.5813 (0.79) 1 1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Legend:
 Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
 OPS: Operations Per Second, computed as 1 / Mean\n```
efiop commented 1 year ago

Does it follow sharing use-case from the docs? Or is this something completely new?

dberenbaum commented 1 year ago

I didn't realize this was intended to follow something from the docs. Where in the docs is it?

I was trying to cover some other common data sharing operations, like adding or deleting files from an existing output and garbage collecting.

efiop commented 1 year ago

Indeed, no trace of it in docs anymore, I guess it got refactored/thrown-out at some point.

Do you need all the previous steps to test what you want? If not, maybe it could be put into a smaller separate test? I'm worried that this might get unwieldy if we shove everything into one test.

Maybe separate tests for each command instead? Or is the whole scenario important here?

dberenbaum commented 1 year ago

Tried to separate the use cases better, PTAL

github-actions[bot] commented 1 year ago 

---------------------------------------------------------------------------- benchmark 'test_gc-gc': 8 tests ----------------------------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_gc-gc[2.11.0] 446.0973 (1.0) 446.0973 (1.0) 446.0973 (1.0) 0.0000 (1.0) 446.0973 (1.0) 0.0000 (1.0) 0;0 2.2417 (1.0) 1 1
test_gc-gc[2.18.1] 458.9156 (1.03) 458.9156 (1.03) 458.9156 (1.03) 0.0000 (1.0) 458.9156 (1.03) 0.0000 (1.0) 0;0 2.1790 (0.97) 1 1
test_gc-gc[2.39.0] 458.2217 (1.03) 458.2217 (1.03) 458.2217 (1.03) 0.0000 (1.0) 458.2217 (1.03) 0.0000 (1.0) 0;0 2.1823 (0.97) 1 1
test_gc-gc[2.40.0] 526.2311 (1.18) 526.2311 (1.18) 526.2311 (1.18) 0.0000 (1.0) 526.2311 (1.18) 0.0000 (1.0) 0;0 1.9003 (0.85) 1 1
test_gc-gc[2.41.1] 511.0742 (1.15) 511.0742 (1.15) 511.0742 (1.15) 0.0000 (1.0) 511.0742 (1.15) 0.0000 (1.0) 0;0 1.9567 (0.87) 1 1
test_gc-gc[2.45.0] 499.0551 (1.12) 499.0551 (1.12) 499.0551 (1.12) 0.0000 (1.0) 499.0551 (1.12) 0.0000 (1.0) 0;0 2.0038 (0.89) 1 1
test_gc-gc[2.6.3] 517.5766 (1.16) 517.5766 (1.16) 517.5766 (1.16) 0.0000 (1.0) 517.5766 (1.16) 0.0000 (1.0) 0;0 1.9321 (0.86) 1 1
test_gc-gc[main] 543.3533 (1.22) 543.3533 (1.22) 543.3533 (1.22) 0.0000 (1.0) 543.3533 (1.22) 0.0000 (1.0) 0;0 1.8404 (0.82) 1 1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Legend:
 Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
 OPS: Operations Per Second, computed as 1 / Mean\n```
dberenbaum commented 1 year ago

Why are there auto-generated comments but only for gc?

efiop commented 1 year ago

Looks like something broke in PR handling. Will take a look later.

Thank you!