Closed dberenbaum closed 1 year ago
---------------------------------------------------------------------------- benchmark 'test_gc-gc': 8 tests ----------------------------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_gc-gc[2.11.0] 518.5391 (1.04) 518.5391 (1.04) 518.5391 (1.04) 0.0000 (1.0) 518.5391 (1.04) 0.0000 (1.0) 0;0 1.9285 (0.96) 1 1
test_gc-gc[2.18.1] 568.8980 (1.14) 568.8980 (1.14) 568.8980 (1.14) 0.0000 (1.0) 568.8980 (1.14) 0.0000 (1.0) 0;0 1.7578 (0.88) 1 1
test_gc-gc[2.39.0] 578.0077 (1.16) 578.0077 (1.16) 578.0077 (1.16) 0.0000 (1.0) 578.0077 (1.16) 0.0000 (1.0) 0;0 1.7301 (0.87) 1 1
test_gc-gc[2.40.0] 589.2863 (1.18) 589.2863 (1.18) 589.2863 (1.18) 0.0000 (1.0) 589.2863 (1.18) 0.0000 (1.0) 0;0 1.6970 (0.85) 1 1
test_gc-gc[2.41.1] 597.6019 (1.19) 597.6019 (1.19) 597.6019 (1.19) 0.0000 (1.0) 597.6019 (1.19) 0.0000 (1.0) 0;0 1.6734 (0.84) 1 1
test_gc-gc[2.45.0] 584.8465 (1.17) 584.8465 (1.17) 584.8465 (1.17) 0.0000 (1.0) 584.8465 (1.17) 0.0000 (1.0) 0;0 1.7099 (0.86) 1 1
test_gc-gc[2.6.3] 500.1550 (1.0) 500.1550 (1.0) 500.1550 (1.0) 0.0000 (1.0) 500.1550 (1.0) 0.0000 (1.0) 0;0 1.9994 (1.0) 1 1
test_gc-gc[main] 632.3890 (1.26) 632.3890 (1.26) 632.3890 (1.26) 0.0000 (1.0) 632.3890 (1.26) 0.0000 (1.0) 0;0 1.5813 (0.79) 1 1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Legend:
Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
OPS: Operations Per Second, computed as 1 / Mean\n```
Does it follow sharing
use-case from the docs? Or is this something completely new?
I didn't realize this was intended to follow something from the docs. Where in the docs is it?
I was trying to cover some other common data sharing operations, like adding or deleting files from an existing output and garbage collecting.
Indeed, no trace of it in docs anymore, I guess it got refactored/thrown-out at some point.
Do you need all the previous steps to test what you want? If not, maybe it could be put into a smaller separate test? I'm worried that this might get unwieldy if we shove everything into one test.
Maybe separate tests for each command instead? Or is the whole scenario important here?
Tried to separate the use cases better, PTAL
---------------------------------------------------------------------------- benchmark 'test_gc-gc': 8 tests ----------------------------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_gc-gc[2.11.0] 446.0973 (1.0) 446.0973 (1.0) 446.0973 (1.0) 0.0000 (1.0) 446.0973 (1.0) 0.0000 (1.0) 0;0 2.2417 (1.0) 1 1
test_gc-gc[2.18.1] 458.9156 (1.03) 458.9156 (1.03) 458.9156 (1.03) 0.0000 (1.0) 458.9156 (1.03) 0.0000 (1.0) 0;0 2.1790 (0.97) 1 1
test_gc-gc[2.39.0] 458.2217 (1.03) 458.2217 (1.03) 458.2217 (1.03) 0.0000 (1.0) 458.2217 (1.03) 0.0000 (1.0) 0;0 2.1823 (0.97) 1 1
test_gc-gc[2.40.0] 526.2311 (1.18) 526.2311 (1.18) 526.2311 (1.18) 0.0000 (1.0) 526.2311 (1.18) 0.0000 (1.0) 0;0 1.9003 (0.85) 1 1
test_gc-gc[2.41.1] 511.0742 (1.15) 511.0742 (1.15) 511.0742 (1.15) 0.0000 (1.0) 511.0742 (1.15) 0.0000 (1.0) 0;0 1.9567 (0.87) 1 1
test_gc-gc[2.45.0] 499.0551 (1.12) 499.0551 (1.12) 499.0551 (1.12) 0.0000 (1.0) 499.0551 (1.12) 0.0000 (1.0) 0;0 2.0038 (0.89) 1 1
test_gc-gc[2.6.3] 517.5766 (1.16) 517.5766 (1.16) 517.5766 (1.16) 0.0000 (1.0) 517.5766 (1.16) 0.0000 (1.0) 0;0 1.9321 (0.86) 1 1
test_gc-gc[main] 543.3533 (1.22) 543.3533 (1.22) 543.3533 (1.22) 0.0000 (1.0) 543.3533 (1.22) 0.0000 (1.0) 0;0 1.8404 (0.82) 1 1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Legend:
Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
OPS: Operations Per Second, computed as 1 / Mean\n```
Why are there auto-generated comments but only for gc?
Looks like something broke in PR handling. Will take a look later.
Thank you!
Added some more operations to the test sharing use case to simulate some other scenarios like additions and deletions from a tracked directory and garbage collecting.
If we start using dvc-bench for https://github.com/iterative/dvc/issues/9108, this should be enough to cover the basic scenarios needed.
Fixes #407