search

ionelmc / pytest-benchmark

py.test fixture for benchmarking code
BSD 2-Clause "Simplified" License
1.22k stars 115 forks source link

compare-fail with different criteria per test or group #219

Open mortalisk opened 2 years ago

mortalisk commented 2 years ago

I have a setup where I am using --benchmark-compare-fail=mean:15%

I would like to specify a different percentage for each group of tests, because some tests have a varying performance, while others are more stable. Is it possible to specify this in code per test?

I want to run all tests in one go, as all test share a large run-time generated data-set and generating this takes most of the time of running the tests. Otherwise I could have just separated the tests in different runs, with different arguments.

ionelmc commented 2 years ago

Right now you could only do it via subsequent CLI compare commands (eg: you compare the exact set of benchmarks with the right fail check). You'd probably end up with some perhaps ugly shell script...

With that in mind, the comparison is implemented in specific classes - there could be a class that implements exactly what you want, see: https://github.com/ionelmc/pytest-benchmark/blob/4a99afe6e90119f52b1e00d249afe5acbdd4e7f0/src/pytest_benchmark/utils.py#L270-L288

But the argument parser doesn't know any of your subclasses, there's is no clear support for hooking your custom regression check. But if you'd like to use the subclassing route you could inject that via conftest either via one of the hooks (https://pytest-benchmark.readthedocs.io/en/latest/hooks.html) that pass the benchmarksession or a builtin pytest hook.

Eg: benchmarksession.compare_fail = MySpecialSubclass(myoptions)

Happy to elaborate more if this looks useful.

mortalisk commented 2 years ago

I could not figure out a way to use the CLI compare commands and make it report failure. Tried something like this:

pytest-benchmark compare Linux-CPython-3.6-64bit/0001_my-benchmark.json Linux-CPython-3.6-64bit/0002_my-benchmark.json fail=mean:1%

I manually edited one of the means to make sure it would fail. But it only prints the stats, and exits. How can I make it fail.

I think this would be a fine apprach if it would work.

Edit: Also, how do I specify a group or test name to compare?

ionelmc commented 2 years ago

fail=mean:1% is interpreted as a filter. Use --fail=mean:1%

mortalisk commented 2 years ago

py.test-benchmark: error: unrecognized arguments: --fail=mean:1%

ionelmc commented 2 years ago

Oh damn, look like I haven't implemented that in the standalone CLI yet...

ionelmc commented 2 years ago

I guess patching stuff is the only route right now.... ooor I implement something for your usecase. Give me more details, I want to understand it more. What do you mean some tests are "more stable"?

mortalisk commented 2 years ago

There are a few tests that can vary quite a bit in how long time it takes to execute. For those tests, I want to specify e.g. 50% variance in the mean to cause a failure. For other tests they typically don't vary more than 15%, so I want to report a failure if they go above this.

Another case is that I would like to be able to update the baseline for these tests independently, so that I do not have to update everything when only some tests are failing (or after having improved one piece of code)

I have a plan to make it work, by separating the runs of the different tests and using a separate mechanism to share my generated data. Then specifying a different storage name for each set of files, hoping pytest-benchmark will then compare against the correct filename based on the supplied --benchmark-save parameter. The tests are running, so we'll see if it works.

If you would have implemented something for my usecase directly, I guess it would be to be able to specify the fail criteria in a decorator per test, perhaps. Or alternatively add several fail criteria, where one could specify group and/or test name. But if my current approach works, I think it might be the preferred one for me specifically, although I can see that might become too cumbersome for some setups.

Another good approach would be if the CLI compare would support both filtering tests based on names/groups, and also to allow fail checks to be made.

SchoolGuy commented 3 months ago

I would like to give this a bump. The performance of my tests varies so greatly that without it I will only be able to use the standard deviation as a comparison target but won't be able to set hard limits per application feature. See gh/cobbler/cobbler for the project if more context is needed (inside tests/performance you will be able to spot the performance testsuite).

Click me ``` ------------------------------------------------------------------------------------------------------------ benchmark: 160 tests ------------------------------------------------------------------------------------------------------------ Name (time in us) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- test_start[False-False] 7.1430 (1.0) 7.1430 (1.0) 7.1430 (1.0) 0.0000 (1.0) 7.1430 (1.0) 0.0000 (1.0) 0;0 139,997.2011 (1.0) 1 1 test_start[True-False] 7.2140 (1.01) 7.2140 (1.01) 7.2140 (1.01) 0.0000 (1.0) 7.2140 (1.01) 0.0000 (1.0) 0;0 138,619.3536 (0.99) 1 1 test_start[False-True] 7.7540 (1.09) 7.7540 (1.09) 7.7540 (1.09) 0.0000 (1.0) 7.7540 (1.09) 0.0000 (1.0) 0;0 128,965.6976 (0.92) 1 1 test_start[True-True] 8.0440 (1.13) 8.0440 (1.13) 8.0440 (1.13) 0.0000 (1.0) 8.0440 (1.13) 0.0000 (1.0) 0;0 124,316.2610 (0.89) 1 1 test_item_edit[False-True-system-False] 23.3230 (3.27) 23.3230 (3.27) 23.3230 (3.27) 0.0000 (1.0) 23.3230 (3.27) 0.0000 (1.0) 0;0 42,876.1308 (0.31) 1 1 test_item_edit[False-False-system-False] 23.9250 (3.35) 23.9250 (3.35) 23.9250 (3.35) 0.0000 (1.0) 23.9250 (3.35) 0.0000 (1.0) 0;0 41,797.2831 (0.30) 1 1 test_item_edit[False-False-image-False] 24.1050 (3.37) 24.1050 (3.37) 24.1050 (3.37) 0.0000 (1.0) 24.1050 (3.37) 0.0000 (1.0) 0;0 41,485.1691 (0.30) 1 1 test_item_edit[False-False-repo-False] 26.4290 (3.70) 26.4290 (3.70) 26.4290 (3.70) 0.0000 (1.0) 26.4290 (3.70) 0.0000 (1.0) 0;0 37,837.2242 (0.27) 1 1 test_item_edit[False-True-image-False] 28.6940 (4.02) 28.6940 (4.02) 28.6940 (4.02) 0.0000 (1.0) 28.6940 (4.02) 0.0000 (1.0) 0;0 34,850.4913 (0.25) 1 1 test_item_edit[False-True-distro-False] 29.3150 (4.10) 29.3150 (4.10) 29.3150 (4.10) 0.0000 (1.0) 29.3150 (4.10) 0.0000 (1.0) 0;0 34,112.2291 (0.24) 1 1 test_item_edit[False-False-distro-False] 29.9060 (4.19) 29.9060 (4.19) 29.9060 (4.19) 0.0000 (1.0) 29.9060 (4.19) 0.0000 (1.0) 0;0 33,438.1061 (0.24) 1 1 test_item_edit[False-True-repo-False] 31.0380 (4.35) 31.0380 (4.35) 31.0380 (4.35) 0.0000 (1.0) 31.0380 (4.35) 0.0000 (1.0) 0;0 32,218.5708 (0.23) 1 1 test_item_edit[False-False-menu-False] 32.7810 (4.59) 32.7810 (4.59) 32.7810 (4.59) 0.0000 (1.0) 32.7810 (4.59) 0.0000 (1.0) 0;0 30,505.4758 (0.22) 1 1 test_item_edit[False-False-profile-False] 33.7430 (4.72) 33.7430 (4.72) 33.7430 (4.72) 0.0000 (1.0) 33.7430 (4.72) 0.0000 (1.0) 0;0 29,635.7763 (0.21) 1 1 test_item_edit[False-True-menu-False] 35.1450 (4.92) 35.1450 (4.92) 35.1450 (4.92) 0.0000 (1.0) 35.1450 (4.92) 0.0000 (1.0) 0;0 28,453.5495 (0.20) 1 1 test_item_edit[False-True-profile-False] 36.7390 (5.14) 36.7390 (5.14) 36.7390 (5.14) 0.0000 (1.0) 36.7390 (5.14) 0.0000 (1.0) 0;0 27,219.0315 (0.19) 1 1 test_item_edit[True-False-image-False] 37.2600 (5.22) 37.2600 (5.22) 37.2600 (5.22) 0.0000 (1.0) 37.2600 (5.22) 0.0000 (1.0) 0;0 26,838.4326 (0.19) 1 1 test_item_edit[True-False-repo-False] 41.8280 (5.86) 41.8280 (5.86) 41.8280 (5.86) 0.0000 (1.0) 41.8280 (5.86) 0.0000 (1.0) 0;0 23,907.4304 (0.17) 1 1 test_item_edit[True-False-system-False] 42.1190 (5.90) 42.1190 (5.90) 42.1190 (5.90) 0.0000 (1.0) 42.1190 (5.90) 0.0000 (1.0) 0;0 23,742.2541 (0.17) 1 1 test_item_edit[True-False-distro-False] 43.4610 (6.08) 43.4610 (6.08) 43.4610 (6.08) 0.0000 (1.0) 43.4610 (6.08) 0.0000 (1.0) 0;0 23,009.1346 (0.16) 1 1 test_item_edit[True-True-image-False] 43.6610 (6.11) 43.6610 (6.11) 43.6610 (6.11) 0.0000 (1.0) 43.6610 (6.11) 0.0000 (1.0) 0;0 22,903.7356 (0.16) 1 1 test_item_edit[True-True-repo-False] 45.9460 (6.43) 45.9460 (6.43) 45.9460 (6.43) 0.0000 (1.0) 45.9460 (6.43) 0.0000 (1.0) 0;0 21,764.6803 (0.16) 1 1 test_item_edit[True-True-distro-False] 46.5670 (6.52) 46.5670 (6.52) 46.5670 (6.52) 0.0000 (1.0) 46.5670 (6.52) 0.0000 (1.0) 0;0 21,474.4347 (0.15) 1 1 test_item_edit[True-True-system-False] 48.5010 (6.79) 48.5010 (6.79) 48.5010 (6.79) 0.0000 (1.0) 48.5010 (6.79) 0.0000 (1.0) 0;0 20,618.1316 (0.15) 1 1 test_item_edit[True-False-menu-False] 56.8960 (7.97) 56.8960 (7.97) 56.8960 (7.97) 0.0000 (1.0) 56.8960 (7.97) 0.0000 (1.0) 0;0 17,575.9280 (0.13) 1 1 test_item_edit[False-False-system-True] 58.7500 (8.22) 58.7500 (8.22) 58.7500 (8.22) 0.0000 (1.0) 58.7500 (8.22) 0.0000 (1.0) 0;0 17,021.2766 (0.12) 1 1 test_item_edit[True-True-menu-False] 60.6330 (8.49) 60.6330 (8.49) 60.6330 (8.49) 0.0000 (1.0) 60.6330 (8.49) 0.0000 (1.0) 0;0 16,492.6690 (0.12) 1 1 test_item_edit[False-True-repo-True] 62.7970 (8.79) 62.7970 (8.79) 62.7970 (8.79) 0.0000 (1.0) 62.7970 (8.79) 0.0000 (1.0) 0;0 15,924.3276 (0.11) 1 1 test_item_edit[False-True-system-True] 63.0570 (8.83) 63.0570 (8.83) 63.0570 (8.83) 0.0000 (1.0) 63.0570 (8.83) 0.0000 (1.0) 0;0 15,858.6675 (0.11) 1 1 test_item_edit[False-True-distro-True] 64.4700 (9.03) 64.4700 (9.03) 64.4700 (9.03) 0.0000 (1.0) 64.4700 (9.03) 0.0000 (1.0) 0;0 15,511.0904 (0.11) 1 1 test_item_edit[True-True-profile-False] 64.6710 (9.05) 64.6710 (9.05) 64.6710 (9.05) 0.0000 (1.0) 64.6710 (9.05) 0.0000 (1.0) 0;0 15,462.8813 (0.11) 1 1 test_item_edit[True-False-profile-False] 65.6630 (9.19) 65.6630 (9.19) 65.6630 (9.19) 0.0000 (1.0) 65.6630 (9.19) 0.0000 (1.0) 0;0 15,229.2768 (0.11) 1 1 test_item_edit[False-False-image-True] 66.7840 (9.35) 66.7840 (9.35) 66.7840 (9.35) 0.0000 (1.0) 66.7840 (9.35) 0.0000 (1.0) 0;0 14,973.6464 (0.11) 1 1 test_item_edit[False-False-distro-True] 71.7440 (10.04) 71.7440 (10.04) 71.7440 (10.04) 0.0000 (1.0) 71.7440 (10.04) 0.0000 (1.0) 0;0 13,938.4478 (0.10) 1 1 test_item_edit[False-True-image-True] 73.0770 (10.23) 73.0770 (10.23) 73.0770 (10.23) 0.0000 (1.0) 73.0770 (10.23) 0.0000 (1.0) 0;0 13,684.1961 (0.10) 1 1 test_item_edit[False-False-repo-True] 87.5240 (12.25) 87.5240 (12.25) 87.5240 (12.25) 0.0000 (1.0) 87.5240 (12.25) 0.0000 (1.0) 0;0 11,425.4376 (0.08) 1 1 test_item_edit[True-False-system-True] 98.0030 (13.72) 98.0030 (13.72) 98.0030 (13.72) 0.0000 (1.0) 98.0030 (13.72) 0.0000 (1.0) 0;0 10,203.7693 (0.07) 1 1 test_item_edit[False-True-menu-True] 101.8000 (14.25) 101.8000 (14.25) 101.8000 (14.25) 0.0000 (1.0) 101.8000 (14.25) 0.0000 (1.0) 0;0 9,823.1827 (0.07) 1 1 test_item_edit[False-True-profile-True] 102.8020 (14.39) 102.8020 (14.39) 102.8020 (14.39) 0.0000 (1.0) 102.8020 (14.39) 0.0000 (1.0) 0;0 9,727.4372 (0.07) 1 1 test_item_edit[True-True-system-True] 113.4910 (15.89) 113.4910 (15.89) 113.4910 (15.89) 0.0000 (1.0) 113.4910 (15.89) 0.0000 (1.0) 0;0 8,811.2714 (0.06) 1 1 test_item_edit[False-False-menu-True] 115.7960 (16.21) 115.7960 (16.21) 115.7960 (16.21) 0.0000 (1.0) 115.7960 (16.21) 0.0000 (1.0) 0;0 8,635.8769 (0.06) 1 1 test_item_edit[False-False-profile-True] 117.5400 (16.46) 117.5400 (16.46) 117.5400 (16.46) 0.0000 (1.0) 117.5400 (16.46) 0.0000 (1.0) 0;0 8,507.7420 (0.06) 1 1 test_item_edit[True-False-image-True] 152.0440 (21.29) 152.0440 (21.29) 152.0440 (21.29) 0.0000 (1.0) 152.0440 (21.29) 0.0000 (1.0) 0;0 6,577.0435 (0.05) 1 1 test_item_edit[True-True-image-True] 156.6230 (21.93) 156.6230 (21.93) 156.6230 (21.93) 0.0000 (1.0) 156.6230 (21.93) 0.0000 (1.0) 0;0 6,384.7583 (0.05) 1 1 test_item_edit[True-True-repo-True] 293.4380 (41.08) 293.4380 (41.08) 293.4380 (41.08) 0.0000 (1.0) 293.4380 (41.08) 0.0000 (1.0) 0;0 3,407.8749 (0.02) 1 1 test_item_edit[True-False-repo-True] 314.9180 (44.09) 314.9180 (44.09) 314.9180 (44.09) 0.0000 (1.0) 314.9180 (44.09) 0.0000 (1.0) 0;0 3,175.4298 (0.02) 1 1 test_item_remove[True-False-repo] 604.5590 (84.64) 604.5590 (84.64) 604.5590 (84.64) 0.0000 (1.0) 604.5590 (84.64) 0.0000 (1.0) 0;0 1,654.0983 (0.01) 1 1 test_item_edit[True-False-distro-True] 615.2290 (86.13) 615.2290 (86.13) 615.2290 (86.13) 0.0000 (1.0) 615.2290 (86.13) 0.0000 (1.0) 0;0 1,625.4110 (0.01) 1 1 test_item_remove[False-True-repo] 641.8590 (89.86) 641.8590 (89.86) 641.8590 (89.86) 0.0000 (1.0) 641.8590 (89.86) 0.0000 (1.0) 0;0 1,557.9746 (0.01) 1 1 test_item_remove[True-True-repo] 646.7240 (90.54) 646.7240 (90.54) 646.7240 (90.54) 0.0000 (1.0) 646.7240 (90.54) 0.0000 (1.0) 0;0 1,546.2547 (0.01) 1 1 test_item_edit[True-True-distro-True] 656.4560 (91.90) 656.4560 (91.90) 656.4560 (91.90) 0.0000 (1.0) 656.4560 (91.90) 0.0000 (1.0) 0;0 1,523.3313 (0.01) 1 1 test_item_remove[False-False-repo] 664.5510 (93.04) 664.5510 (93.04) 664.5510 (93.04) 0.0000 (1.0) 664.5510 (93.04) 0.0000 (1.0) 0;0 1,504.7754 (0.01) 1 1 test_repos_create[cache_enabled1] 822.5350 (115.15) 822.5350 (115.15) 822.5350 (115.15) 0.0000 (1.0) 822.5350 (115.15) 0.0000 (1.0) 0;0 1,215.7537 (0.01) 1 1 test_repos_create[cache_enabled0] 893.9990 (125.16) 893.9990 (125.16) 893.9990 (125.16) 0.0000 (1.0) 893.9990 (125.16) 0.0000 (1.0) 0;0 1,118.5695 (0.01) 1 1 test_item_edit[True-True-profile-True] 1,073.7840 (150.33) 1,073.7840 (150.33) 1,073.7840 (150.33) 0.0000 (1.0) 1,073.7840 (150.33) 0.0000 (1.0) 0;0 931.2860 (0.01) 1 1 test_item_edit[True-False-profile-True] 1,079.1350 (151.08) 1,079.1350 (151.08) 1,079.1350 (151.08) 0.0000 (1.0) 1,079.1350 (151.08) 0.0000 (1.0) 0;0 926.6681 (0.01) 1 1 test_item_rename[True-True-system] 1,145.2640 (160.33) 1,145.2640 (160.33) 1,145.2640 (160.33) 0.0000 (1.0) 1,145.2640 (160.33) 0.0000 (1.0) 0;0 873.1611 (0.01) 1 1 test_item_rename[False-True-system] 1,182.0440 (165.48) 1,182.0440 (165.48) 1,182.0440 (165.48) 0.0000 (1.0) 1,182.0440 (165.48) 0.0000 (1.0) 0;0 845.9922 (0.01) 1 1 test_item_rename[True-False-system] 1,326.5700 (185.72) 1,326.5700 (185.72) 1,326.5700 (185.72) 0.0000 (1.0) 1,326.5700 (185.72) 0.0000 (1.0) 0;0 753.8238 (0.01) 1 1 test_item_copy[True-False-repo] 1,353.5160 (189.49) 1,353.5160 (189.49) 1,353.5160 (189.49) 0.0000 (1.0) 1,353.5160 (189.49) 0.0000 (1.0) 0;0 738.8165 (0.01) 1 1 test_item_copy[False-False-repo] 1,362.8340 (190.79) 1,362.8340 (190.79) 1,362.8340 (190.79) 0.0000 (1.0) 1,362.8340 (190.79) 0.0000 (1.0) 0;0 733.7651 (0.01) 1 1 test_item_rename[True-False-repo] 1,398.6840 (195.81) 1,398.6840 (195.81) 1,398.6840 (195.81) 0.0000 (1.0) 1,398.6840 (195.81) 0.0000 (1.0) 0;0 714.9578 (0.01) 1 1 test_item_copy[True-True-repo] 1,405.6830 (196.79) 1,405.6830 (196.79) 1,405.6830 (196.79) 0.0000 (1.0) 1,405.6830 (196.79) 0.0000 (1.0) 0;0 711.3979 (0.01) 1 1 test_item_copy[False-True-repo] 1,433.0840 (200.63) 1,433.0840 (200.63) 1,433.0840 (200.63) 0.0000 (1.0) 1,433.0840 (200.63) 0.0000 (1.0) 0;0 697.7958 (0.00) 1 1 test_item_rename[False-False-system] 1,498.7120 (209.82) 1,498.7120 (209.82) 1,498.7120 (209.82) 0.0000 (1.0) 1,498.7120 (209.82) 0.0000 (1.0) 0;0 667.2396 (0.00) 1 1 test_item_rename[False-True-repo] 1,670.4300 (233.86) 1,670.4300 (233.86) 1,670.4300 (233.86) 0.0000 (1.0) 1,670.4300 (233.86) 0.0000 (1.0) 0;0 598.6483 (0.00) 1 1 test_item_remove[False-True-system] 1,773.1100 (248.23) 1,773.1100 (248.23) 1,773.1100 (248.23) 0.0000 (1.0) 1,773.1100 (248.23) 0.0000 (1.0) 0;0 563.9808 (0.00) 1 1 test_item_remove[True-True-system] 1,787.9180 (250.30) 1,787.9180 (250.30) 1,787.9180 (250.30) 0.0000 (1.0) 1,787.9180 (250.30) 0.0000 (1.0) 0;0 559.3098 (0.00) 1 1 test_item_remove[False-False-system] 1,840.4380 (257.66) 1,840.4380 (257.66) 1,840.4380 (257.66) 0.0000 (1.0) 1,840.4380 (257.66) 0.0000 (1.0) 0;0 543.3489 (0.00) 1 1 test_item_rename[True-True-repo] 1,845.9170 (258.42) 1,845.9170 (258.42) 1,845.9170 (258.42) 0.0000 (1.0) 1,845.9170 (258.42) 0.0000 (1.0) 0;0 541.7362 (0.00) 1 1 test_item_rename[False-False-repo] 1,851.8680 (259.26) 1,851.8680 (259.26) 1,851.8680 (259.26) 0.0000 (1.0) 1,851.8680 (259.26) 0.0000 (1.0) 0;0 539.9953 (0.00) 1 1 test_item_remove[True-False-system] 1,942.3560 (271.92) 1,942.3560 (271.92) 1,942.3560 (271.92) 0.0000 (1.0) 1,942.3560 (271.92) 0.0000 (1.0) 0;0 514.8387 (0.00) 1 1 test_menus_create[cache_enabled0] 2,139.6530 (299.55) 2,139.6530 (299.55) 2,139.6530 (299.55) 0.0000 (1.0) 2,139.6530 (299.55) 0.0000 (1.0) 0;0 467.3655 (0.00) 1 1 test_item_remove[True-False-image] 2,140.7470 (299.70) 2,140.7470 (299.70) 2,140.7470 (299.70) 0.0000 (1.0) 2,140.7470 (299.70) 0.0000 (1.0) 0;0 467.1267 (0.00) 1 1 test_item_remove[True-True-image] 2,194.6260 (307.24) 2,194.6260 (307.24) 2,194.6260 (307.24) 0.0000 (1.0) 2,194.6260 (307.24) 0.0000 (1.0) 0;0 455.6585 (0.00) 1 1 test_item_remove[False-False-image] 2,393.4990 (335.08) 2,393.4990 (335.08) 2,393.4990 (335.08) 0.0000 (1.0) 2,393.4990 (335.08) 0.0000 (1.0) 0;0 417.7984 (0.00) 1 1 test_item_remove[False-True-image] 2,473.3880 (346.27) 2,473.3880 (346.27) 2,473.3880 (346.27) 0.0000 (1.0) 2,473.3880 (346.27) 0.0000 (1.0) 0;0 404.3037 (0.00) 1 1 test_menus_create[cache_enabled1] 2,536.8240 (355.15) 2,536.8240 (355.15) 2,536.8240 (355.15) 0.0000 (1.0) 2,536.8240 (355.15) 0.0000 (1.0) 0;0 394.1937 (0.00) 1 1 test_item_edit[True-False-menu-True] 2,714.6870 (380.05) 2,714.6870 (380.05) 2,714.6870 (380.05) 0.0000 (1.0) 2,714.6870 (380.05) 0.0000 (1.0) 0;0 368.3666 (0.00) 1 1 test_item_edit[True-True-menu-True] 2,771.6440 (388.02) 2,771.6440 (388.02) 2,771.6440 (388.02) 0.0000 (1.0) 2,771.6440 (388.02) 0.0000 (1.0) 0;0 360.7967 (0.00) 1 1 test_item_copy[True-False-menu] 2,923.6470 (409.30) 2,923.6470 (409.30) 2,923.6470 (409.30) 0.0000 (1.0) 2,923.6470 (409.30) 0.0000 (1.0) 0;0 342.0386 (0.00) 1 1 test_item_copy[False-False-menu] 2,976.8660 (416.75) 2,976.8660 (416.75) 2,976.8660 (416.75) 0.0000 (1.0) 2,976.8660 (416.75) 0.0000 (1.0) 0;0 335.9238 (0.00) 1 1 test_item_copy[False-True-menu] 2,981.9060 (417.46) 2,981.9060 (417.46) 2,981.9060 (417.46) 0.0000 (1.0) 2,981.9060 (417.46) 0.0000 (1.0) 0;0 335.3560 (0.00) 1 1 test_item_copy[True-True-menu] 3,054.9810 (427.69) 3,054.9810 (427.69) 3,054.9810 (427.69) 0.0000 (1.0) 3,054.9810 (427.69) 0.0000 (1.0) 0;0 327.3343 (0.00) 1 1 test_deserialize[True-False] 4,580.0980 (641.20) 4,580.0980 (641.20) 4,580.0980 (641.20) 0.0000 (1.0) 4,580.0980 (641.20) 0.0000 (1.0) 0;0 218.3359 (0.00) 1 1 test_deserialize[True-True] 4,737.5820 (663.25) 4,737.5820 (663.25) 4,737.5820 (663.25) 0.0000 (1.0) 4,737.5820 (663.25) 0.0000 (1.0) 0;0 211.0781 (0.00) 1 1 test_deserialize[False-True] 4,751.3270 (665.17) 4,751.3270 (665.17) 4,751.3270 (665.17) 0.0000 (1.0) 4,751.3270 (665.17) 0.0000 (1.0) 0;0 210.4675 (0.00) 1 1 test_deserialize[False-False] 4,812.9720 (673.80) 4,812.9720 (673.80) 4,812.9720 (673.80) 0.0000 (1.0) 4,812.9720 (673.80) 0.0000 (1.0) 0;0 207.7718 (0.00) 1 1 test_distros_create[cache_enabled1] 7,210.0350 (>1000.0) 7,210.0350 (>1000.0) 7,210.0350 (>1000.0) 0.0000 (1.0) 7,210.0350 (>1000.0) 0.0000 (1.0) 0;0 138.6956 (0.00) 1 1 test_distros_create[cache_enabled0] 7,538.3660 (>1000.0) 7,538.3660 (>1000.0) 7,538.3660 (>1000.0) 0.0000 (1.0) 7,538.3660 (>1000.0) 0.0000 (1.0) 0;0 132.6547 (0.00) 1 1 test_get_autoinstall[True-system] 7,647.5720 (>1000.0) 7,647.5720 (>1000.0) 7,647.5720 (>1000.0) 0.0000 (1.0) 7,647.5720 (>1000.0) 0.0000 (1.0) 0;0 130.7605 (0.00) 1 1 test_get_autoinstall[False-system] 8,235.3600 (>1000.0) 8,235.3600 (>1000.0) 8,235.3600 (>1000.0) 0.0000 (1.0) 8,235.3600 (>1000.0) 0.0000 (1.0) 0;0 121.4276 (0.00) 1 1 test_make_pxe_menu[True-False] 11,934.0220 (>1000.0) 11,934.0220 (>1000.0) 11,934.0220 (>1000.0) 0.0000 (1.0) 11,934.0220 (>1000.0) 0.0000 (1.0) 0;0 83.7940 (0.00) 1 1 test_images_create[cache_enabled0] 21,258.0330 (>1000.0) 21,258.0330 (>1000.0) 21,258.0330 (>1000.0) 0.0000 (1.0) 21,258.0330 (>1000.0) 0.0000 (1.0) 0;0 47.0410 (0.00) 1 1 test_images_create[cache_enabled1] 21,258.9360 (>1000.0) 21,258.9360 (>1000.0) 21,258.9360 (>1000.0) 0.0000 (1.0) 21,258.9360 (>1000.0) 0.0000 (1.0) 0;0 47.0390 (0.00) 1 1 test_item_copy[True-False-image] 25,925.5660 (>1000.0) 25,925.5660 (>1000.0) 25,925.5660 (>1000.0) 0.0000 (1.0) 25,925.5660 (>1000.0) 0.0000 (1.0) 0;0 38.5720 (0.00) 1 1 test_item_rename[True-False-distro] 29,194.4120 (>1000.0) 29,194.4120 (>1000.0) 29,194.4120 (>1000.0) 0.0000 (1.0) 29,194.4120 (>1000.0) 0.0000 (1.0) 0;0 34.2531 (0.00) 1 1 test_make_pxe_menu[False-False] 35,356.9040 (>1000.0) 35,356.9040 (>1000.0) 35,356.9040 (>1000.0) 0.0000 (1.0) 35,356.9040 (>1000.0) 0.0000 (1.0) 0;0 28.2830 (0.00) 1 1 test_item_copy[True-False-distro] 35,397.9750 (>1000.0) 35,397.9750 (>1000.0) 35,397.9750 (>1000.0) 0.0000 (1.0) 35,397.9750 (>1000.0) 0.0000 (1.0) 0;0 28.2502 (0.00) 1 1 test_get_autoinstall[True-profile] 36,086.1250 (>1000.0) 36,086.1250 (>1000.0) 36,086.1250 (>1000.0) 0.0000 (1.0) 36,086.1250 (>1000.0) 0.0000 (1.0) 0;0 27.7115 (0.00) 1 1 test_make_pxe_menu[True-True] 53,082.2700 (>1000.0) 53,082.2700 (>1000.0) 53,082.2700 (>1000.0) 0.0000 (1.0) 53,082.2700 (>1000.0) 0.0000 (1.0) 0;0 18.8387 (0.00) 1 1 test_item_remove[True-False-profile] 68,524.4470 (>1000.0) 68,524.4470 (>1000.0) 68,524.4470 (>1000.0) 0.0000 (1.0) 68,524.4470 (>1000.0) 0.0000 (1.0) 0;0 14.5933 (0.00) 1 1 test_item_remove[True-False-distro] 70,548.9290 (>1000.0) 70,548.9290 (>1000.0) 70,548.9290 (>1000.0) 0.0000 (1.0) 70,548.9290 (>1000.0) 0.0000 (1.0) 0;0 14.1746 (0.00) 1 1 test_profiles_create[True-False] 73,345.8290 (>1000.0) 73,345.8290 (>1000.0) 73,345.8290 (>1000.0) 0.0000 (1.0) 73,345.8290 (>1000.0) 0.0000 (1.0) 0;0 13.6340 (0.00) 1 1 test_item_copy[False-False-distro] 74,969.2230 (>1000.0) 74,969.2230 (>1000.0) 74,969.2230 (>1000.0) 0.0000 (1.0) 74,969.2230 (>1000.0) 0.0000 (1.0) 0;0 13.3388 (0.00) 1 1 test_item_copy[False-False-image] 78,697.1570 (>1000.0) 78,697.1570 (>1000.0) 78,697.1570 (>1000.0) 0.0000 (1.0) 78,697.1570 (>1000.0) 0.0000 (1.0) 0;0 12.7069 (0.00) 1 1 test_item_rename[False-False-distro] 80,645.4420 (>1000.0) 80,645.4420 (>1000.0) 80,645.4420 (>1000.0) 0.0000 (1.0) 80,645.4420 (>1000.0) 0.0000 (1.0) 0;0 12.4000 (0.00) 1 1 test_item_remove[True-False-menu] 81,359.2980 (>1000.0) 81,359.2980 (>1000.0) 81,359.2980 (>1000.0) 0.0000 (1.0) 81,359.2980 (>1000.0) 0.0000 (1.0) 0;0 12.2912 (0.00) 1 1 test_item_copy[True-False-profile] 84,188.3180 (>1000.0) 84,188.3180 (>1000.0) 84,188.3180 (>1000.0) 0.0000 (1.0) 84,188.3180 (>1000.0) 0.0000 (1.0) 0;0 11.8781 (0.00) 1 1 test_make_pxe_menu[False-True] 88,408.6370 (>1000.0) 88,408.6370 (>1000.0) 88,408.6370 (>1000.0) 0.0000 (1.0) 88,408.6370 (>1000.0) 0.0000 (1.0) 0;0 11.3111 (0.00) 1 1 test_get_autoinstall[False-profile] 104,560.0840 (>1000.0) 104,560.0840 (>1000.0) 104,560.0840 (>1000.0) 0.0000 (1.0) 104,560.0840 (>1000.0) 0.0000 (1.0) 0;0 9.5639 (0.00) 1 1 test_item_copy[True-True-image] 106,843.1370 (>1000.0) 106,843.1370 (>1000.0) 106,843.1370 (>1000.0) 0.0000 (1.0) 106,843.1370 (>1000.0) 0.0000 (1.0) 0;0 9.3595 (0.00) 1 1 test_item_copy[True-True-distro] 108,777.7520 (>1000.0) 108,777.7520 (>1000.0) 108,777.7520 (>1000.0) 0.0000 (1.0) 108,777.7520 (>1000.0) 0.0000 (1.0) 0;0 9.1931 (0.00) 1 1 test_item_rename[True-False-menu] 112,138.5860 (>1000.0) 112,138.5860 (>1000.0) 112,138.5860 (>1000.0) 0.0000 (1.0) 112,138.5860 (>1000.0) 0.0000 (1.0) 0;0 8.9175 (0.00) 1 1 test_item_rename[True-True-distro] 128,841.5200 (>1000.0) 128,841.5200 (>1000.0) 128,841.5200 (>1000.0) 0.0000 (1.0) 128,841.5200 (>1000.0) 0.0000 (1.0) 0;0 7.7615 (0.00) 1 1 test_profiles_create[False-False] 139,035.0290 (>1000.0) 139,035.0290 (>1000.0) 139,035.0290 (>1000.0) 0.0000 (1.0) 139,035.0290 (>1000.0) 0.0000 (1.0) 0;0 7.1924 (0.00) 1 1 test_item_remove[False-False-profile] 156,284.8080 (>1000.0) 156,284.8080 (>1000.0) 156,284.8080 (>1000.0) 0.0000 (1.0) 156,284.8080 (>1000.0) 0.0000 (1.0) 0;0 6.3986 (0.00) 1 1 test_item_remove[False-False-distro] 158,001.7330 (>1000.0) 158,001.7330 (>1000.0) 158,001.7330 (>1000.0) 0.0000 (1.0) 158,001.7330 (>1000.0) 0.0000 (1.0) 0;0 6.3290 (0.00) 1 1 test_item_remove[True-True-distro] 168,920.5910 (>1000.0) 168,920.5910 (>1000.0) 168,920.5910 (>1000.0) 0.0000 (1.0) 168,920.5910 (>1000.0) 0.0000 (1.0) 0;0 5.9199 (0.00) 1 1 test_item_remove[True-True-profile] 173,599.4850 (>1000.0) 173,599.4850 (>1000.0) 173,599.4850 (>1000.0) 0.0000 (1.0) 173,599.4850 (>1000.0) 0.0000 (1.0) 0;0 5.7604 (0.00) 1 1 test_item_copy[False-True-distro] 185,958.6040 (>1000.0) 185,958.6040 (>1000.0) 185,958.6040 (>1000.0) 0.0000 (1.0) 185,958.6040 (>1000.0) 0.0000 (1.0) 0;0 5.3775 (0.00) 1 1 test_item_remove[False-False-menu] 186,208.0460 (>1000.0) 186,208.0460 (>1000.0) 186,208.0460 (>1000.0) 0.0000 (1.0) 186,208.0460 (>1000.0) 0.0000 (1.0) 0;0 5.3703 (0.00) 1 1 test_item_copy[False-True-image] 187,718.8610 (>1000.0) 187,718.8610 (>1000.0) 187,718.8610 (>1000.0) 0.0000 (1.0) 187,718.8610 (>1000.0) 0.0000 (1.0) 0;0 5.3271 (0.00) 1 1 test_profiles_create[True-True] 207,943.8880 (>1000.0) 207,943.8880 (>1000.0) 207,943.8880 (>1000.0) 0.0000 (1.0) 207,943.8880 (>1000.0) 0.0000 (1.0) 0;0 4.8090 (0.00) 1 1 test_item_rename[False-True-distro] 238,275.4390 (>1000.0) 238,275.4390 (>1000.0) 238,275.4390 (>1000.0) 0.0000 (1.0) 238,275.4390 (>1000.0) 0.0000 (1.0) 0;0 4.1968 (0.00) 1 1 test_item_copy[False-False-profile] 282,182.7770 (>1000.0) 282,182.7770 (>1000.0) 282,182.7770 (>1000.0) 0.0000 (1.0) 282,182.7770 (>1000.0) 0.0000 (1.0) 0;0 3.5438 (0.00) 1 1 test_item_remove[True-True-menu] 282,559.4790 (>1000.0) 282,559.4790 (>1000.0) 282,559.4790 (>1000.0) 0.0000 (1.0) 282,559.4790 (>1000.0) 0.0000 (1.0) 0;0 3.5391 (0.00) 1 1 test_item_remove[False-True-distro] 285,305.7900 (>1000.0) 285,305.7900 (>1000.0) 285,305.7900 (>1000.0) 0.0000 (1.0) 285,305.7900 (>1000.0) 0.0000 (1.0) 0;0 3.5050 (0.00) 1 1 test_item_remove[False-True-profile] 291,133.9220 (>1000.0) 291,133.9220 (>1000.0) 291,133.9220 (>1000.0) 0.0000 (1.0) 291,133.9220 (>1000.0) 0.0000 (1.0) 0;0 3.4348 (0.00) 1 1 test_item_rename[False-False-menu] 311,165.8130 (>1000.0) 311,165.8130 (>1000.0) 311,165.8130 (>1000.0) 0.0000 (1.0) 311,165.8130 (>1000.0) 0.0000 (1.0) 0;0 3.2137 (0.00) 1 1 test_profiles_create[False-True] 340,849.4180 (>1000.0) 340,849.4180 (>1000.0) 340,849.4180 (>1000.0) 0.0000 (1.0) 340,849.4180 (>1000.0) 0.0000 (1.0) 0;0 2.9338 (0.00) 1 1 test_item_copy[True-True-profile] 428,460.9790 (>1000.0) 428,460.9790 (>1000.0) 428,460.9790 (>1000.0) 0.0000 (1.0) 428,460.9790 (>1000.0) 0.0000 (1.0) 0;0 2.3339 (0.00) 1 1 test_item_remove[False-True-menu] 446,046.5910 (>1000.0) 446,046.5910 (>1000.0) 446,046.5910 (>1000.0) 0.0000 (1.0) 446,046.5910 (>1000.0) 0.0000 (1.0) 0;0 2.2419 (0.00) 1 1 test_item_rename[True-True-menu] 470,827.3540 (>1000.0) 470,827.3540 (>1000.0) 470,827.3540 (>1000.0) 0.0000 (1.0) 470,827.3540 (>1000.0) 0.0000 (1.0) 0;0 2.1239 (0.00) 1 1 test_item_copy[False-True-profile] 747,877.8660 (>1000.0) 747,877.8660 (>1000.0) 747,877.8660 (>1000.0) 0.0000 (1.0) 747,877.8660 (>1000.0) 0.0000 (1.0) 0;0 1.3371 (0.00) 1 1 test_item_rename[False-True-menu] 829,675.2850 (>1000.0) 829,675.2850 (>1000.0) 829,675.2850 (>1000.0) 0.0000 (1.0) 829,675.2850 (>1000.0) 0.0000 (1.0) 0;0 1.2053 (0.00) 1 1 test_item_rename[True-False-image] 1,087,750.6020 (>1000.0) 1,087,750.6020 (>1000.0) 1,087,750.6020 (>1000.0) 0.0000 (1.0) 1,087,750.6020 (>1000.0) 0.0000 (1.0) 0;0 0.9193 (0.00) 1 1 test_item_rename[False-False-image] 1,119,791.4910 (>1000.0) 1,119,791.4910 (>1000.0) 1,119,791.4910 (>1000.0) 0.0000 (1.0) 1,119,791.4910 (>1000.0) 0.0000 (1.0) 0;0 0.8930 (0.00) 1 1 test_item_rename[True-False-profile] 1,132,550.4270 (>1000.0) 1,132,550.4270 (>1000.0) 1,132,550.4270 (>1000.0) 0.0000 (1.0) 1,132,550.4270 (>1000.0) 0.0000 (1.0) 0;0 0.8830 (0.00) 1 1 test_item_rename[True-True-image] 1,155,507.0060 (>1000.0) 1,155,507.0060 (>1000.0) 1,155,507.0060 (>1000.0) 0.0000 (1.0) 1,155,507.0060 (>1000.0) 0.0000 (1.0) 0;0 0.8654 (0.00) 1 1 test_item_rename[False-True-image] 1,203,782.3530 (>1000.0) 1,203,782.3530 (>1000.0) 1,203,782.3530 (>1000.0) 0.0000 (1.0) 1,203,782.3530 (>1000.0) 0.0000 (1.0) 0;0 0.8307 (0.00) 1 1 test_sync[True-False] 1,251,665.2800 (>1000.0) 1,251,665.2800 (>1000.0) 1,251,665.2800 (>1000.0) 0.0000 (1.0) 1,251,665.2800 (>1000.0) 0.0000 (1.0) 0;0 0.7989 (0.00) 1 1 test_sync[False-False] 1,263,655.2020 (>1000.0) 1,263,655.2020 (>1000.0) 1,263,655.2020 (>1000.0) 0.0000 (1.0) 1,263,655.2020 (>1000.0) 0.0000 (1.0) 0;0 0.7914 (0.00) 1 1 test_item_rename[False-False-profile] 1,272,759.7200 (>1000.0) 1,272,759.7200 (>1000.0) 1,272,759.7200 (>1000.0) 0.0000 (1.0) 1,272,759.7200 (>1000.0) 0.0000 (1.0) 0;0 0.7857 (0.00) 1 1 test_sync[True-True] 1,327,599.2360 (>1000.0) 1,327,599.2360 (>1000.0) 1,327,599.2360 (>1000.0) 0.0000 (1.0) 1,327,599.2360 (>1000.0) 0.0000 (1.0) 0;0 0.7532 (0.00) 1 1 test_sync[False-True] 1,367,545.7940 (>1000.0) 1,367,545.7940 (>1000.0) 1,367,545.7940 (>1000.0) 0.0000 (1.0) 1,367,545.7940 (>1000.0) 0.0000 (1.0) 0;0 0.7312 (0.00) 1 1 test_item_rename[True-True-profile] 1,392,447.3040 (>1000.0) 1,392,447.3040 (>1000.0) 1,392,447.3040 (>1000.0) 0.0000 (1.0) 1,392,447.3040 (>1000.0) 0.0000 (1.0) 0;0 0.7182 (0.00) 1 1 test_item_rename[False-True-profile] 1,640,813.0950 (>1000.0) 1,640,813.0950 (>1000.0) 1,640,813.0950 (>1000.0) 0.0000 (1.0) 1,640,813.0950 (>1000.0) 0.0000 (1.0) 0;0 0.6095 (0.00) 1 1 test_item_copy[True-False-system] 2,155,762.7360 (>1000.0) 2,155,762.7360 (>1000.0) 2,155,762.7360 (>1000.0) 0.0000 (1.0) 2,155,762.7360 (>1000.0) 0.0000 (1.0) 0;0 0.4639 (0.00) 1 1 test_systems_create[True-False] 2,169,008.6710 (>1000.0) 2,169,008.6710 (>1000.0) 2,169,008.6710 (>1000.0) 0.0000 (1.0) 2,169,008.6710 (>1000.0) 0.0000 (1.0) 0;0 0.4610 (0.00) 1 1 test_item_copy[False-False-system] 2,207,707.9720 (>1000.0) 2,207,707.9720 (>1000.0) 2,207,707.9720 (>1000.0) 0.0000 (1.0) 2,207,707.9720 (>1000.0) 0.0000 (1.0) 0;0 0.4530 (0.00) 1 1 test_systems_create[False-False] 2,224,561.9640 (>1000.0) 2,224,561.9640 (>1000.0) 2,224,561.9640 (>1000.0) 0.0000 (1.0) 2,224,561.9640 (>1000.0) 0.0000 (1.0) 0;0 0.4495 (0.00) 1 1 test_all_items_create[True-False] 2,291,228.1450 (>1000.0) 2,291,228.1450 (>1000.0) 2,291,228.1450 (>1000.0) 0.0000 (1.0) 2,291,228.1450 (>1000.0) 0.0000 (1.0) 0;0 0.4364 (0.00) 1 1 test_systems_create[True-True] 2,320,496.3230 (>1000.0) 2,320,496.3230 (>1000.0) 2,320,496.3230 (>1000.0) 0.0000 (1.0) 2,320,496.3230 (>1000.0) 0.0000 (1.0) 0;0 0.4309 (0.00) 1 1 test_item_copy[True-True-system] 2,327,735.0640 (>1000.0) 2,327,735.0640 (>1000.0) 2,327,735.0640 (>1000.0) 0.0000 (1.0) 2,327,735.0640 (>1000.0) 0.0000 (1.0) 0;0 0.4296 (0.00) 1 1 test_item_copy[False-True-system] 2,419,706.9320 (>1000.0) 2,419,706.9320 (>1000.0) 2,419,706.9320 (>1000.0) 0.0000 (1.0) 2,419,706.9320 (>1000.0) 0.0000 (1.0) 0;0 0.4133 (0.00) 1 1 test_systems_create[False-True] 2,425,554.5290 (>1000.0) 2,425,554.5290 (>1000.0) 2,425,554.5290 (>1000.0) 0.0000 (1.0) 2,425,554.5290 (>1000.0) 0.0000 (1.0) 0;0 0.4123 (0.00) 1 1 test_all_items_create[False-False] 2,435,827.4160 (>1000.0) 2,435,827.4160 (>1000.0) 2,435,827.4160 (>1000.0) 0.0000 (1.0) 2,435,827.4160 (>1000.0) 0.0000 (1.0) 0;0 0.4105 (0.00) 1 1 test_all_items_create[True-True] 2,687,397.3290 (>1000.0) 2,687,397.3290 (>1000.0) 2,687,397.3290 (>1000.0) 0.0000 (1.0) 2,687,397.3290 (>1000.0) 0.0000 (1.0) 0;0 0.3721 (0.00) 1 1 test_all_items_create[False-True] 2,931,534.8660 (>1000.0) 2,931,534.8660 (>1000.0) 2,931,534.8660 (>1000.0) 0.0000 (1.0) 2,931,534.8660 (>1000.0) 0.0000 (1.0) 0;0 0.3411 (0.00) 1 1 ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ```