search

KhronosGroup / VK-GL-CTS

Khronos Vulkan, OpenGL, and OpenGL ES Conformance Tests
https://www.khronos.org/
Apache License 2.0
525 stars 293 forks source link

Add command line option to randomize test execution order #472

Open vkushwaha-nv opened 3 months ago

vkushwaha-nv commented 3 months ago

While working on an extension implementation, I noticed that ordering of tests help isolate some bugs related to driver incorrectly saving some state. I did not see an option to randomize test order from a case list. Would that be something that can be added?

AlexGalazin-IMG commented 3 months ago

why is sort -r not sufficient for this?

sort -r mustpass.txt > mustpass-randomized.txt
--deqp-caselist-file=mustpass-randomized.txt
vkushwaha-nv commented 3 months ago

--deqp-caselist-file does not honor the order in the file. I have tried finding how it orders them by just eyeballing the runs and it's not sorted by test name.

AlexGalazin-IMG commented 3 months ago

What makes you think that the order in the file is not respected?

vkushwaha-nv commented 3 months ago

compute.txt:

dEQP-VK.compute.pipeline.basic.branch_past_barrier dEQP-VK.compute.pipeline.basic.concurrent_compute dEQP-VK.compute.pipeline.basic.copy_image_to_ssbo_large

deqp-vk-64.exe --deqp-caselist-file=compute.txt Writing test log into TestResults.qpa dEQP Core vulkan-cts-1.3.9.0-67-gea4b0dc51098a2580708d39c571decd5c5980db1 (0xea4b0dc5) starting.. target implementation = 'Default'

Test case 'dEQP-VK.compute.pipeline.basic.concurrent_compute'.. Pass (Test passed)

Test case 'dEQP-VK.compute.pipeline.basic.copy_image_to_ssbo_large'.. Pass (Compute succeeded)

Test case 'dEQP-VK.compute.pipeline.basic.branch_past_barrier'.. Pass (Pass)

DONE!

Test run totals: Passed: 3/3 (100.0%) Failed: 0/3 (0.0%) Not supported: 0/3 (0.0%) Warnings: 0/3 (0.0%) Waived: 0/3 (0.0%)

vkushwaha-nv commented 3 months ago

compute.txt

dEQP-VK.compute.pipeline.basic.copy_image_to_ssbo_large dEQP-VK.compute.pipeline.basic.branch_past_barrier dEQP-VK.compute.pipeline.basic.concurrent_compute

deqp-vk-64.exe --deqp-caselist-file=compute.txt Writing test log into TestResults.qpa dEQP Core vulkan-cts-1.3.9.0-67-gea4b0dc51098a2580708d39c571decd5c5980db1 (0xea4b0dc5) starting.. target implementation = 'Default'

Test case 'dEQP-VK.compute.pipeline.basic.concurrent_compute'.. Pass (Test passed)

Test case 'dEQP-VK.compute.pipeline.basic.copy_image_to_ssbo_large'.. Pass (Compute succeeded)

Test case 'dEQP-VK.compute.pipeline.basic.branch_past_barrier'.. Pass (Pass)

DONE!

Test run totals: Passed: 3/3 (100.0%) Failed: 0/3 (0.0%) Not supported: 0/3 (0.0%) Warnings: 0/3 (0.0%) Waived: 0/3 (0.0%)

vkushwaha-nv commented 2 months ago

Ping.