Open vkushwaha-nv opened 3 months ago
why is sort -r
not sufficient for this?
sort -r mustpass.txt > mustpass-randomized.txt
--deqp-caselist-file=mustpass-randomized.txt
--deqp-caselist-file does not honor the order in the file. I have tried finding how it orders them by just eyeballing the runs and it's not sorted by test name.
What makes you think that the order in the file is not respected?
compute.txt:
dEQP-VK.compute.pipeline.basic.branch_past_barrier dEQP-VK.compute.pipeline.basic.concurrent_compute dEQP-VK.compute.pipeline.basic.copy_image_to_ssbo_large
deqp-vk-64.exe --deqp-caselist-file=compute.txt Writing test log into TestResults.qpa dEQP Core vulkan-cts-1.3.9.0-67-gea4b0dc51098a2580708d39c571decd5c5980db1 (0xea4b0dc5) starting.. target implementation = 'Default'
Test case 'dEQP-VK.compute.pipeline.basic.concurrent_compute'.. Pass (Test passed)
Test case 'dEQP-VK.compute.pipeline.basic.copy_image_to_ssbo_large'.. Pass (Compute succeeded)
Test case 'dEQP-VK.compute.pipeline.basic.branch_past_barrier'.. Pass (Pass)
DONE!
Test run totals: Passed: 3/3 (100.0%) Failed: 0/3 (0.0%) Not supported: 0/3 (0.0%) Warnings: 0/3 (0.0%) Waived: 0/3 (0.0%)
compute.txt
dEQP-VK.compute.pipeline.basic.copy_image_to_ssbo_large dEQP-VK.compute.pipeline.basic.branch_past_barrier dEQP-VK.compute.pipeline.basic.concurrent_compute
deqp-vk-64.exe --deqp-caselist-file=compute.txt Writing test log into TestResults.qpa dEQP Core vulkan-cts-1.3.9.0-67-gea4b0dc51098a2580708d39c571decd5c5980db1 (0xea4b0dc5) starting.. target implementation = 'Default'
Test case 'dEQP-VK.compute.pipeline.basic.concurrent_compute'.. Pass (Test passed)
Test case 'dEQP-VK.compute.pipeline.basic.copy_image_to_ssbo_large'.. Pass (Compute succeeded)
Test case 'dEQP-VK.compute.pipeline.basic.branch_past_barrier'.. Pass (Pass)
DONE!
Test run totals: Passed: 3/3 (100.0%) Failed: 0/3 (0.0%) Not supported: 0/3 (0.0%) Warnings: 0/3 (0.0%) Waived: 0/3 (0.0%)
Ping.
While working on an extension implementation, I noticed that ordering of tests help isolate some bugs related to driver incorrectly saving some state. I did not see an option to randomize test order from a case list. Would that be something that can be added?