search

KhronosGroup / VK-GL-CTS

Khronos Vulkan, OpenGL, and OpenGL ES Conformance Tests
https://www.khronos.org/
Apache License 2.0
521 stars 290 forks source link

Is a drastic reduction in number of tests from vulkan-cts-1.3.7.3 to vulkan-cts-1.3.8.0 expected and intentional? #447

Open billhollings opened 7 months ago

billhollings commented 7 months ago

vulkan-cts-1.3.8.0 seems to have only about two-thirds the number of tests that vulkan-cts-1.3.7.3 has.

From a recent comparison running tests on MoltenVK:

dEQP Core vulkan-cts-1.3.7.3-0-gd71a36db16d98313c431829432a136dbda692a08 (0xd71a36db) starting..
Test run totals:
  Passed:        404332/4006625 (10.1%)
  Failed:        11140/4006625 (0.3%)
  Not supported: 3591092/4006625 (89.6%)
  Warnings:      61/4006625 (0.0%)
  Waived:        0/4006625 (0.0%)

dEQP Core vulkan-cts-1.3.8.0-0-gd13582ab1e753a41b2cedf8e3c504e1cb92edfe5 (0xd13582ab) starting..
Test run totals:
  Passed:        392275/2702883 (14.5%)
  Failed:        9919/2702883 (0.4%)
  Not supported: 2300634/2702883 (85.1%)
  Warnings:      55/2702883 (0.0%)
  Waived:        0/2702883 (0.0%)

We seem to have an overall reduction of 1.3 M tests.

I found these comments in the vulkan-cts-1.3.8.0 release notes:

Remove duplicate pipeline protected access tests

Which seems to account for about 320 K tests.

Simplify BorderSwizzle TestNode

Which seems to account for about 280 K tests.

Reduce the amount of variations in VK_EXT_host_image_copy tests

Which seems to account for about 108 K tests.

Remove duplicate shader object protected image tests

Which seems to account for about 861 K tests.

There seems to be a large reduction in dEQP-VK.pipeline.*.extended_dynamic_state.* tests, but I can't see a clear pattern to those removed tests, nor do a see a clear note about this in the release notes.

These seem to account for about 145 K tests.

But this is all assuming that the test reductions are driven by these notes. I would have expected such a large change in overall number to be mentioned in the release notes, since the trend has always been to increase the test count from release to release.

So, just to confirm, is this overall drastic reduction correct, and intentional? On a full run, should I expect "only" 2.7M tests? Or is it possible that I have somehow pulled or built CTS in a manner that has lost a large number of tests?

AlexGalazin-IMG commented 7 months ago

Sounds about right. In addition to what you listed 1fa0038d687544faf9696505b90688e4ebc4e83e removed about 1.09m (1088703) tests faf4fbbc8f8e2ae777a679965e9fb79a1701ff42 removed about 139k (138866) tests

billhollings commented 7 months ago

Sounds about right. In addition to what you listed 1fa0038 removed about 1.09m (1088703) tests faf4fbb removed about 139k (138866) tests

Ah. Great. Good work in consolidating!