The total number of test cases in conformance_log.xml is inconsistent with the number in the openxr_conformance.xml

WeijieWanng commented 1 year ago

In v1.0.27.0, we found that the total number of test cases is inconsistent with the number in the openxr_conformance.xml. For example, we run non-interactive test cases on the Android platform.

Test options:
   graphicsPlugin: OpenGLES
   formFactor: Hmd
   hands: both
   environmentBlendMode: Opaque
   viewConfiguration: Stereo
   enabledAPILayers:
   enabledInstanceExtensions:
   invalidHandleValidation: no
   fileLineLoggingEnabled: yes
   debugMode: no

The number of test cases is 61 in conformance_log.xml, but the number of test cases listed in openxr_conformance.xml is 57. The below 4 test cases are executed in conformance_log.xml, but they are not listed in openxr_conformance.xml.

XR_KHR_convert_timespec_time //It is enabled(or disabled) by macro XR_USE_TIMESPEC
XR_KHR_headless //It is enabled(or disabled) by macro XR_KHR_headless
XR_EXT_performance_settings //It is an empty test case.
XR_EXT_local_floor

We checked the source code and found that this is caused by statistical rules. The current test report openxr_conformance.xml is prone to confusion. From our perspective, it will be better if we use the defined macro SKIP to do some optimization.

Here is the detailed analysis:

The rule that the test case will be written in openxr_conformance.xml:

Has assertions or
stdOut is not empty or
stdErr is not empty. https://github.com/KhronosGroup/OpenXR-CTS/blob/3d02d70eab669b72429cb6065e65b817ba34c1dd/src/conformance/framework/catch_reporter_cts.cpp#L217

For test case XR_KHR_convert_timespec_time, the macro XR_USE_TIMESPEC is not defined, it is the equivalent of an empty test case, so it doesn’t meet the rule that writes in openxr_conformance.xml.

Proposal:

TEST_CASE("XR_KHR_convert_timespec_time", "")
{
    bool executed = false
    #ifdef XR_USE_TIMESPEC
            ......
            ......
    executed = true
            ......
            ......
    #endif  // XR_USE_TIMESPEC
    If (!executed) {
        SKIP("XR_KHR_convert_timespec_time not supported");
    }
}

For test case XR_KHR_headless, the macro is not defined, it is the equivalent of an empty test case, so it doesn’t meet the rule that writes in openxr_conformance.xml.

https://github.com/KhronosGroup/OpenXR-CTS/blob/3d02d70eab669b72429cb6065e65b817ba34c1dd/src/conformance/conformance_test/test_XR_KHR_headless.cpp#L32

Proposal:

TEST_CASE("XR_KHR_headless", "")
{
    bool executed = false
    #ifdef XR_KHR_headless
            ......
            ......
    executed = true
            ......
            ......
    #endif  // XR_KHR_headless
    If (!executed) {
    SKIP("XR_KHR_headless not supported");
    }
}

For test case XR_EXT_performance_settings, it is an empty test case, so it doesn’t meet the rule that writes in openxr_conformance.xml.

https://github.com/KhronosGroup/OpenXR-CTS/blob/3d02d70eab669b72429cb6065e65b817ba34c1dd/src/conformance/conformance_test/test_XR_EXT_performance_settings.cpp#L22 Proposal:

TEST_CASE("XR_EXT_performance_settings", "")
{
    // XrResult xrPerfSettingsSetPerformanceLevelEXT(XrSession session, XrPerfSettingsDomainEXT domain,
    //              XrPerfSettingsLevelEXT level);
    SKIP("XR_EXT_performance_settings not supported");
 }

For test case XR_EXT_local_floor, it is the equivalent of an empty test case if not supported, it doesn’t meet the rule that writes in openxr_conformance.xml. https://github.com/KhronosGroup/OpenXR-CTS/blob/3d02d70eab669b72429cb6065e65b817ba34c1dd/src/conformance/conformance_test/test_XR_EXT_local_floor.cpp#L33 Proposal:
```
TEST_CASE("XR_EXT_local_floor", "")
{
GlobalData& globalData = GetGlobalData();
if (!globalData.IsInstanceExtensionSupported(XR_EXT_LOCAL_FLOOR_EXTENSION_NAME)) {
    SKIP(XR_EXT_LOCAL_FLOOR_EXTENSION_NAME  " not supported");
    return;
}
    ......
    ......
    ......
}
```

rpavlik-bot commented 1 year ago

An issue (number 2072) has been filed to correspond to this issue in the internal Khronos GitLab (Khronos members only: KHR:openxr/openxr#2072 ), to facilitate working group processes.

This GitHub issue will continue to be the main site of discussion.

rpavlik commented 1 year ago

Thanks for this report. Some of the issues you identified (always empty tests) are resolved in the 1.0.29.0 release just published. The next release will have a few more small fixes.

rpavlik commented 8 months ago

This has been fixed in the 1.0.30.0 release. A newer release is now the latest, I accidentally missed closing this issue when releasing 1.0.30.0.

KhronosGroup / OpenXR-CTS

The total number of test cases in conformance_log.xml is inconsistent with the number in the openxr_conformance.xml #60