uPortal-Project / uPortal

Enterprise open source portal built by and for the higher education community.
https://www.apereo.org/projects/uportal
Apache License 2.0
272 stars 273 forks source link

Fix flaky test: testStylesheetDescriptor40ImportExport #2732

Closed KiruthikaJanakiraman closed 11 months ago

KiruthikaJanakiraman commented 11 months ago
Checklist
Description of change

This PR aims to fix the flaky test: org.apereo.portal.io.xml.IdentityImportExportTest.testStylesheetDescriptor40ImportExport similar to previously raised PRs: https://github.com/uPortal-Project/uPortal/pull/2718 and https://github.com/uPortal-Project/uPortal/pull/2719

I found and confirmed the flaky behaviour of the tests using an open-source research tool NonDex, which shuffles implementations of nondeterminism operations.

Problem The test case org.apereo.portal.io.xml.IdentityImportExportTest.testStylesheetDescriptor40ImportExport fails because the order of targetElements: channel and portlet in the exported xml varies upon nondeterministic shuffling.

https://github.com/uPortal-Project/uPortal/blob/7fe3d5d9078bd0be9b0442be37067bb399eab7fd/uPortal-webapp/src/test/resources/org/apereo/portal/io/xml/ssd/test_4-0.stylesheet-descriptor.xml#L49-L50

Solution This PR fixes the flakiness by sorting the targetElements in exportData before comparing importData and exportData XMLs.

Steps to reproduce The following command can be used to reproduce the assertion errors and verify the fix.

Integrate NonDex:

Add the following snippet to the top of the build.gradle in $PROJ_DIR:

plugins {
    id 'edu.illinois.nondex' version '2.1.1-1'
}

Append to build.gradle in $PROJ_DIR:

apply plugin: 'edu.illinois.nondex'

Execute Test with Gradle:

./gradlew --info uPortal-webapp:test --tests org.apereo.portal.io.xml.IdentityImportExportTest.testStylesheetDescriptor40ImportExport

Run NonDex:

./gradlew --info uPortal-webapp:nondexTest --tests=org.apereo.portal.io.xml.IdentityImportExportTest.testStylesheetDescriptor40ImportExport --nondexRuns=50 -x autoLintGradle

Test Environment:

Java version "1.8.0_381"
macOS Venture Version 13.4.1 (22F82)

Please let me know if you have any concerns or questions.

KiruthikaJanakiraman commented 11 months ago

Got it. In that case, "under determined" makes more sense. Thanks!