search

oppia / oppia-android

A free, online & offline learning platform to make quality education accessible for all.
https://www.oppia.org
Apache License 2.0
300 stars 502 forks source link

[BUG]: SurveyProgressControllerTest is approximately 1% flaky #5338

Open BenHenning opened 4 months ago

BenHenning commented 4 months ago

Describe the bug

I found in #5335 that SurveyProgressControllerTest has a rare chance of flaking. I encountered 11 failures out of 1000 runs, with the following results:

6/11 failures:

testSubmitAnswer_forMarketFitQuestion_succeeds(org.oppia.android.domain.survey.SurveyProgressControllerTest)
java.lang.IllegalStateException: Expected next result to be a success, not: Pending(resultTimeMillis=100)
        at org.oppia.android.testing.data.DataProviderTestMonitor.retrieveSuccess(DataProviderTestMonitor.kt:130)
        at org.oppia.android.testing.data.DataProviderTestMonitor.waitForNextSuccessResult(DataProviderTestMonitor.kt:66)
        at org.oppia.android.testing.data.DataProviderTestMonitor$Factory.waitForNextSuccessfulResult(DataProviderTestMonitor.kt:201)
        at org.oppia.android.domain.survey.SurveyProgressControllerTest.startSuccessfulSurveySession(SurveyProgressControllerTest.kt:432)
        at org.oppia.android.domain.survey.SurveyProgressControllerTest.testSubmitAnswer_forMarketFitQuestion_succeeds(SurveyProgressControllerTest.kt:166)

2/11 failures:

testMoveToPreviousQuestion_afterMovingToNextQuestion_isSuccess(org.oppia.android.domain.survey.SurveyProgressControllerTest)
java.lang.IllegalStateException: Expected next result to be a success, not: Pending(resultTimeMillis=100)
        at org.oppia.android.testing.data.DataProviderTestMonitor.retrieveSuccess(DataProviderTestMonitor.kt:130)
        at org.oppia.android.testing.data.DataProviderTestMonitor.waitForNextSuccessResult(DataProviderTestMonitor.kt:66)
        at org.oppia.android.testing.data.DataProviderTestMonitor$Factory.waitForNextSuccessfulResult(DataProviderTestMonitor.kt:201)
        at org.oppia.android.domain.survey.SurveyProgressControllerTest.submitAnswer(SurveyProgressControllerTest.kt:452)
        at org.oppia.android.domain.survey.SurveyProgressControllerTest.submitUserTypeAnswer(SurveyProgressControllerTest.kt:459)
        at org.oppia.android.domain.survey.SurveyProgressControllerTest.testMoveToPreviousQuestion_afterMovingToNextQuestion_isSuccess(SurveyProgressControllerTest.kt:295)

1/11 failures:

testStartSurveySession_sessionStartsWithInitialQuestion(org.oppia.android.domain.survey.SurveyProgressControllerTest)
java.lang.IllegalStateException: Expected next result to be a success, not: Pending(resultTimeMillis=100)
        at org.oppia.android.testing.data.DataProviderTestMonitor.retrieveSuccess(DataProviderTestMonitor.kt:130)
        at org.oppia.android.testing.data.DataProviderTestMonitor.waitForNextSuccessResult(DataProviderTestMonitor.kt:66)
        at org.oppia.android.testing.data.DataProviderTestMonitor$Factory.waitForNextSuccessfulResult(DataProviderTestMonitor.kt:201)
        at org.oppia.android.domain.survey.SurveyProgressControllerTest.startSuccessfulSurveySession(SurveyProgressControllerTest.kt:432)
        at org.oppia.android.domain.survey.SurveyProgressControllerTest.testStartSurveySession_sessionStartsWithInitialQuestion(SurveyProgressControllerTest.kt:97)

1/11 failures:

testSubmitAnswer_afterMovingToPreviousQuestion_isSuccess(org.oppia.android.domain.survey.SurveyProgressControllerTest)
expected not to be an instance of: org.oppia.android.util.data.AsyncResult$Pending
but was                          : Pending(resultTimeMillis=100)
        at org.oppia.android.testing.data.DataProviderTestMonitor$Factory.ensureDataProviderExecutes(DataProviderTestMonitor.kt:189)
        at org.oppia.android.domain.survey.SurveyProgressControllerTest.moveToPreviousQuestion(SurveyProgressControllerTest.kt:445)
        at org.oppia.android.domain.survey.SurveyProgressControllerTest.testSubmitAnswer_afterMovingToPreviousQuestion_isSuccess(SurveyProgressControllerTest.kt:313)

1/11 failures:

testSubmitAnswer_submitNpsScore0f3_loadsDetractorFeedbackQuestion(org.oppia.android.domain.survey.SurveyProgressControllerTest)
java.lang.IllegalStateException: Expected next result to be a success, not: Pending(resultTimeMillis=100)
        at org.oppia.android.testing.data.DataProviderTestMonitor.retrieveSuccess(DataProviderTestMonitor.kt:130)
        at org.oppia.android.testing.data.DataProviderTestMonitor.waitForNextSuccessResult(DataProviderTestMonitor.kt:66)
        at org.oppia.android.testing.data.DataProviderTestMonitor$Factory.waitForNextSuccessfulResult(DataProviderTestMonitor.kt:201)
        at org.oppia.android.domain.survey.SurveyProgressControllerTest.startSuccessfulSurveySession(SurveyProgressControllerTest.kt:432)
        at org.oppia.android.domain.survey.SurveyProgressControllerTest.testSubmitAnswer_submitNpsScore0f3_loadsDetractorFeedbackQuestion(SurveyProgressControllerTest.kt:238)

Steps To Reproduce

Check out the latest develop (018e9c97244ea8c86b3b6800a81a3c6f5fd84c74) and run the following:

bazel test --runs_per_test=1000 //domain/src/test/java/org/oppia/android/domain/survey:SurveyProgressControllerTest

Expected Behavior

No tests should flake.

Screenshots/Videos

N/A

What device/emulator are you using?

N/A

Which Android version is your device/emulator running?

N/A

Which version of the Oppia Android app are you using?

N/A

Additional Context

From the stack traces, this appears to be a timing issue. Upon inspecting the controller and its tests, I don't see an obvious reason as to why there are some syncing problems; more investigation is needed to determine a possible root cause (since finding a deterministic repro is probably going to be even harder to do).