Intermittent UI test failure ui-samples-glean - rg.mozilla.samples.glean.pings.BaselinePingTest - validateBaselinePing - java.lang.NullPointerException

[Archaeopteryx]

Firebase Test Run:

https://console.firebase.google.com/project/moz-android-components-230120/testlab/histories/bh.acacddb44ee28c64/matrices/6036139925997368609/executions/bs.b204ab25d59b8eae/testcases/1

Stacktrace:

java.lang.NullPointerException: Attempt to invoke virtual method 'java.lang.String okhttp3.mockwebserver.RecordedRequest.getPath()' on a null object reference
    at org.mozilla.samples.glean.pings.BaselinePingTest.waitForPingContent(BaselinePingTest.kt:92)
    at org.mozilla.samples.glean.pings.BaselinePingTest.waitForPingContent$default(BaselinePingTest.kt:83)
    at org.mozilla.samples.glean.pings.BaselinePingTest.validateBaselinePing(BaselinePingTest.kt:125)
    at java.lang.reflect.Method.invoke(Native Method)
    at java.lang.reflect.Method.invoke(Method.java:372)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
    at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
    at org.junit.rules.RunRules.evaluate(RunRules.java:20)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at androidx.test.ext.junit.runners.AndroidJUnit4.run(AndroidJUnit4.java:154)
    at org.junit.runners.Suite.runChild(Suite.java:128)
    at org.junit.runners.Suite.runChild(Suite.java:27)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
    at androidx.test.internal.runner.TestExecutor.execute(TestExecutor.java:56)
    at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:395)
    at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1837)

Build:

https://github.com/mozilla-mobile/android-components/commit/5c1494b17b001c7b39a6c0661e113280194c02e1

┆Issue is synchronized with this Jira Task

[AaronMT]

(CC @mdboom @badboy @csadilek - deja vu? https://github.com/mozilla-mobile/android-components/issues/9714) - this started appearing yesterday)

[AaronMT]

Another instance https://github.com/mozilla-mobile/android-components/pull/11369 in this push (https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/cA1VolkYRfK9vWNuuOkC6A/runs/0/artifacts/public/logs/live_backing.log)

[sv-ohorvath]

Failed on 5/26 on Nexus 6 java.lang.NullPointerException: Attempt to invoke virtual method 'java.lang.String okhttp3.mockwebserver.RecordedRequest.getPath()' on a null object reference at org.mozilla.samples.glean.pings.BaselinePingTest.waitForPingContent(BaselinePingTest.kt:92) at org.mozilla.samples.glean.pings.BaselinePingTest.waitForPingContent$default(BaselinePingTest.kt:83) at org.mozilla.samples.glean.pings.BaselinePingTest.validateBaselinePing(BaselinePingTest.kt:125)

https://console.firebase.google.com/project/moz-android-components-230120/testlab/histories/bh.acacddb44ee28c64/matrices/8559290400236241257/details?stepId=bs.f48197ffdbd8924e&testCaseId=1

[AndiAJ]

Failed 1x on 8/22 Firebase link

java.lang.NullPointerException: Attempt to invoke virtual method 'java.lang.String okhttp3.mockwebserver.RecordedRequest.getPath()' on a null object reference at org.mozilla.samples.glean.pings.BaselinePingTest.waitForPingContent(BaselinePingTest.kt:92) at org.mozilla.samples.glean.pings.BaselinePingTest.waitForPingContent$default(BaselinePingTest.kt:83) at org.mozilla.samples.glean.pings.BaselinePingTest.validateBaselinePing(BaselinePingTest.kt:125)

[badboy]

I'll take a look this week.

[AndiAJ]

Failed 2x on 8/22 Firebase link

java.lang.NullPointerException: Attempt to invoke virtual method 'java.lang.String okhttp3.mockwebserver.RecordedRequest.getPath()' on a null object reference at org.mozilla.samples.glean.pings.BaselinePingTest.waitForPingContent(BaselinePingTest.kt:92) at org.mozilla.samples.glean.pings.BaselinePingTest.waitForPingContent$default(BaselinePingTest.kt:83) at org.mozilla.samples.glean.pings.BaselinePingTest.validateBaselinePing(BaselinePingTest.kt:125)

Firebase link

java.lang.NullPointerException: Attempt to invoke virtual method 'java.lang.String okhttp3.mockwebserver.RecordedRequest.getPath()' on a null object reference at org.mozilla.samples.glean.pings.BaselinePingTest.waitForPingContent(BaselinePingTest.kt:92) at org.mozilla.samples.glean.pings.BaselinePingTest.waitForPingContent$default(BaselinePingTest.kt:83) at org.mozilla.samples.glean.pings.BaselinePingTest.validateBaselinePing(BaselinePingTest.kt:125)

Failed 1x on 8/22 Firebase link

java.lang.NullPointerException: Attempt to invoke virtual method 'java.lang.String okhttp3.mockwebserver.RecordedRequest.getPath()' on a null object reference at org.mozilla.samples.glean.pings.BaselinePingTest.waitForPingContent(BaselinePingTest.kt:92) at org.mozilla.samples.glean.pings.BaselinePingTest.waitForPingContent$default(BaselinePingTest.kt:83) at org.mozilla.samples.glean.pings.BaselinePingTest.validateBaselinePing(BaselinePingTest.kt:125)

[badboy]

I think I now know what's going on: A weird interaction of different tests running. I'm looking into how to best fix that.

[Archaeopteryx]

Issue has been observed again. @badboy could you take another look?

[AndiAJ]

Flaky 1x on 9/16 Main Firebase link

androidx.test.uiautomator.UiObjectNotFoundException: UiSelector[CONTAINS_DESCRIPTION=Firefox Fenix] at androidx.test.uiautomator.UiObject.click(UiObject.java:416) at org.mozilla.fenix.glean.BaselinePingTest.validateBaselinePing(BaselinePingTest.kt:169)

@badboy

[badboy]

Flaky 1x on 9/16 Main Firebase link

androidx.test.uiautomator.UiObjectNotFoundException: UiSelector[CONTAINS_DESCRIPTION=Firefox Fenix] at androidx.test.uiautomator.UiObject.click(UiObject.java:416) at org.mozilla.fenix.glean.BaselinePingTest.validateBaselinePing(BaselinePingTest.kt:169)

@badboy

This is a Fenix test run though. UiObjectNotFoundException -- it couldn't find the app to click on? What? This seems like a real intermittent, and I would have no idea how to fix that. That part of the code has not been touched in a long time (except some recent refactoring) and has nothing to do with the Glean part of it really.

[badboy]

Issue has been observed again. @badboy could you take another look?

This indeed looks like the old problem again. I guess we have to rethink how this test works, apparently we can't rely on the timing of things really. I filed https://bugzilla.mozilla.org/show_bug.cgi?id=1791559 for tracking. I'll throw some effort against it soon.

[csadilek]

Moved to bugzilla: https://bugzilla.mozilla.org/show_bug.cgi?id=1793239

Change performed by the Move to Bugzilla add-on.

[AndiAJ]

Failed 1x on 10/4 Firebase link

java.lang.NullPointerException: Attempt to invoke virtual method 'java.lang.String okhttp3.mockwebserver.RecordedRequest.getPath()' on a null object reference at org.mozilla.samples.glean.pings.BaselinePingTest.waitForPingContent(BaselinePingTest.kt:92) at org.mozilla.samples.glean.pings.BaselinePingTest.waitForPingContent$default(BaselinePingTest.kt:83) at org.mozilla.samples.glean.pings.BaselinePingTest.validateBaselinePing(BaselinePingTest.kt:125)

[AndiAJ]

Failed 1x on 10/10 Firebase link

java.lang.NullPointerException: Attempt to invoke virtual method 'java.lang.String okhttp3.mockwebserver.RecordedRequest.getPath()' on a null object reference at org.mozilla.samples.glean.pings.BaselinePingTest.waitForPingContent(BaselinePingTest.kt:92) at org.mozilla.samples.glean.pings.BaselinePingTest.waitForPingContent$default(BaselinePingTest.kt:83) at org.mozilla.samples.glean.pings.BaselinePingTest.validateBaselinePing(BaselinePingTest.kt:125)