First test on FTL fails because of native exception

[jBorkowska]

Sometimes, the first test running on Firebase Test Lab fails on the very beginning. This is the exception that crashes the test:

java.net.ConnectException: Failed to connect to localhost/127.0.0.1:8082
at com.squareup.okhttp.internal.io.RealConnection.connectSocket(RealConnection.java:143)
at com.squareup.okhttp.internal.io.RealConnection.connect(RealConnection.java:112)
at com.squareup.okhttp.internal.http.StreamAllocation.findConnection(StreamAllocation.java:184)
at com.squareup.okhttp.internal.http.StreamAllocation.findHealthyConnection(StreamAllocation.java:126)
at com.squareup.okhttp.internal.http.StreamAllocation.newStream(StreamAllocation.java:95)
at com.squareup.okhttp.internal.http.HttpEngine.connect(HttpEngine.java:281)
at com.squareup.okhttp.internal.http.HttpEngine.sendRequest(HttpEngine.java:224)
at com.squareup.okhttp.Call.getResponse(Call.java:286)
at com.squareup.okhttp.Call$ApplicationInterceptorChain.proceed(Call.java:243)
at com.squareup.okhttp.Call.getResponseWithInterceptorChain(Call.java:205)
at com.squareup.okhttp.Call.execute(Call.java:80)
at pl.leancode.patrol.contracts.PatrolAppServiceClient.performRequest(PatrolAppServiceClient.kt:45)
at pl.leancode.patrol.contracts.PatrolAppServiceClient.runDartTest(PatrolAppServiceClient.kt:23)
at pl.leancode.patrol.PatrolAppServiceClient.runDartTest(PatrolAppServiceClient.kt:53)
at pl.leancode.patrol.PatrolJUnitRunner.runDartTest(PatrolJUnitRunner.java:133)
at com.exmaple.app.MainActivityTest.runDartTest(MainActivityTest.java:29)

It happens only to the first tests, no matter which one it is. About half of the test runs fails like this. Spotted in internal project (cars). Only on Android.

[petrnymsa]

Experiencig same behavior after tests are uploaded and ran in Firebase Test Lab

`` java.net.ConnectException: Failed to connect to localhost/127.0.0.1:8082 at com.squareup.okhttp.internal.io.RealConnection.connectSocket(RealConnection.java:143) at com.squareup.okhttp.internal.io.RealConnection.connect(RealConnection.java:112) at com.squareup.okhttp.internal.http.StreamAllocation.findConnection(StreamAllocation.java:184) at com.squareup.okhttp.internal.http.StreamAllocation.findHealthyConnection(StreamAllocation.java:126) at com.squareup.okhttp.internal.http.StreamAllocation.newStream(StreamAllocation.java:95) at com.squareup.okhttp.internal.http.HttpEngine.connect(HttpEngine.java:281) at com.squareup.okhttp.internal.http.HttpEngine.sendRequest(HttpEngine.java:224) at com.squareup.okhttp.Call.getResponse(Call.java:286) at com.squareup.okhttp.Call$ApplicationInterceptorChain.proceed(Call.java:243) at com.squareup.okhttp.Call.getResponseWithInterceptorChain(Call.java:205) at com.squareup.okhttp.Call.execute(Call.java:80) at pl.leancode.patrol.contracts.PatrolAppServiceClient.performRequest(PatrolAppServiceClient.kt:45) at pl.leancode.patrol.contracts.PatrolAppServiceClient.runDartTest(PatrolAppServiceClient.kt:23) at pl.leancode.patrol.PatrolAppServiceClient.runDartTest(PatrolAppServiceClient.kt:53) at pl.leancode.patrol.PatrolJUnitRunner.runDartTest(PatrolJUnitRunner.java:133) at cz.bakalari.mobile.MainActivityTest.runDartTest(MainActivityTest.java:29)

[jBorkowska]

Lately I got the same error on second test. Though I still think that it is some time-related issue. I added a dummy test as a first one, so it can fail and the real tests will execute after, though it was too short - the test callback was empty and second test failed

[jBorkowska]

It also happens during our Patrol internal tests - though the error looks a bit different:

java.lang.RuntimeException: pl.leancode.patrol.contracts.PatrolAppServiceClientException: Invalid response 500, Internal Server Error
    at pl.leancode.patrol.PatrolJUnitRunner.runDartTest(PatrolJUnitRunner.java:140)
    at pl.leancode.patrol.e2e_app.MainActivityTest.runDartTest(MainActivityTest.java:29)
    ... 39 trimmed
Caused by: pl.leancode.patrol.contracts.PatrolAppServiceClientException: Invalid response 500, Internal Server Error
    at pl.leancode.patrol.contracts.PatrolAppServiceClient.performRequest(PatrolAppServiceClient.kt:47)
    at pl.leancode.patrol.contracts.PatrolAppServiceClient.runDartTest(PatrolAppServiceClient.kt:23)
    at pl.leancode.patrol.PatrolAppServiceClient.runDartTest(PatrolAppServiceClient.kt:53)
    at pl.leancode.patrol.PatrolJUnitRunner.runDartTest(PatrolJUnitRunner.java:133)
    ... 41 more

[ivan-horchakov]

In my case (Flutter 3.19.6, patrol 3.7.1, patrol_cli 2.7.0) mentioned ConnectException reproducible only on Firebase TestLab's virtual Android device (real Android device is Ok).

[pdenert]

I've tried to investigate this issue. I've checked it on various tests, but for reproducibility purposes here are the steps I took on packages/patrol/example:

patrol build android

output

```bash example % patrol build android Resolving dependencies in `/Users/piotrdenert/FlutterProjects/patrol/packages/patrol_cli`... Downloading packages... Got dependencies in `/Users/piotrdenert/FlutterProjects/patrol/packages/patrol_cli`. Resolving dependencies in `/Users/piotrdenert/FlutterProjects/patrol/packages/patrol_cli`... Downloading packages... Got dependencies in `/Users/piotrdenert/FlutterProjects/patrol/packages/patrol_cli`. Building package executable... Built patrol_cli:main. • Building apk with entrypoint test_bundle.dart... ✓ Completed building apk with entrypoint test_bundle.dart (26.9s) ```

Then I tested it in FTL, on a variety of devices with different API SDK versions, using this command:

gcloud firebase test android run \
    --type instrumentation \        
    --app build/app/outputs/apk/dev/debug/app-dev-debug.apk \
        --test build/app/outputs/apk/androidTest/dev/debug/app-dev-debug-androidTest.apk \
    --device model="$ANDROID_DEVICE_MODEL",version="$ANDROID_DEVICE_VERSION",locale=en,orientation=portrait \
    --timeout 10m \
    --results-bucket="patrol_runs" \
    --use-orchestrator \
    --environment-variables clearPackageData=true

All available in FTL devices may be listed by:

gcloud firebase test android models list

What I've found is that it fails randomly, but only when I use one of the VIRTUAL devices (e.g. MediumPhone.arm) with API SDK version = 34. On API SDK = 33 and earlier it never throws this exception. Same for PHYSICAL devices (e.g. Pixel 8):

java.net.ConnectException: Failed to connect to localhost/127.0.0.1:8082
    at com.squareup.okhttp.internal.io.RealConnection.connectSocket(RealConnection.java:143)
    at com.squareup.okhttp.internal.io.RealConnection.connect(RealConnection.java:112)
    at com.squareup.okhttp.internal.http.StreamAllocation.findConnection(StreamAllocation.java:184)
    at com.squareup.okhttp.internal.http.StreamAllocation.findHealthyConnection(StreamAllocation.java:126)
    at com.squareup.okhttp.internal.http.StreamAllocation.newStream(StreamAllocation.java:95)
    at com.squareup.okhttp.internal.http.HttpEngine.connect(HttpEngine.java:281)
    at com.squareup.okhttp.internal.http.HttpEngine.sendRequest(HttpEngine.java:224)
    at com.squareup.okhttp.Call.getResponse(Call.java:286)
    at com.squareup.okhttp.Call$ApplicationInterceptorChain.proceed(Call.java:243)
    at com.squareup.okhttp.Call.getResponseWithInterceptorChain(Call.java:205)
    at com.squareup.okhttp.Call.execute(Call.java:80)
    at pl.leancode.patrol.contracts.PatrolAppServiceClient.performRequest(PatrolAppServiceClient.kt:45)
    at pl.leancode.patrol.contracts.PatrolAppServiceClient.runDartTest(PatrolAppServiceClient.kt:23)
    at pl.leancode.patrol.PatrolAppServiceClient.runDartTest(PatrolAppServiceClient.kt:53)
    at pl.leancode.patrol.PatrolJUnitRunner.runDartTest(PatrolJUnitRunner.java:133)
    at pl.leancode.patrol.example.MainActivityTest.runDartTest(MainActivityTest.java:29)

MediumPhone.arm, API Level 34

```bash example % gcloud firebase test android run \ --type instrumentation \ --app build/app/outputs/apk/dev/debug/app-dev-debug.apk \ --test build/app/outputs/apk/androidTest/dev/debug/app-dev-debug-androidTest.apk \ --device model=MediumPhone.arm,version=34,locale=en,orientation=portrait \ --timeout 10m \ --results-bucket="patrol_runs" \ --use-orchestrator \ --environment-variables clearPackageData=true Have questions, feedback, or issues? Get support by visiting: https://firebase.google.com/support/ Uploading [build/app/outputs/apk/dev/debug/app-dev-debug.apk] to Firebase Test Lab... Uploading [build/app/outputs/apk/androidTest/dev/debug/app-dev-debug-androidTest.apk] to Firebase Test Lab... Raw results will be stored in your GCS bucket at [xxx] Test [matrix-amzh5y2lzgona] has been created in the Google Cloud. Creating individual test executions...done. Firebase Test Lab will execute your instrumentation test on 1 device(s). More devices may be added later if flaky test attempts are specified. Test results will be streamed to [ xxx ]. 16:41:27 Test is Pending 16:42:14 Starting attempt 1. 16:42:14 Test is Running 16:42:21 Setting up Android test. 16:42:21 Starting Android test. 16:43:09 Completed Android test. 16:43:09 Tearing down Android test. 16:43:09 Done. Test time = 34 (secs) 16:43:09 Starting results processing. Attempt: 1 16:43:30 Completed results processing. Time taken = 14 (secs) 16:43:30 Test is Finished Instrumentation testing complete. More details are available at [xxx]. ┌─────────┬────────────────────────────────┬───────────────────────────────┐ │ OUTCOME │ TEST_AXIS_VALUE │ TEST_DETAILS │ ├─────────┼────────────────────────────────┼───────────────────────────────┤ │ Failed │ MediumPhone.arm-34-en-portrait │ 1 test cases failed, 1 passed │ └─────────┴────────────────────────────────┴───────────────────────────────┘ ```

Patrol doctor

```bash Patrol doctor: Patrol CLI version: 2.8.1 Flutter command: flutter Flutter 3.22.0 • channel stable Android: • Program adb found in /Users/piotrdenert/Library/Android/sdk/platform-tools/adb • Env var $ANDROID_HOME set to /Users/piotrdenert/Library/Android/sdk iOS / macOS: • Program xcodebuild found in /usr/bin/xcodebuild • Program ideviceinstaller found in /opt/homebrew/bin/ideviceinstaller ```

flutter doctor

```bash flutter doctor Doctor summary (to see all details, run flutter doctor -v): [✓] Flutter (Channel stable, 3.22.0, on macOS 14.4.1 23E224 darwin-arm64, locale pl-PL) [✓] Android toolchain - develop for Android devices (Android SDK version 34.0.0) [✓] Xcode - develop for iOS and macOS (Xcode 15.1) [✓] Chrome - develop for the web [✓] Android Studio (version 2023.1) [✓] VS Code (version 1.90.0) [✓] Connected device (5 available) [✓] Network resources • No issues found! ```

[ivan-horchakov]

U.P.D.: just caught it locally on Android emulator API 34 (Flutter 3.19.6, patrol 3.7.1, patrol_cli 2.8.1). It can be related to reusing of tests (for example we have LogIn test & reusung it more than one time in test set).

[ivan-horchakov]

U.P.D.: just caught it locally on Android emulator API 34 (Flutter 3.19.6, patrol 3.7.1, patrol_cli 2.8.1). It can be related to reusing of tests (for example we have LogIn test & reusung it more than one time in test set).

U.P.D.2: so yeah, I was right. If we have separate "patrolTest" (in my case LogIn) in separate file & trying to reuse it - we receiving mentioned "Invalid response 500". But if we put what we need to reuse in method - everything is Ok.

[adrian-moisa]

Any solutions found? I stumbled on this problem as well. I have no idea what triggered it. My pipelines worked fine for a long time.

Edit: Looks like I managed to pass this error by using flutter clean and flushing the cache from my gitlab runner pipeline. This error kept showing because I made some changes in the /android folder in order to rename the app and add improved app icons. So, most likely, due to the rename, I had to use flutter clean to remove and rebuild the graddle files. After this it worked. Error was gone. Not sure if this applies to your issues, but at least for my project that's what worked.

Edit 2 The error keeps showing back. I even updated to latest patrol and patrol_cli. Sometimes it works sometimes not. Keeps failing with java.net.ConnectException: Failed to connect to localhost/127.0.0.1:8082. Even more annoying. on local it works just fine. Also on my test VPS works fine if use my maintenance account. If I then run it via gitlab runner agent and account I then hit this issue. The pipeline has been working fine for a long while. This is a new problem.

Edit 3 I think I know why local and VPS maintenance user work fine. It's because I use patrol develop there. In the gitlab runner I use patrol test. Just tried it locally, patrol test works just fine.

[ahlstro]

I also have this problem. Any solutions?

Edit: As pdenert stated, changing the sdk to 33 works. 34 doesn't

Edit2: Now i got the error again in firebase testlab, but only for one of my 6 tests. Rest of them passed.