Running the same test twice in a run causes internal server error

lockieRichter commented 1 month ago

Steps to reproduce

Use the example test from the docs - integration_test/example_test.dart

import 'dart:io';

import 'package:flutter/material.dart';
import 'package:flutter_test/flutter_test.dart';
import 'package:patrol/patrol.dart';

void main() {
  patrolTest(
    'counter state is the same after going to home and switching apps',
    ($) async {
      // Replace later with your app's main widget
      await $.pumpWidgetAndSettle(
        MaterialApp(
          home: Scaffold(
            appBar: AppBar(title: const Text('app')),
            backgroundColor: Colors.blue,
          ),
        ),
      );

      expect($('app'), findsOneWidget);
      if (!Platform.isMacOS) {
        await $.native.pressHome();
      }
    },
  );
}

Run the following command.

patrol test -t integration_test/example_test.dart -t integration_test/example_test.dart

Actual results

The tests should pass, but they fail with java.lang.RuntimeException: pl.leancode.patrol.contracts.PatrolAppServiceClientException: Invalid response 500, Internal Server Error

Logs

[log.txt](https://github.com/user-attachments/files/16258202/log.txt)

Patrol version

patrol: ^3.6.1

Patrol Doctor output

``` Patrol doctor: Patrol CLI version: 3.0.1 Flutter command: flutter Flutter 3.19.5 • channel stable Android: • Program adb found in /Users/lockie/Library/Android/sdk/platform-tools/adb • Env var $ANDROID_HOME set to /Users/lockie/Library/Android/sdk iOS / macOS: • Program xcodebuild found in /usr/bin/xcodebuild • Program ideviceinstaller found in /opt/homebrew/bin/ideviceinstaller ```

Flutter Doctor output

``` Doctor summary (to see all details, run flutter doctor -v): [✓] Flutter (Channel stable, 3.19.5, on macOS 14.5 23F79 darwin-arm64, locale en-AU) [✓] Android toolchain - develop for Android devices (Android SDK version 34.0.0) [✓] Xcode - develop for iOS and macOS (Xcode 15.4) [✓] Chrome - develop for the web [✓] Android Studio (version 2023.1) [✓] VS Code (version 1.91.1) [✓] Connected device (3 available) ! Error: Browsing on the local area network for Testing’s iPhone. Ensure the device is unlocked and attached with a cable or associated with the same local area network as this Mac. The device must be opted into Developer Mode to connect wirelessly. (code -27) [✓] Network resources • No issues found!```

jBorkowska commented 1 month ago

Hi @lockieRichter! Thank you for reporting this. I reproduced this issue, got the same result. We haven't thought about such a use case, so we are not prepared for it. I consider it as a feature request, since it wasn't supposed to work.

Also could you please describe what is your use case here? We'd like to know what you need, so we can think how to solve it, maybe there is another way than what you tried here

lockieRichter commented 1 month ago

Hi Julia, thanks for the quick reply!

Our usecase currently stands that we sign into our app as a user with a set of permissions, test some actions, the sign out. We then sign in as a different user with less permissions (which is in a different test file), test some different actions, then sign out (this is the same test file as before and the one that is causing failures).

patrol test -t admin_sign_in_test.dart \
-t check_admin_actions_test.dart \
-t sign_out_test.dart \
-t standard_sign_in_test.dart \
-t check_standard_actions_test.dart \
-t sign_out_test.dart

I realise we could separate the sign out logic into a method and reuse that across two different test files, but we try to avoid doing that so that the test file contains the full context for what we are doing (this is dumb IMO, but it's what we do so I guess that's that lol)

Right now, we are working around this by having an admin_sign_out_test.dart and standard_sign_out_test.dart file with the same contents.

Our stupid usecase aside, I do feel that there is no reason why running the same test file multiple times should fail.

tegankr commented 1 month ago

Hi Julia, is there any planned task for Patrol to implement individual test retries? I would think this would be a necessity.. Then we wouldn't need to do this and encounter this error 🙏

tegankr commented 1 month ago

Another use case, because we couldn't find where Patrol implements individual test retries, we implemented a JUnit Rule which uses a custom RetryClass, but when we use this on the test re-run we also experience error raised here

jBorkowska commented 1 month ago

@lockieRichter I understand and agree that this use case is weird. Though we aren't convinced about making it work - tests should be separated from each other, and the order of their execution should not matter. We are designing our tool having those assumptions in mind and unfortunately changing it is not that simple. Those assumptions are also true in native frameworks for Android and iOS that we rely on under-the-hood, so we would need to somehow go around those limitations.

ReubenTurner-esusu commented 1 month ago

I encounter this issue when running a test with variants. Our use case is similar: we need to run users that have varying data through our tests so that we can confirm variations on the scenario we are testing, and we need to do that for each flavor we have. Using variants to run N users through one test is much more efficient for us. But we get this error when we do so. It's strange because the tests clearly pass - we watch them pass. But each time the test is completed, we get the 500 internal server error.

I looked at using takeException() to ignore these, but it doesn't work.

leancodepl / patrol