search

leancodepl / patrol

Flutter-first UI testing framework. Ready for action!
https://patrol.leancode.co
Apache License 2.0
854 stars 127 forks source link

Tool Failures: Instrumentation did not complete #2082

Open nc-jeni opened 7 months ago

nc-jeni commented 7 months ago

Steps to reproduce

I am unable to find the output of the tests. After I have ran the tests, it just says the tests have been executed successfully regardless of the results of the individual tests even that I know that some have failed and expect to see something like this.

[PASS] Test 1
[FAIL] Test 2
[PASS] Test 3
[PASS] Test 4

I just started the test by patrol test or patrol test --flavor development and waited for the the result of the test such as

group('Dummy tests =>', ()
{
    patrolWidgetTest(
      'counter state is the same after going to home and switching apps',
      config: const PatrolTesterConfig(visibleTimeout: Duration(seconds: 10)),
          ($) async {
        await $.pumpWidgetAndSettle(
          MaterialApp(
            home: Scaffold(
              appBar: AppBar(title: const Text('app')),
              backgroundColor: Colors.blue,
            ),
          ),
        );
        expect($('app'), findsOneWidget);
      },
    );
});

Here goes my MainActivityTest class placed in \android\app\src\androidTest\kotlin\<applicationId> while the MainActivity class is placed in \android\app\src\main\kotlin\<applicationId>. Both of the classes are Kotlin. Secondly, I am unable to use a Java implementation of the MainActivityTest class as provided in your sample app hence it cannot find the Kotlin implementation of the MainActivity class.

package <applicationId>

import androidx.test.rule.ActivityTestRule
import dev.flutter.plugins.integration_test.FlutterTestRunner
import org.junit.Rule
import org.junit.runner.RunWith

@RunWith(FlutterTestRunner::class)
class MainActivityTest {
    @get:Rule
    val rule = ActivityTestRule(MainActivity::class.java, true, false)
}

I am expecting one or more of the following output options

It is also related to: https://github.com/leancodepl/patrol/issues/2066, https://github.com/flutter/flutter/issues/62237, https://github.com/leancodepl/patrol/issues/1170

Note: I am not sure whether it is because it does not produce any output, or because no tests are found even tests actually exists.

Actual results

Patrol Test Output ``` No device specified, using the first one (emulator-5554) • Building apk with entrypoint test_bundle.dart... ✓ Completed building apk with entrypoint test_bundle.dart (40.2s) • Executing tests of apk with entrypoint test_bundle.dart on emulator-5554... ✓ Completed executing apk with entrypoint test_bundle.dart on emulator-5554 (8.8s) ```

Logs

LogCat ``` ----- begin exception ----- java.lang.RuntimeException: Unable to run tests due to missing activity rule at dev.flutter.plugins.integration_test.FlutterTestRunner.run(FlutterTestRunner.java:57) at org.junit.runners.Suite.runChild(Suite.java:128) at org.junit.runners.Suite.runChild(Suite.java:27) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.ParentRunner.run(ParentRunner.java:413) at org.junit.runner.JUnitCore.run(JUnitCore.java:137) at org.junit.runner.JUnitCore.run(JUnitCore.java:115) at androidx.test.internal.runner.TestExecutor.execute(TestExecutor.java:67) at androidx.test.internal.runner.TestExecutor.execute(TestExecutor.java:58) at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:446) at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2402) ----- end exception ----- ```

Patrol version

Patrol Doctor output

Patrol Doctor output ``` Patrol doctor: Patrol CLI version: 2.6.1 Android: • Program adb found in AppData\Local\Android\Sdk\platform-tools\adb.exe • Env var $ANDROID_HOME set to AppData\Local\Android\Sdk ```

Flutter Doctor output

Flutter Doctor output ``` [√] Android toolchain - develop for Android devices (Android SDK version 33.0.2) [√] Chrome - develop for the web [!] Visual Studio - develop Windows apps (Visual Studio Build Tools 2022 17.6.2) X Visual Studio is missing necessary components. Please re-run the Visual Studio installer for the "Desktop development with C++" workload, and include these components: MSVC v142 - VS 2019 C++ x64/x86 build tools - If there are multiple build tool versions available, install the latest C++ CMake tools for Windows Windows 10 SDK [√] Android Studio (version 2022.2) [√] IntelliJ IDEA Community Edition (version 2022.3) [√] VS Code (version 1.77.3) [√] Connected device (4 available) [√] Network resources ```

Please note that the issue from flutter doctor (see above) can be ignored hence I use Android Studio and not Visual Studio.

UPDATE

I tried to change the runner 

// testInstrumentationRunner "pl.leancode.patrol.PatrolJUnitRunner"
testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner"

Plus adding the dependencies

androidTestImplementation  'com.android.support.test:rules:0.5'
androidTestImplementation  'com.android.support:support-annotations:25.2.0'
androidTestImplementation  'com.android.support.test:runner:0.5'

This gives the following error that boils down to stack trace with roots in Patrol

StackTrace from LogCat ``` ✗ Failed to execute tests of apk with entrypoint test_bundle.dart on emulator-5554 (Gradle test execution failed with code 1) (15.9s) Exception: Gradle test execution failed with code 1 #0 AndroidTestBackend.execute. (package:patrol_cli/src/android/android_test_backend.dart:140:9) #1 DisposeScope.run (package:dispose_scope/src/dispose_scope.dart:46:7) #2 AndroidTestBackend.execute (package:patrol_cli/src/android/android_test_backend.dart:106:5) #3 TestCommand._execute (package:patrol_cli/src/commands/test.dart:309:7) #4 TestCommand.run (package:patrol_cli/src/commands/test.dart:201:23) #5 CommandRunner.runCommand (package:args/command_runner.dart:212:13) #6 PatrolCommandRunner.runCommand (package:patrol_cli/src/runner/patrol_command_runner.dart:341:18) #7 PatrolCommandRunner.run (package:patrol_cli/src/runner/patrol_command_runner.dart:285:18) #8 patrolCommandRunner (package:patrol_cli/src/runner/patrol_command_runner.dart:70:20) #9 main (file:///C:/Users/jen/AppData/Local/Pub/Cache/hosted/pub.dev/patrol_cli-2.6.2/bin/main.dart:6:20) ```

The reason for changing the runner away from PatrolJUnitRunner was that it was unable to run tests due to missing activity rule.

testInstrumentationRunner "pl.leancode.patrol.PatrolJUnitRunner"
StackTrace when using PatrolJUnitRunner ``` ----- begin exception ----- java.lang.RuntimeException: Unable to run tests due to missing activity rule at dev.flutter.plugins.integration_test.FlutterTestRunner.run(FlutterTestRunner.java:57) at org.junit.runners.Suite.runChild(Suite.java:128) at org.junit.runners.Suite.runChild(Suite.java:27) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.ParentRunner.run(ParentRunner.java:413) at org.junit.runner.JUnitCore.run(JUnitCore.java:137) at org.junit.runner.JUnitCore.run(JUnitCore.java:115) at androidx.test.internal.runner.TestExecutor.execute(TestExecutor.java:67) at androidx.test.internal.runner.TestExecutor.execute(TestExecutor.java:58) at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:446) at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2402) ----- end exception ----- ```

Sticking to the PatrolJUnitRunner by using the Java version of the MainActivityTest and porting the MainActivity to Java gives the following StackTrace and hangs forever. However, the binding exception was solved by removing IntegrationTestWidgetsFlutterBinding.ensureInitialized(); in top of the main() method in the integration test.

StackTrace when MainActivity and MainActivityTest is Java and PatrolJUnitRunner is used ``` E [ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: 'package:flutter/src/foundation/binding.dart': Failed assertion: line 153 pos 12: '_debugInitializedType == null': Binding is already initialized to PatrolBinding #0 _AssertionError._doThrowNew (dart:core-patch/errors_patch.dart:51:61) #1 _AssertionError._throwNew (dart:core-patch/errors_patch.dart:40:5) #2 new BindingBase (package:flutter/src/foundation/binding.dart:153:12) #3 new _TestWidgetsFlutterBinding&BindingBase&SchedulerBinding (package:flutter_test/src/binding.dart) #4 new _TestWidgetsFlutterBinding&BindingBase&SchedulerBinding&ServicesBinding (package:flutter_test/src/binding.dart) #5 new _TestWidgetsFlutterBinding&BindingBase&SchedulerBinding&ServicesBinding&GestureBinding (package:flutter_test/src/binding.dart) #6 new _TestWidgetsFlutterBinding&BindingBase&SchedulerBinding&ServicesBinding&GestureBinding&SemanticsBinding (package:flutter_test/src/binding.dart) #7 new _TestWidgetsFlutterBinding&BindingBase&SchedulerBinding&ServicesBinding&GestureBinding&SemanticsBinding&RendererBinding (package:flutter_test/src/binding.dart) #8 new _TestWidgetsFlutterBinding&BindingBase&SchedulerBinding&ServicesBinding&GestureBinding&SemanticsBinding&RendererBinding&PaintingBinding (package:flutter_test/src/binding.dart) #9 new _TestWidgetsFlutterBinding&BindingBase&SchedulerBinding&ServicesBinding&GestureBinding&SemanticsBinding&RendererBinding&PaintingBinding&WidgetsBinding (package:flutter_test/src/binding.dart) #10 new _TestWidgetsFlutterBinding&BindingBase&SchedulerBinding&ServicesBinding&GestureBinding&SemanticsBinding&RendererBinding&PaintingBinding&WidgetsBinding&TestDefaultBinaryMessengerBinding (package:flutter_test/src/binding.dart) #11 new TestWidgetsFlutterBinding (package:flutter_test/src/binding.dart) #12 new LiveTestWidgetsFlutterBinding (package:flutter_test/src/binding.dart) #13 new IntegrationTestWidgetsFlutterBinding (package:integration_test/integration_test.dart) #14 IntegrationTestWidgetsFlutterBinding.ensureInitialized (package:integration_test/integration_test.dart:154:7) #15 main (file:///.../integration_test/app_test.dart:48:40) #16 Declarer.group. (package:test_api/src/backend/declarer.dart:267:37) #17 _rootRun (dart:async/zone.dart:1399:13) #18 _CustomZone.run (dart:async/zone.dart:1301:19) #19 _runZoned (dart:async/zone.dart:1804:10) #20 runZoned (dart:async/zone.dart:1747:10) #21 Declarer.declare (package:test_api/src/backend/declarer.dart:169:7) #22 Declarer.group (package:test_api/src/backend/declarer.dart:264:14) #23 group (package:flutter_test/src/test_compat.dart:189:13) #24 main (file:///.../integration_test/test_bundle.dart:69:3) ```

SOLVED

  1. Keep MainActivityTest in Java at android\app\src\androidTest\java\<applicationId>
package <applicationId>;

import androidx.test.platform.app.InstrumentationRegistry;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
import org.junit.runners.Parameterized.Parameters;
import pl.leancode.patrol.PatrolJUnitRunner;

@RunWith(Parameterized.class)
public class MainActivityTest {
    @Parameters(name = "{0}")
    public static Object[] testCases() {
        PatrolJUnitRunner instrumentation = (PatrolJUnitRunner) InstrumentationRegistry.getInstrumentation();
        instrumentation.setUp(MainActivity.class);
        instrumentation.waitForPatrolAppService();
        return instrumentation.listDartTests();
    }

    public MainActivityTest(String dartTestName) {
        this.dartTestName = dartTestName;
    }

    private final String dartTestName;

    @Test
    public void runDartTest() {
        PatrolJUnitRunner instrumentation = (PatrolJUnitRunner) InstrumentationRegistry.getInstrumentation();
        instrumentation.runDartTest(dartTestName);
    }
}
  1. Rewrite MainActivity in Java at android\app\src\main\java\<applicationId>
  2. Removes MainActivity in Kotlin that is placed at android\app\src\main\kotlin\<applicationId>

Kotlin approach does not compute!

  1. Adds the following to the dependencies section in build.gradle
dependencies {
    // ...
    androidTestUtil 'androidx.test:orchestrator:1.4.2'
}
  1. Adds the following to the defaultConfig section in build.gradle
defaultConfig {
    // ...
    testInstrumentationRunner "pl.leancode.patrol.PatrolJUnitRunner"
    testInstrumentationRunnerArguments clearPackageData: "true"
}

  1. Avoid calling IntegrationTestWidgetsFlutterBinding.ensureInitialized(); in main()

  2. Find the result at build\app\reports\androidTests\connected\index.html as expected, but not in console when all tests pass. However, when failures it is shown with a link to the test overview, but when all tests are passed it will be appreciated if it is visible upfront that they have been found and executed with success.

Note: A better guide that more strictly describes how to setup, and clearly describes what does not work is appreciated. Especially, the Java vs. Kotlin approach as it is pretty confusing hence your sample app mixes those even that it does work. Feel free to use the steps I have outlined above for inspiration to an improved guide.

piotruela commented 6 months ago

Hi @nc-jeni. Sorry for the late reply. I'm glad that you managed to fix your issue. I see two action points based on what you've written:

SantoshGunashekar-Freo commented 1 month ago

Makes it more clear in docs how to setup the patrol in project when main android file is MainActivity.kt. So let's use this issue to track the second one.

Does this require additional change?