`patrol test` reports tests as passed even when native automation is not set up at all

[bartekpacia]

First discovered by @KmlBaranowski, reported to me by @jBorkowska.

Describe the bug

When the developer has not completed the necessary steps here, patrol test wrongly reports that all tests have passed, when in reality, no tests are run at all.

To Reproduce Steps to reproduce the behavior:

1. flutter create a new app, add integration_test package, add a simple test in the integration_test
2. patrol test

Output:

✓ Built and ran apk for widget_test.dart on emulator-5554 (5.8s)
 PASS  widget_test.dart on emulator-5554

Expected behavior

Tests should fail.

Device information

• Device: Android
• Does it also both on physical and virtual devices? Yes, doesn't matter

Additional information

``` Flutter 3.3.10 • channel stable • https://github.com/flutter/flutter.git Framework • revision 135454af32 (6 weeks ago) • 2022-12-15 07:36:55 -0800 Engine • revision 3316dd8728 Tools • Dart 2.18.6 • DevTools 2.15.0 ``` patrol_cli v0.8.4

Additional context

The root cause is that the Gradle :app:connectedDebugAndroidTest task succeeds even though it is not set up at all. It acts as if it was a no-op.

[mzdm]

happens also to me

[MarcinHradowicz]

I have a similar issue patrol test doesn't run any tests but shows that they passed. 🤔 patrol 0.10.11 patrol_cli 0.9.3 flutter 3.3.7 flutter test or flutter drive commands work fine.

[bartekpacia]

@MarcinHradowicz Did you set up native automation feature according to docs? If not, then you're indeed facing this issue.

But if you have set it up and patrol test still doesn't run tests but only reports them as passed, then it's a separate issue - please create a new issue in this case and provide reproduction steps :)

[MarcinHradowicz]

Did you set up native automation feature according to docs? If not, then you're indeed facing this issue.

@bartekpacia I think I solved that. In the MainActivityTest.java I had package com.myApp.stg and it turned out that I have to remove flavor suffix from there so package com.myApp worked fine :D But MainActivityTest.java is still placed in src/androidTest/java/com/myApp/stg/ directory. 😅

[jBorkowska]

We'd like to print a warning when no tests were found (bc it probably means that the native setup is wrong).

[jBorkowska]

Size: need to research