Warn users about common Patrol misconfiguration problems

[bartekpacia]

The problem

Developers often have problems with setting up Patrol in their apps.

Possible solutions

Because of the complexity and dynamic nature of mobile development tooling (Gradle build scripts, a myriad of ways to configure Xcode projects, medium-size and large-size apps having custom project layouts, etc.), it's impossible to introduce a patrol bootstrap-like command that would perform the set up.

What we can try doing instead is developing heuristics-based checks that would warn developers about common Patrol misconfiguration problems. When a developer were to run patrol test (or patrol build, or any other command that requires native setup), the checks would be performed.

(Most of) the checks would be warnings, not errors. As noted above, mobile projects, especially cross-platform ones, have lots of moving parts. We can't reliably detect all misconfiguration problems, but there are a few common patterns we can at least try to detect (thus heuristics). We don't want developers to be unable to use Patrol if one of the heuristics gives false positives.

List of Android heuristics

1. File MainActivityTest.java not found

   Inputs: app package name (com.example.myapp) - we could get it from the patrol section in pubspec.yaml Action: check if file MainActivityTest.java exists in android/app/src/androidTest/java/com/example/myapp

List of iOS heuristics

1. Target RunnerUITests doesn't exist

   Inputs: none Actions: check if file ios/RunnerUITests/RunnerUITests.m exists

Alternatives

Another way to reduce configuration-related churn would be to make patrol develop not require configuration at all. Native configuration would still be required to get test reports and to run on devicefarms, though.

[bartekpacia]

Please comment on this issue to add ideas for more heuristics.

[Albert221]

When a developer were to run patrol test (or patrol build, or any other command that requires native setup), the checks would be performed.

Yes, absolutely. I think that would reduce the number of people that don't understand why native integration doesn't work significantly while still offering a good UX in the CLI.

it's impossible to introduce a patrol bootstrap-like command that would perform the set up.

I don't agree. Maybe for more complex projects it would indeed be very tough, and then we could show a message that a setup requires manual work. But for simpler projects (that don't have any changes in their native files or if some basic criteria are met) such bootstraping could be helpful, for fresh projects (like PoC, checking how patrol works) or for basic projects, created by less experienced devs (that would greater benefit from such bootstrap)

[bartekpacia]

Maybe for more complex projects it would indeed be very tough, and then we could show a message that a setup requires manual work. But for simpler projects (that don't have any changes in their native files or if some basic criteria are met) such bootstraping could be helpful, for fresh projects (like PoC, checking how patrol works) or for basic projects, created by less experienced devs (that would greater benefit from such bootstrap)

That'd be hard to develop, and hard to test across different Flutter versions.

I prefer no automatic patrol bootstrap at all than a patrol bootstrap that is flaky and doesn't work except for fresh, simple, flutter created counter app. Then the advice we'd give to our users, who will complain that "patrol bootstrap doesn't work", would probably be "don't use patrol bootstrap", and it just won't look good.

Also, I'm pretty certain we don't have the resources to maintain the complexity that such patrol bootstrap feature would add. I'd also argue that the benefit/cost ratio would be too small to rationalize developing it, since native setup happens once for a project, and we have many more problems to work on.

[neiljaywarner]

@bartekpacia i am not sure if this warrants another issue, but i'll be honest, at this point I'm confused whether or not we can use patrol WITHOUT native automatin in all widget and integration tests, ie 1) with flutter test instead of patro ltest 2) with coverage 3) with.. well. i hate to say it but "less issues". we are having some problems still where tests pass in patrol test but fail in flutter logs

[bartekpacia]

Hi @neiljaywarner,

Patrol has 2 features: custom finders and native automation. You can use one of them, or both.

If you want to only use custom finders ("the dollar" – $), you can definitely do this. What problem are you experiencing?

we are having some problems still where tests pass in patrol test but fail in flutter logs

This sounds like a bug - and one that we don't experience ourselves. It'd be nice if you created a new issue and shared a simple app that reproduces the problem.

[bartekpacia]

Minimal version: create a page in docs about these common problems

[fylyppo]

Maybe it'll be good to warn users about incompatible:

• Gradle + Java versions https://docs.gradle.org/current/userguide/compatibility.html
• Java class versions e.g when you try to load something compiled for Java 6 in a Java 5 runtime