Flutter Support - Githubissues

charlesbensonsonos commented 5 years ago

With the up and coming Flutter.io, some apps may be 'hybrid' with both native and Flutter UI scenarios to support and test. Curious if anyone has thought about EarlGrey in this world and the support for bringing visibility of both worlds into the library.

bootstraponline commented 5 years ago

FlutterDriver is the recommended way for writing Flutter app UI tests.

https://flutter.io/docs/testing

khandpur commented 5 years ago

We are working to support the Flutter use-case. Because of the A11Y bindings that flutter provides, EG 2 (tested) worked fine locating and interacting with the elements however there are major synchronization gaps which makes some tests as much as 90% flaky. We will be working with them to add proper synchronization support next quarter.

bootstraponline commented 5 years ago

@khandpur will we be able to use EarlGrey 2 from Dart?

khandpur commented 5 years ago

That's still being debated. However, you should be able to do the opposite: Use EG2 to test all of Flutter hybrid apps from within XCTest (swift or objc)

bootstraponline commented 5 years ago

That's still being debated. However, you should be able to do the opposite: Use EG2 to test all of Flutter hybrid apps from within XCTest (swift or objc)

And then duplicate the tests on Android using Espresso? 🤔

I think Flutter devs would prefer to stay in Dart to realize the cross platform benefit. It's similar to the use case of detox for react native.

charlesbensonsonos commented 5 years ago

Thanks for the update and discussion, much appreciated.

When the app flow moves between native and flutter views, something 'all inclusive' would be ideal. Be it in Dart or supported in EarlGrey/Espresso, or all of the above. EarlGrey being more ideal for an app just starting to integrate with Flutter, and Flutter driver ideal for a dominantly Flutter built app. It seems the Flutter test driver is just as blind to the native UI right now. Any angle of attack on this is bound to be a fun one I think.

mehmetf commented 5 years ago

@bootstraponline I agree with the sentiment but we have to look at the use cases.

If your app is mostly Flutter, you likely want to write Flutter Driver tests. Just like your code moves away from ObjC/Swift/Kotlin/Java to Dart, your tests should follow. However, we (Flutter team) acknowledge that there are a lot of use cases where people will still maintain a significant portion of native code along with Flutter code. In that scenario, it is highly likely that you already have native tests so having those tests cover both Native and Flutter portion of the app is the natural way to go.

The only thing that's not on the roadmap is having Flutter Driver drive native apps. Having said that, there are already frameworks that combine different technologies to afford x-platform testing for mobile apps (e.g. http://appium.io/).

bootstraponline commented 5 years ago

there are already frameworks that combine different technologies to afford x-platform testing for mobile apps (e.g. http://appium.io/).

As an appium developer, I'm confident it's not a fit for Flutter. Maybe you could click a button, however without solving the sync issues it's not going to be stable. Appium has the same issue with react native apps.

In that scenario, it is highly likely that you already have native tests so having those tests cover both Native and Flutter portion of the app is the natural way to go.

I'm curious to hear specific examples of users adopting a cross platform technology who then want to write UI tests twice, once per platform. When companies are adopting react native, they'd like to gain the developer productivity win by writing tests in JavaScript for the entire app (see detox).

At Instructure, when we adopted react native, it was a natural choice to try and write tests in JavaScript. Even though we had an existing native app with UI tests, there was value in being able to write UI tests once and in a common language (JavaScript) for both Android/iOS. Unfortunately react native as a technology has issues and we went back to native development.

The only thing that's not on the roadmap is having Flutter Driver drive native apps

I guess my feedback is that should be on the roadmap, and makes more sense than writing ObjC/Swift/Kotlin/Java tests for Flutter UI. Otherwise Flutter testing will be a mix of 3 different approaches (espresso, EarlGrey, Flutter Driver) in various languages instead of simply Dart + Flutter Driver.

If Flutter Driver solved the native automation problem by wrapping EarlGrey/Espresso as detox has done, I'd expect native apps to adopt it who aren't even using Flutter. 🙂

mehmetf commented 5 years ago

We are just starting to understand the add2app (the codename for using Flutter in existing apps) use cases. Our recommendation so far has been:

Write as much Flutter as possible.
Write Flutter Driver tests as much as possible using fakes/mocks for the native integration points.
To test those integration points and get full E2E coverage, use native testing solutions.

We have seen some success with this. Developers generally do not complain about native testing when they already have native code to maintain. They start complaining about it when that's the only way to test a Flutter component but clearly that's not the case because you can do #2. In fact, we have seen more complaints like the one filed here: "Espresso should support Flutter", "XCUITest should support Flutter" etc. There's clearly more appetite for using native testing solutions to drive Flutter rather than a Dart solution driving native.

Perhaps a different landscape will emerge as add2app grows in popularity. We will watch it closely.

PS: I will give you two other foods for thought, these are just my personal opinions (I am not on the Flutter team but I work very closely with them):

Supporting a test framework that drives native apps goes against the way Flutter was designed. We never wanted to "play nice with all platforms", we wanted to revamp them. To provide such a testing tool though, we would have to deeply understand and integrate with the platform detracting us from improving the platform-agnostic Flutter.
Even if such a test framework existed, it would be along the lines of "learn once, write anywhere" because despite your Flutter UI being the same, your native UI would likely differ so you can't quite use the same test code. That goes against Flutter's philosophy of "write once, run everywhere".

Again, I am not saying that what you are proposing is a bad idea. I am just saying it does not align with how Flutter wants to execute its mission. Thus, it might emerge from community but it is unlikely to come from the Flutter team.

bootstraponline commented 5 years ago

Cool. Thanks for the insight. It'll be interesting to see how this evolves in the future.

google / EarlGrey

Flutter Support #778