flutter / flutter

Flutter makes it easy and fast to build beautiful apps for mobile and beyond
https://flutter.dev
BSD 3-Clause "New" or "Revised" License
166.82k stars 27.65k forks source link

[flutter_driver] when running test, capture the state of the application in case of failure #65954

Open dotdoom opened 4 years ago

dotdoom commented 4 years ago

Use case

When running a Flutter Drive test, capture the state of the application in case of failure.

State may include some whitebox data (e.g. retrieved via requestData if connection isn't completely broken), or blackbox data (screenshot).

Especially useful when a flaky test is failing at CI but passing at developer's workstation.

Proposal

Driver exposes an errors stream with error and stack trace information, which can be used like that:

setUpAll(() async {
  driver = await FlutterDriver.connect();
  driver.errors.listen((errorAndStackTrace) async {
    final pixels = driver.screenshot();
    final File file = File('screenshots/${sanitizeForFileName(errorAndStackTrace)}');
    await file.writeAsBytes(pixels);
  });
});
dotdoom commented 4 years ago

Another use case it to be able to interrupt the whole test suite when a single test fails. Oftentimes, drive tests would depend on the result of previous tests because they have to clean the state. When a test fails, it would leave an app in the wrong state (e.g. wrong screen). Advanced tests may reset the state at the beginning, though, but it's not always the case.