Plan to add support for integration_test package

[elmickey]

I don't know if the question has already been answered but is there any plan to support integration_test. For me it would help to write steps and help the execution time. The last time, i tried to use gherkin with integration_test was the fact that gherkin needs to read the features files on disk which is incompatible with the way integration_test is working.

[jonsamwell]

I don't have any experience with that package, what would be the benefit?

[elmickey]

I may be wrong but it's the new recommended way to write integration tests https://flutter.dev/docs/testing/integration-tests . But for me the benefits are :

• same syntax as widget test
• can have access to the internal state of the app (would help to write tests which are right now impossible to write with fluttter driver)
• faster tests
• tests can run on firebase test labs

[vrnvorona]

Second this, currently researching what that package does, but it may be solution for a lot of driver problems with same syntax as WT.

[jonsamwell]

@vrnvorona please share your findings. I will have to look into it as well but if it is the way the Flutter team suggest doing integration tests they will probably eventually deprecate the driver. I won't have any time to look at this before the new year unfortunately. So if anyone want to do a proof of concept that would be awesome

[vrnvorona]

@jonsamwell We now need target and driver files, target has main() function with groups and tests. In each test we need to call main() of our app without enabled extensions and not as pumpWidget(app) (otherwise all exceptions of app will break tests). All waits have to be done through pump/pumpAndSettle. All actions are as in WidgetTests.

target

import 'package:flutter_test/flutter_test.dart';
import 'package:integration_test/integration_test.dart';
import 'package:your_app/main.dart' as app;

void main() {
  IntegrationTestWidgetsFlutterBinding.ensureInitialized();
  group('name', () {
    testWidgets('name', (WidgetTester tester) async {
      app.main();
      await tester.pumpAndSettle();
    });
  } 

driver

import 'package:integration_test/integration_test_driver.dart';

Future<void> main() => integrationDriver();

I can't really tell more on top of this, i tried poking source code of your package, but I lack skills to do it properly as well as understanding in how it all worked before/how it supposed to work now.

I also kinda wonder, what are your plans for after new year? And if you plan to support new package, do you have any estimates for how long before it may be done?

[jonsamwell]

@vrnvorona I have to investigate this to see if it is possible as I think the feature files will need to be assets on the device as at the moment this library is a parent process that invokes the flutter driver process so it is able to read the feature files off disk. The actual swapping of the basic driver commands such as driver.waitFor -> tester.pumpAndSettle() isn't too bad, I will just put a facade over the mechanism to instrument the app and then either the driver or this new integration-tests package can be used.

If this is the way the future of e2e tests are envisaged in flutter then this lib will support it (if possible).

I would be unable to give a timescale for this unfortunately, I have a full time job, and other interests so it is a case of getting some free time but I will try.

[jonsamwell]

I've had a quick look at this and it seems it may be quite hard as the test file is basically launched as the app (as the test file then invokes the app's main method). Therefore, there would be no way for the gherkin lib to read the feature files off the disk. My first though would be to do some sort of code generation which turns the feature files into a single file that can be used for testing. The other method would be to make the feature files assets of the app but that seems wrong. I need to play around with these ideas more.

The actual conversion from the driver api to the tester api won't be hard so I'm confident that is doable.

@elmickey @vrnvorona any thoughts on this?

[jonsamwell]

Hi All,

I have done a very rough first pass at using integration_test. It would be great to get some feedback. See this branch https://github.com/jonsamwell/flutter_gherkin/tree/integration_test__package_support

Look at the changelog for instruction on how to get started and some explanation of the approach.

https://github.com/jonsamwell/flutter_gherkin/blob/integration_test__package_support/CHANGELOG.md

Basically as the tests now need to be in the deployed app I have had to use code generation to create the tests at development time so they can be run without need to access the file system etc. There are still some challenges to overcome such as reporting as the build in flutter_test reporter seems absolutely useless. So I am not quite sure how to get the report data from the app to another source. Any idea would be welcome.

Let me know how you get on and any thoughts, suggestions or PR's would be great.

[clpo]

@jonsamwell I've had an opportunity to make use of your integration_test branch. Seems to work great locally but I haven't had a chance to send my app and tests to firebase yet, but can expect it to work. Logging seems fine as well. Nice work! I'll be working more with this next week and i'll let you know if i experience any issues.

[jonsamwell]

@clpo many thanks for the feedback. I'd be interested to see what happens when testing in firebase. I need to get this setup too. I think the biggest problem when testing on firebase will be the reports. I am not sure how to save reports outside of the standard fluttter_test reporting.

On that point I have made the JsonReporter work today when running from flutter drive so update and have a go. You will need to regenerate your tests as well if you do this.

On another note, I suspect there will be some tweaks needed when we start using this in in replace of flutter driver so please add and and all issues you come across.

[clpo]

@jonsamwell i've had chance to run the tests through firebase test labs and it seems to be having issues initiating the tests. The app launches but there is no mention of the driver starting and no interaction is made with the app. I'm not certain if this is a configuration mistake on my part, but this issue does not occur locally. I looked through the logs provided by test labs but i cant pick any particular thing out which would make sense as the cause of the issue. This is just an FYI really.

[vrnvorona]

@jonsamwell regarding reporting, what if we just generate additional steps for reporting purposes? Because approach sounds really different from what was with flutter_driver, I had this blunt stupid idea :)

I will refine my comment when I try your work a little bit

[jonsamwell]

@clpo thanks for trying with firebase. At that stage it is just a normal integration_test app but something might be happening with the way the reporters are doing stuff. I'll try and get a simple test app running in firebase to see what is going on.

@vrnvorona yes, I originally had this idea too so each step would be a test but you cannot have any test calls within a testWidget block unfortunately. I think the reporting is fine locally or on a build server that uses the flutter drive command. I think the problem comes when running against device farm type things like firebase.

[vrnvorona]

@jonsamwell If i understand correctly, we are still using driver exposed methods instead of WidgetTester? I have yet to successfully migrate my integration_test tests onto new flutter_gherkin since i have no idea how to use pump in it etc. I have quite a while of endless animations i need to skip, and waitForAppToSettle() doesn't work same way pumpAndSettle() does. Plus code is not reusable with WidgetTests because we use Driver again 🤔

[jonsamwell]

No it is all using WidgetTester. I had to do an adapter so that the same api could be used for the flutter driver implementations as the WidgetTetser. Flutter driver is only used to actual start the test app. Once the app is under test the appDriver property on the world object uses WidgetTester.

The adapter api is still under flux so of you need something added to it let me know.

See https://github.com/jonsamwell/flutter_gherkin/blob/integration_test__package_support/lib/src/flutter/adapters/widget_tester_app_driver_adapter.dart

On Tue, Jan 12, 2021, 22:02 Vladislav Voronin notifications@github.com wrote:

@jonsamwell https://github.com/jonsamwell If i understand correctly, we are still using driver exposed methods instead of WidgetTester? I have yet to successfully migrate my integration_test tests onto new flutter_gherkin since i have no idea how to use pump in it etc. I have quite a while of endless animations i need to skip, and waitForAppToSettle() doesn't work same way pumpAndSettle() does. Plus code is not reusable with WidgetTests because we use Driver again 🤔

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jonsamwell/flutter_gherkin/issues/101#issuecomment-758580699, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA4F7ILNWKVKLJDL5PCSG3LSZQT5XANCNFSM4UUB6Y4A .

[vrnvorona]

@jonsamwell Is it possible/viable to just expose WidgetTester through world/context? Or it is bad idea? For me the most appealing part of new package aside from access to better api (like keyboard event buttons) is code similarity with WidgetTests themselves, possibly allowing to write tests for E2E and Widgets faster than with two different approaches.

[vrnvorona]

Btw while i was investigating, found little errors in changelog, here is PR https://github.com/jonsamwell/flutter_gherkin/pull/109 (for md file lmao but still) so others who use it won't have to debug.

[jonsamwell]

@vrnvorona many thanks for the PR much appreciated! I'll get that merged in.

The WidgetTester is exposed via the world object, you can use world.appDriver.rawDriver however, this is not typed to WidgetTester so I'll see if I can type it. (see https://github.com/jonsamwell/flutter_gherkin/blob/integration_test__package_support/lib/src/flutter/adapters/app_driver_adapter.dart#L21)

Sorry for lack of documentation around this, once we have things working I'll focus on API documentation and updating the readme etc

[vrnvorona]

So i can assume rawDriver as WidgetTester and work with it same way? Thanks, i cam to this conclusion too when checking adapter files, but didn't verify it yet mostly because had issues with app starting.

Many thanks again for quick work on adaptation.

[jonsamwell]

Yes this is safe to assume, I will try and get a World with strongly typed adapter so intellisense will work on it

[vrnvorona]

@jonsamwell I have this step with custom timeout (because 10 secs is not enough), but when i run step, i get null config and it uses default duration.

when1<int, FlutterWorld>('I wait {int} seconds with animation', (duration, context) async {
          final tester = context.world.appDriver.rawDriver as WidgetTester;
          try {
            await tester.pumpAndSettle(
                const Duration(milliseconds: 100), EnginePhase.sendSemanticsUpdate, Duration(seconds: duration));
            // ignore: avoid_catching_errors
          } on FlutterError {
            // pump for 2 seconds and stop
          }
        }, configuration: StepDefinitionConfiguration()..timeout = const Duration(minutes: 5)),

I checked StepDefinitionGeneric constructor and found that when steps are declared, they have config, but if i check run function they don't. In runStep(), because there they also all have null config, but i didn't yet narrow problem any further.

[jonsamwell]

Thanks @vrnvorona I'll take a look at this today

[vrnvorona]

Thanks @vrnvorona I'll take a look at this today

Found that issue was not in that.

I have two declared step defs: "I wait {int} seconds" and "I wait {int} seconds with animation" (one is looping pump, other is puming with set timeout, then catches FlutterError so i can wait some time with infinite animation), and firstMatch method returns "I wait {int} seconds" instead of "I wait {int} seconds with animation" despite my feature file and generated code having runStep("I wait 5 seconds with animation").

Not entirely correct behaviour. Sorry for false lead above.

When i commented "I wait {int} seconds" step def, everything is working fine, even the original issue with timeout was cause by infinite pumpAndSettle() because wrong step was used. I need to be more careful with debugging :(

Update: fixed by using regexp with $ on end. For some reason ^ as for beginning of string doesn't work.

[vrnvorona]

@jonsamwell btw, did you notice that tests now run from root folder or it's something with my setup? I tried to use json reporter and noticed that CWD is always '/' while tests are running. Which ofc breaks './report.json' thing

[jonsamwell]

@vrnvorona Yes I couldn't replicate your issue with the timeout so glad you fixed it. I actually added a test in for that in the example (see https://github.com/jonsamwell/flutter_gherkin/blob/integration_test__package_support/example_with_integration_test/integration_test/gherkin/steps/when_await_animation.dart)

I am not sure what you mean about the CWD but any reporter that writes to the file system will not work with this as the tests are actually run on the app and not the host, so the app will not be able to save any files outside of the app file system drive. As a work around the JsonReporter can now be serialise the report and send it over the wire. See the example but you need to make the write operation on the JsonReporter a noop (https://github.com/jonsamwell/flutter_gherkin/blob/integration_test__package_support/example_with_integration_test/integration_test/gherkin/configuration.dart#L35) then the results will be sent back to the host and saved as here (https://github.com/jonsamwell/flutter_gherkin/blob/integration_test__package_support/example_with_integration_test/test_driver/integration_test_driver.dart#L22)

Perhaps you are not using the latest commit to this branch? Add this to your pubspec.yaml:

  flutter_gherkin:
    git:
      url: https://github.com/jonsamwell/flutter_gherkin.git
      ref: 230a8e8d5fcc86d8544b9a196f611a375bab4a0c

[jonsamwell]

I have this new branch running for a really complex suite of 100 test in a real life application now and it works really well and it is fast and more stable that the flutter_driver implementation. There is still quite a bit of work to do updating the documentation but I think this is pretty much code complete now.

[jonsamwell]

Also with the latest the appDriver property and rawDriver properties are now correctly typed to either WidgetTester or FlutterDriver. If you use WidgetTester your world needs to extend FlutterWidgetTesterWorld (https://github.com/jonsamwell/flutter_gherkin/blob/integration_test__package_support/lib/src/flutter/world/flutter_widget_tester_world.dart#L9). On a side note I have been able to do over 100 complex tests without the need to use the WidgetTester directly but relying on the AppDriverAdapter instance on the world context. I think this is best practice for flexibility and code correctness.

Scenarios are also correctly filtered by tags when run now

[vrnvorona]

@jonsamwell Thanks for explanation on how reports work, I just couldn't find that report because I totally forgot about that path and was looking for usual report.json in root of project. Now I can find report.

As for rawDriver I will check latest commit, but so far I just did final tester = context.world.appDriver.rawDriver as WidgetTester and it worked.

So far we don't have a lot of tests since we prototype architecture for how we will use them for apps and so far we think that having same syntax as literal WidgetTests can be beneficial (allows us to have all finders in one folder for elements we need etc), but we would adapt if needed.

I am glad to hear that tests work good and better than flutter_driver, very good news.

[vrnvorona]

As for WidgetTester again, i just change when<FlutterWorld> with when<FlutterWidgetTesterWorld> and it's good to go?

[jonsamwell]

Yes that is correct

On Fri, Jan 15, 2021, 21:44 Vladislav Voronin notifications@github.com wrote:

As for WidgetTester again, i just change when with when and it's good to go?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jonsamwell/flutter_gherkin/issues/101#issuecomment-760834457, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA4F7IPRBHOOVA4EGWHXITTS2AL7RANCNFSM4UUB6Y4A .

[vrnvorona]

I am currently debugging report creating after failed tests, do you personally have any problems or your reports are created every time? So far I narrowed it down to tearDownAll not being called for some reason, will try with simpler app on monday to verify that it's not app problem (if it isn't). Just asking though, please don't spend much of your time on it.

[jonsamwell]

Hi @vrnvorona,

You won't be able to use reporters that save files to the file system as they actually run on the device now so files would only be on the device. To get around this there is a new interface which indicates the result of a reporter can be serialised to JSON. This is then sent back over the wire by the driver to the host process which then saves it on the host file system (i.e. your computer).

The JsonReporter already supports this see (https://github.com/jonsamwell/flutter_gherkin/blob/integration_test__package_support/example_with_integration_test/integration_test/gherkin/configuration.dart#L34). Also here is the interface you need to implement if you have your own reporter an need to save a file to disk (https://github.com/jonsamwell/dart_gherkin/blob/code_gen_changes/lib/src/reporters/serializable_reporter.dart). Note how the JsonReporter implements it https://github.com/jonsamwell/dart_gherkin/blob/code_gen_changes/lib/src/reporters/json/json_reporter.dart#L104. When all tests have run the runner will look over all the reporters and serialise the results (https://github.com/jonsamwell/flutter_gherkin/blob/integration_test__package_support/lib/src/flutter/runners/gherkin_integration_test_runner.dart#L76). Which is then sent back to the driver app and saved (https://github.com/jonsamwell/flutter_gherkin/blob/integration_test__package_support/example_with_integration_test/test_driver/integration_test_driver.dart#L22). I hope this makes sense

[vrnvorona]

@jonsamwell Yes, that i already understand, so far i use JsonReporter only. When my tests fail, it won't save report, while if test passes it does generate file in the path which is set up in integration_test.dart.

To test this i used example app in integration_test folder of plugins repo. I created simple step, with expect(2 + 2, equals(4));. If expect is true, there is generated JSON. If i set it to 5 to emulate failed assertion), there is no JSON despite await writeReport(report, path); function being called in (https://github.com/jonsamwell/dart_gherkin/blob/code_gen_changes/lib/src/reporters/json/json_reporter.dart#L64)

[vrnvorona]

@jonsamwell I found that responseDataCallback is not called if tests fail (https://github.com/flutter/plugins/blob/faa26ec364cd6d3c738d73526ea55a95b7f4ab1a/packages/integration_test/lib/integration_test_driver.dart#L82). Though response.data is null when they fail, so i still can't get report with info after failed tests.

[vrnvorona]

For some reason tearDownAll() is called later than responseDataCallback if test fails, because of it testRunner.reportData is null. I will keep trying to debug it.

[vrnvorona]

https://github.com/flutter/flutter/issues/74324 https://github.com/flutter/flutter/issues/74391 reported both issues i've encountered with reporting so far. Also preparing PR with new reporter, because current JSONreporter is not in cucumber format (and it is even saving in string with a lot of escaped quotes, which is no really a JSON).

[jonsamwell]

@vrnvorona awesome thanks for reporting these. Funny enough I was trying to solve all these issues today. I got the reporting working but only if all tests pass. We should add another bug (which really bugs me) is that if a single test fails it will stop the whole test run which is not really good behaviour.

Also please see my latest commit today (https://github.com/jonsamwell/flutter_gherkin/commit/ef04fa338678ffb59833ebf6affdff2189a8a21c) which fixes the reports not getting created if a test fails.

This bit of code will right out the individual reports and sort out the JSON issue you encountered

import 'dart:convert';
import 'dart:io';

import 'package:flutter_driver/flutter_driver.dart';
import 'package:path/path.dart' as path;
import 'package:integration_test/integration_test_driver.dart'
    as integration_test_driver;

const JsonEncoder _prettyEncoder = JsonEncoder.withIndent('  ');

String _encodeJson(Object jsonObject, bool pretty) {
  return pretty ? _prettyEncoder.convert(jsonObject) : json.encode(jsonObject);
}

DriverLogCallback logDriverMessages = (String source, String message) {
  final msg = '$source:: $message';
  if (message.toLowerCase().contains('error')) {
    stderr.writeln(msg);
  } else {
    stdout.writeln(msg);
  }
};

Future<void> writeGherkinReports(
  Map<String, dynamic> data, {
  String testOutputFilename = 'integration_response_data',
  String destinationDirectory,
}) async {
  await integration_test_driver.writeResponseData(
    data,
    testOutputFilename: 'gherkin_reports',
    destinationDirectory: destinationDirectory,
  );

  final reports =
      json.decode(data['gherkin_reports'].toString()) as List<dynamic>;

  for (var i = 0; i < reports.length; i += 1) {
    final reportData = reports.elementAt(i) as List<dynamic>;

    await fs
        .directory(integration_test_driver.testOutputsDirectory)
        .create(recursive: true);
    final File file = fs.file(path.join(
      integration_test_driver.testOutputsDirectory,
      'report_$i.json',
    ));
    final String resultString = _encodeJson(reportData, true);
    await file.writeAsString(resultString);
  }
}

Future<void> main() {
  // Flutter Driver logs all messages to stderr by default so if this is run on a build server
  // the process will fail due to writing errors. So handle this yourself for now
  driverLog = logDriverMessages;
  // The Gherkin report data send back to this runner by the app after
  // the tests have run will be saved to this directory
  integration_test_driver.testOutputsDirectory = 'test/integration/reports';

  return integration_test_driver.integrationDriver(
    timeout: const Duration(minutes: 120),
    responseDataCallback: writeGherkinReports,
  );
}

[vrnvorona]

@jonsamwell Thanks for your work. But i have issue with this line final String resultString = _encodeJson(reportData, true);, The function '_encodeJson' isn't defined (not a surprise, since it's function from integration_test_driver.dart, and it is private). 🤔

Am I doing something wrong?

[jonsamwell]

@vrnvorona sorry I missed that bit, the above code has been updated

[vrnvorona]

@jonsamwell Hm, i tried code above and new commit and when it fails it won't create file (well, not a surprise because responseDataCallback is not called when tests fail, and if it is called then it's still null and i get exception:

Unhandled exception:
NoSuchMethodError: The method '[]' was called on null.
Receiver: null
Tried calling: []("gherkin_reports")

[vrnvorona]

Btw, do we support/possible to do nested steps? Ruby has it like this:

Given /two turtles/ do
  steps %{
    Given a turtle
    And a turtle
  }
end

[jonsamwell]

@vrnvorona I've got no plans to implement this but PR's are welcome

[vrnvorona]

Also regarding issue with reports being created: what if we try cucumber messages approach and send it all to stdout and redirect it to ndjson file, for later parsing? Not exactly asking to do it, just thinking of approaches. Also could help with, for example, attachments for reports.

[jonsamwell]

@vrnvorona that is basically what is happening now

[vrnvorona]

Is it? AFAIK, currently we edit binding data for integration_test to use it to generate file later, stdout is just short messages with info for console view, while it's possibly plausible to use stdout optionally as reporter, collect all messages in NDJSON file and use external json-reporter without any reliance on integration_test to create file.

The issue though is to basically silence both app under test and flutter, though it's not really hard to remove non-json objects from ndjson file.

[jonsamwell]

@vrnvorona what is NDJSON. The trouble is this all executes on the phone . Currently the app is sending back some json which is saved to disk just as you said. I would be hesitant to pollute the stdout. The JSONReporter is working the way it is currently implemented which is basically doing what you said (apart from writing to stdout).

[vrnvorona]

Well ruby-cucumber doesn't hesitate to pollute stdout for some reason

You just collect with > into file and then parse it.

[jonsamwell]

Sorry I've got no experience with that. Feel free to put together a pr and we can go from there 🙂

On Thu, Feb 11, 2021, 22:10 Vladislav Voronin notifications@github.com wrote:

Well ruby-cucumber doesn't hesitate to pollute stdout for some reason

You just collect with > into file and then parse it.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jonsamwell/flutter_gherkin/issues/101#issuecomment-777371328, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA4F7IM5BY2WNYBZJJXDIALS6O3MDANCNFSM4UUB6Y4A .

[jonsamwell]

Aiming to get this publish next week

[juanagu]

News?