search

jonsamwell / flutter_gherkin

A Gherkin parsers and runner for Dart and Flutter which is very similar to cucumber
MIT License
206 stars 113 forks source link

Call tests via `flutter test` instead of `flutter drive` possible? #275

Open golane-august opened 2 years ago

golane-august commented 2 years ago

Can you use integration_test package instead of driver package to test the app?

Use:

flutter test integration_test/gherkin_suite_test.dart

Instead of:

flutter drive --driver=test_driver/integration_test_driver.dart --target=integration_test/gherkin_suite_test.dart

But I get the error with flutter test:

flutter test integration_test/gherkin_suite_test.dart
00:00 +0: loading /Users/august/Coding/git/jonsamwell/flutter_gherkin/example_with_integration_test/integration_test/gherkin_suite_test.dart                                                                                                                                                                           R00:11 +0: loading /Users/august/Coding/git/jonsamwell/flutter_gherkin/example_with_integration_test/integration_test/gherkin_suite_test.dart                                                                                                                                                                      11.7s
✓  Built build/app/outputs/flutter-apk/app-debug.apk.
00:12 +0: loading /Users/august/Coding/git/jonsamwell/flutter_gherkin/example_with_integration_test/integration_test/gherkin_suite_test.dart                                                                                                                                                                           I00:13 +0: loading /Users/august/Coding/git/jonsamwell/flutter_gherkin/example_with_integration_test/integration_test/gherkin_suite_test.dart                                                                                                                                                                      656ms
00:15 +0 -1: loading /Users/august/Coding/git/jonsamwell/flutter_gherkin/example_with_integration_test/integration_test/gherkin_suite_test.dart [E]                                                                                                                                                                    
  Failed to load "/Users/august/Coding/git/jonsamwell/flutter_gherkin/example_with_integration_test/integration_test/gherkin_suite_test.dart": Bad state: Can't call group() once tests have begun running.
  package:test_api                                                                                                                  Declarer.group
  package:flutter_test/src/test_compat.dart 189:13                                                                                  group
  package:flutter_gherkin/src/flutter/runners/gherkin_integration_test_runner.dart 117:5                                            GherkinIntegrationTestRunner.runFeature
  Users/august/Coding/git/jonsamwell/flutter_gherkin/example_with_integration_test/integration_test/gherkin_suite_test.g.dart 34:5  _CustomGherkinIntegrationTestRunner.testFeature0
  Users/august/Coding/git/jonsamwell/flutter_gherkin/example_with_integration_test/integration_test/gherkin_suite_test.g.dart 26:5  _CustomGherkinIntegrationTestRunner.onRun
  package:flutter_gherkin/src/flutter/runners/gherkin_integration_test_runner.dart 94:5                                             GherkinIntegrationTestRunner.run

00:16 +0 -1: (tearDownAll) - did not complete [E]                                                                                                                                                                                                                                                                      
00:16 +0 -1: Some tests failed.

I used the example in integration_test__package_support: https://github.com/jonsamwell/flutter_gherkin/tree/integration_test__package_support/example_with_integration_test

I also tested flutter run with tagExpression @failure-expected, but then the result doesn't match the failure:

flutter run integration_test/gherkin_suite_test.dart
...
I/flutter (16379): FAILED: Scenario Failed expect() should be added to json report # ./integration_test/features/failure.feature:0
...
I/flutter (16379): 00:03 +3: All tests passed!
golane-august commented 2 years ago

Also calling this with tagExpression: '@failure-expected':

 flutter drive --driver=test_driver/integration_test_driver.dart --target=integration_test/gherkin_suite_test.dart -d chrome

results in:

FAILED: Scenario Failed expect() should be added to json report # ./integration_test/features/failure.feature:0
00:03 +2: (tearDownAll)
2 scenarios (2 failed)
4 steps (2 passed, 2 failed)
0:00:03.088000
00:03 +3: All tests passed!
Application finished.

Note the "All tests passed!", which sould not occur.

luvetter commented 1 year ago

The problem seems to be that the generated executeTestSuite method is synchronous, but the run method is asynchronous:

void executeTestSuite({
  required FlutterTestConfiguration configuration,
  required StartAppFn appMainFunction,
  Timeout scenarioExecutionTimeout = const Timeout(const Duration(minutes: 10)),
  AppLifecyclePumpHandlerFn? appLifecyclePumpHandler,
  LiveTestWidgetsFlutterBindingFramePolicy? framePolicy,
}) {
  _CustomGherkinIntegrationTestRunner(
    configuration: configuration,
    appMainFunction: appMainFunction,
    appLifecyclePumpHandler: appLifecyclePumpHandler,
    scenarioExecutionTimeout: scenarioExecutionTimeout,
    framePolicy: framePolicy,
  ).run();
}

Also the main method is done before the test setup is complete:

@GherkinTestSuite(
  useAbsolutePaths: false,
)
void main() {
  executeTestSuite(
    appMainFunction: appInitializationFn,
    configuration: gherkinTestConfiguration,
  );
}

But Flutter assumes that with the end of the main method the test declaration is finished and does not allow any more calls on the test api (group, test, testWidgets, ...)

You could probably make the executeTestSuite method asynchronous and await it.

Future executeTestSuite({
  required FlutterTestConfiguration configuration,
  required StartAppFn appMainFunction,
  Timeout scenarioExecutionTimeout = const Timeout(const Duration(minutes: 10)),
  AppLifecyclePumpHandlerFn? appLifecyclePumpHandler,
  LiveTestWidgetsFlutterBindingFramePolicy? framePolicy,
}) async {
  return _CustomGherkinIntegrationTestRunner(
    configuration: configuration,
    appMainFunction: appMainFunction,
    appLifecyclePumpHandler: appLifecyclePumpHandler,
    scenarioExecutionTimeout: scenarioExecutionTimeout,
    framePolicy: framePolicy,
  ).run();
}

@GherkinTestSuite(
  useAbsolutePaths: false,
)
void main() async {
  await executeTestSuite(
    appMainFunction: appInitializationFn,
    configuration: gherkinTestConfiguration,
  );
}

What do you think @jonsamwell ?

AFASbart commented 1 year ago

@luvetter I edited you comment to translate the german sentence 😉