search

getlantern / lantern-client

Lantern Client code
GNU General Public License v3.0
14 stars 3 forks source link

Setup Testing #1207

Open jigar-f opened 1 month ago

jigar-f commented 1 month ago

Overview of This PR:

Initial Test Environment Setup-:Establishes the foundation for testing across the app. Reusable Widget Testing Structure-: Creates a framework that can be reused for widget tests in future test cases.

Upcoming Work:

Test Updates

List of app workflow test

Run Tests

Note: To run this integration on mobile make sure you have the device connected

To run Single test cases use this will work with widget and integration test make runTest testfileName

To run all integration/app workflow tests use make appWorkflowTest

To run all desktop test make desktopWorkflowTest

To run a single test on a desktop make runDesktopTest testfileName

atavism commented 1 month ago

The changes here look amazing, @jigar-f!!! Sorry I haven't had a chance to take a closer look at this one yet, but I will do that first thing in the morning

jigar-f commented 1 month ago

@atavism I updated PR description to help to run the test across all devices, do take a look

atavism commented 4 weeks ago

Thanks for adding instructions on how to run the tests. Maybe we add those to the integration testing section of the README?

atavism commented 4 weeks ago

I tried running the tests with make desktopWorkflowTest on macOS (14.6.1), and it looks like the mock test for the home page flow is failing for some reason (tried a few times):

VPNTapSkeleton
Using desktop switch;
01:14 +1: loading /Users/paul/go/src/github.com/getlantern/lantern-client/integration_test/features/home/homepage_flow_mock_test.dart                                                                                                                                                                                                                                         B
--- xcodebuild: WARNING: Using the first of multiple matching destinations:
{ platform:macOS, arch:arm64, id:00008112-000809843A08201E, name:My Mac }
{ platform:macOS, arch:x86_64, id:00008112-000809843A08201E, name:My Mac }
01:40 +1: loading /Users/paul/go/src/github.com/getlantern/lantern-client/integration_test/features/home/homepage_flow_mock_test.dart
✓ Built build/macos/Build/Products/Debug/Lantern.app
Error waiting for a debug connection: The log reader stopped unexpectedly, or never started.
01:40 +1 -1: loading /Users/paul/go/src/github.com/getlantern/lantern-client/integration_test/features/home/homepage_flow_mock_test.dart [E]
  TestDeviceException(Unable to start the app on the device.)
  package:flutter_tools/src/test/integration_test_device.dart 63:7  IntegrationTestTestDevice.start

  Bad state: Cannot add new events after calling close
  dart:async/broadcast_stream_controller.dart 243:24  _BroadcastStreamController.add
  dart:async/zone.dart 1407:47                        _rootRunUnary
  dart:async/zone.dart 1308:19                        _CustomZone.runUnary
  dart:async/zone.dart 1217:7                         _CustomZone.runUnaryGuarded
  dart:async/stream_impl.dart 365:11                  _BufferingStreamSubscription._sendData
  dart:async/stream_impl.dart 297:7                   _BufferingStreamSubscription._add
  dart:async/stream_controller.dart 784:19            _SyncStreamControllerDispatch._sendData
  dart:async/stream_controller.dart 658:7             _StreamController._add
  dart:async/stream_controller.dart 606:5             _StreamController.add
  dart:io-patch/socket_patch.dart 2455:41             _Socket._onData
  dart:async/zone.dart 1415:13                        _rootRunUnary
  dart:async/zone.dart 1308:19                        _CustomZone.runUnary
  dart:async/zone.dart 1217:7                         _CustomZone.runUnaryGuarded
  dart:async/stream_impl.dart 365:11                  _BufferingStreamSubscription._sendData
  dart:async/stream_impl.dart 297:7                   _BufferingStreamSubscription._add
  dart:async/stream_controller.dart 784:19            _SyncStreamControllerDispatch._sendData
  dart:async/stream_controller.dart 658:7             _StreamController._add
  dart:async/stream_controller.dart 606:5             _StreamController.add
  dart:io-patch/socket_patch.dart 1942:33             new _RawSocket.<fn>
  dart:io-patch/socket_patch.dart 1385:14             _NativeSocket.issueReadEvent.issue
  dart:async/schedule_microtask.dart 40:21            _microtaskLoop
  dart:async/schedule_microtask.dart 49:5             _startMicrotaskLoop
  dart:isolate-patch/isolate_patch.dart 118:13        _runPendingImmediateCallback
  dart:isolate-patch/isolate_patch.dart 185:5         _RawReceivePort._handleMessage

  Bad state: Cannot add new events after calling close
  dart:async/broadcast_stream_controller.dart 243:24  _BroadcastStreamController.add
  dart:async/zone.dart 1407:47                        _rootRunUnary
  dart:async/zone.dart 1308:19                        _CustomZone.runUnary
  dart:async/zone.dart 1217:7                         _CustomZone.runUnaryGuarded
  dart:async/stream_impl.dart 365:11                  _BufferingStreamSubscription._sendData
  dart:async/stream_impl.dart 297:7                   _BufferingStreamSubscription._add
  dart:async/stream_controller.dart 784:19            _SyncStreamControllerDispatch._sendData
  dart:async/stream_controller.dart 658:7             _StreamController._add
  dart:async/stream_controller.dart 606:5             _StreamController.add

The Lantern app just has a black screen. BTW, love that you implemented these tests to work on desktop as well!

atavism commented 4 weeks ago

When I run the integration/workflow tests on my Android device (using make appWorkflowTest), all of the tests pass except for two:

══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following WaitUntilVisibleTimeoutException was thrown running a test:
TimeoutException after 0:00:10.000000: Finder "Found 0 widgets with key [<'language'>]: []" did not
find any visible (i.e. hit-testable) widgets

When the exception was thrown, this was the stack:
#0      PatrolTester.waitUntilVisible.<anonymous closure> (package:patrol_finders/src/custom_finders/patrol_tester.dart:405:11)
<asynchronous suspension>
#1      TestAsyncUtils.guard.<anonymous closure> (package:flutter_test/src/test_async_utils.dart:120:7)
<asynchronous suspension>
#2      PatrolTester.tap.<anonymous closure> (package:patrol_finders/src/custom_finders/patrol_tester.dart:247:30)
<asynchronous suspension>
#3      TestAsyncUtils.guard.<anonymous closure> (package:flutter_test/src/test_async_utils.dart:120:7)
<asynchronous suspension>
#4      PatrolFinder.tap (package:patrol_finders/src/custom_finders/patrol_finder.dart:207:5)
<asynchronous suspension>
#5      main.<anonymous closure> (file:///Users/paul/go/src/github.com/getlantern/lantern-client/integration_test/features/account/language_flow_test.dart:25:5)
<asynchronous suspension>
#6      patrolWidget.<anonymous closure> (file:///Users/paul/go/src/github.com/getlantern/lantern-client/integration_test/utils/test_utils.dart:100:7)
<asynchronous suspension>
#7      patrolWidgetTest.<anonymous closure> (package:patrol_finders/src/common.dart:52:7)
<asynchronous suspension>
#8      testWidgets.<anonymous closure>.<anonymous closure> (package:flutter_test/src/widget_tester.dart:189:15)
<asynchronous suspension>
#9      TestWidgetsFlutterBinding._runTestBody (package:flutter_test/src/binding.dart:1032:5)
<asynchronous suspension>
<asynchronous suspension>
(elided one frame from package:stack_trace)

The test description was:
  language end to end test
════════════════════════════════════════════════════════════════════════════════════════════════════
11:15 +8 ~1 -1: /Users/paul/go/src/github.com/getlantern/lantern-client/integration_test/features/account/language_flow_test.dart: language end to end test [E]
  Test failed. See exception logs above.
  The test description was: language end to end test
store version: false
══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following TestFailure was thrown running a test:
Expected: at least 2 matching candidates
  Actual: PatrolFinder:<Found 0 widgets with type "PlanCard": []>
   Which: means none were found but some were expected

When the exception was thrown, this was the stack:
#4      main.<anonymous closure>.<anonymous closure> (file:///Users/paul/go/src/github.com/getlantern/lantern-client/integration_test/features/account/account_flow_test.dart:35:11)
<asynchronous suspension>
#5      patrolWidget.<anonymous closure> (file:///Users/paul/go/src/github.com/getlantern/lantern-client/integration_test/utils/test_utils.dart:100:7)
<asynchronous suspension>
#6      patrolWidgetTest.<anonymous closure> (package:patrol_finders/src/common.dart:52:7)
<asynchronous suspension>
#7      testWidgets.<anonymous closure>.<anonymous closure> (package:flutter_test/src/widget_tester.dart:189:15)
<asynchronous suspension>
#8      TestWidgetsFlutterBinding._runTestBody (package:flutter_test/src/binding.dart:1032:5)
<asynchronous suspension>
<asynchronous suspension>
(elided one frame from package:stack_trace)

This was caught by the test expectation on the following line:
  file:///Users/paul/go/src/github.com/getlantern/lantern-client/integration_test/features/account/account_flow_test.dart line 35
The test description was:
  render account screen and navigation test [free user]
════════════════════════════════════════════════════════════════════════════════════════════════════
12:38 +8 ~1 -2: /Users/paul/go/src/github.com/getlantern/lantern-client/integration_test/features/account/account_flow_test.dart: account page end to end render account screen and navigation test [free user] [E]
  Test failed. See exception logs above.
  The test description was: render account screen and navigation test [free user]
jigar-f commented 4 weeks ago

Thanks, @atavism for taking a look, Let me see whats going on and get back to you.

jigar-f commented 3 weeks ago

@atavism, Can you try running the test now on the desktop with make desktopWorkflowTest and on Android with make appWorkflowTest and see if the tests pass? They should now.