search

leancodepl / patrol

Flutter-first UI testing framework. Ready for action!
https://patrol.leancode.co
Apache License 2.0
856 stars 126 forks source link

NativeAutomator.disableDarkMode does not work if Settings App is not in Default View on iOS #2273

Closed Friesenjung closed 1 month ago

Friesenjung commented 1 month ago

Steps to reproduce

  1. Open Simulator (iOS, iPhone 15 Pro e.g.)

  2. Go To Settings App and open a View that is not the default view (e.g. General View)

  3. Use the following example_test.dart:

    example_test.dart ```js import 'package:flutter/material.dart'; import 'package:patrol/patrol.dart'; void main() { patrolTest( 'Using Native functionality when Settings App was open before', ($) async { // Replace later with your app's main widget await $.pumpWidgetAndSettle( MaterialApp( home: Scaffold( appBar: AppBar(title: const Text('app')), backgroundColor: Colors.blue, ), ), ); $.native.disableDarkMode(); }, ); } ```

  4. Run test with the following command: patrol test -t integration_test/example_test.dart -d "iPhone 15 Pro"

Actual results

The Test results in an error.

Logs

Logs ``` $ flutter --suppress-analytics --no-version-check pub deps --style=list Received 1 test target(s) Received test target: /Users/user/IdeaProjects/MoneyOps/money_ops/integration_test/example_test.dart Generated entrypoint /Users/user/IdeaProjects/MoneyOps/money_ops/integration_test/test_bundle.dart with 1 bundled test(s) $ flutter --no-version-check --suppress-analytics devices --machine Received 1 device(s) to run on Received device: iPhone 15 Pro Received 9 --dart-define(s) (0 custom, 9 internal) Received internal --dart-define: PATROL_WAIT=0 Received internal --dart-define: PATROL_APP_PACKAGE_NAME=de.hexagonstudios.money_ops Received internal --dart-define: PATROL_APP_BUNDLE_ID=de.hexagonstudios.moneyops Received internal --dart-define: PATROL_ANDROID_APP_NAME=money_ops Received internal --dart-define: PATROL_IOS_APP_NAME=money_ops Received internal --dart-define: INTEGRATION_TEST_SHOULD_REPORT_RESULTS_TO_NATIVE=false Received internal --dart-define: PATROL_TEST_LABEL_ENABLED=true Received internal --dart-define: PATROL_TEST_SERVER_PORT=8081 Received internal --dart-define: PATROL_APP_SERVER_PORT=8082 • Building app with entrypoint test_bundle.dart for iOS simulator (debug)... $ flutter build ios --no-version-check --suppress-analytics --config-only --no-codesign --debug --simulator --target /Users/user/IdeaProjects/MoneyOps/money_ops/integration_test/test_bundle.dart --dart-define PATROL_WAIT=0 --dart-define PATROL_APP_PACKAGE_NAME=de.hexagonstudios.money_ops --dart-define PATROL_APP_BUNDLE_ID=de.hexagonstudios.moneyops --dart-define PATROL_ANDROID_APP_NAME=money_ops --dart-define PATROL_IOS_APP_NAME=money_ops --dart-define INTEGRATION_TEST_SHOULD_REPORT_RESULTS_TO_NATIVE=false --dart-define PATROL_TEST_LABEL_ENABLED=true --dart-define PATROL_TEST_SERVER_PORT=8081 --dart-define PATROL_APP_SERVER_PORT=8082 Building de.hexagonstudios.moneyops for simulator (ios)... $ xcodebuild build-for-testing -workspace Runner.xcworkspace -scheme Runner -configuration Debug -sdk iphonesimulator -destination generic/platform=iOS Simulator -quiet -derivedDataPath ../build/ios_integ OTHER_SWIFT_FLAGS=$(inherited) -D PATROL_ENABLED /Users/user/IdeaProjects/MoneyOps/money_ops/ios/Pods/Pods.xcodeproj: warning: The iOS Simulator deployment target 'IPHONEOS_DEPLOYMENT_TARGET' is set to 9.0, but the range of supported deployment target versions is 12.0 to 17.5.99. (in target 'CocoaAsyncSocket' from project 'Pods') /Users/user/IdeaProjects/MoneyOps/money_ops/ios/Pods/Pods.xcodeproj: warning: The iOS Simulator deployment target 'IPHONEOS_DEPLOYMENT_TARGET' is set to 9.0, but the range of supported deployment target versions is 12.0 to 17.5.99. (in target 'flutter_native_splash-flutter_native_splash_privacy' from project 'Pods') /Users/user/IdeaProjects/MoneyOps/money_ops/ios/Pods/Pods.xcodeproj: warning: The iOS Simulator deployment target 'IPHONEOS_DEPLOYMENT_TARGET' is set to 11.0, but the range of supported deployment target versions is 12.0 to 17.5.99. (in target 'patrol-patrol_privacy' from project 'Pods') note: Run script build phase 'Thin Binary' will be run during every build because the option to run the script phase "Based on dependency analysis" is unchecked. (in target 'Runner' from project 'Runner') note: Run script build phase 'Run Script' will be run during every build because the option to run the script phase "Based on dependency analysis" is unchecked. (in target 'Runner' from project 'Runner') warning: Run script build phase 'xcode_backend embed_and_thin' will be run during every build because it does not specify any outputs. To address this warning, either add output dependencies to the script phase, or configure it to run in every build by unchecking "Based on dependency analysis" in the script phase. (in target 'RunnerUITests' from project 'Runner') warning: Run script build phase 'xcode_backend build' will be run during every build because it does not specify any outputs. To address this warning, either add output dependencies to the script phase, or configure it to run in every build by unchecking "Based on dependency analysis" in the script phase. (in target 'RunnerUITests' from project 'Runner') ✓ Completed building app with entrypoint test_bundle.dart for iOS simulator (28.2s) Will uninstall apps before running tests $ xcrun simctl uninstall 6131C4F9-D171-4F2E-A9E7-99574EF75B5D de.hexagonstudios.moneyops $ xcrun simctl uninstall 6131C4F9-D171-4F2E-A9E7-99574EF75B5D de.hexagonstudios.moneyops.RunnerUITests.xctrunner • Running app with entrypoint test_bundle.dart for iOS simulator on simulator iPhone 15 Pro... $ xcodebuild -showsdks -json Assuming SDK version 17.5 for iphonesimulator Looking for .xctestrun matching Runner_*iphonesimulator17.5*.xctestrun at /Users/user/IdeaProjects/MoneyOps/money_ops/build/ios_integ/Build/Products Found 1 match(es), the first one will be used Found /Users/user/IdeaProjects/MoneyOps/money_ops/build/ios_integ/Build/Products/Runner_iphonesimulator17.5-arm64-x86_64.xctestrun $ xcodebuild test-without-building -xctestrun /Users/user/IdeaProjects/MoneyOps/money_ops/build/ios_integ/Build/Products/Runner_iphonesimulator17.5-arm64-x86_64.xctestrun -only-testing RunnerUITests/RunnerUITests -destination platform=iOS Simulator,name=iPhone 15 Pro -destination-timeout 1 -resultBundlePath /Users/user/IdeaProjects/MoneyOps/money_ops/build/ios_results_1721576476173.xcresult Command line invocation: /Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild test-without-building -xctestrun /Users/user/IdeaProjects/MoneyOps/money_ops/build/ios_integ/Build/Products/Runner_iphonesimulator17.5-arm64-x86_64.xctestrun -only-testing RunnerUITests/RunnerUITests -destination "platform=iOS Simulator,name=iPhone 15 Pro" -destination-timeout 1 -resultBundlePath /Users/user/IdeaProjects/MoneyOps/money_ops/build/ios_results_1721576476173.xcresult User defaults from command line: IDEBuildOperationResultBundlePath = /Users/user/IdeaProjects/MoneyOps/money_ops/build/ios_results_1721576476173.xcresult IDEPackageSupportUseBuiltinSCM = YES Writing result bundle at path: /Users/user/IdeaProjects/MoneyOps/money_ops/build/ios_results_1721576476173.xcresult --- xcodebuild: WARNING: Using the first of multiple matching destinations: { platform:iOS Simulator, id:6131C4F9-D171-4F2E-A9E7-99574EF75B5D, OS:17.5, name:iPhone 15 Pro } { platform:iOS Simulator, id:6131C4F9-D171-4F2E-A9E7-99574EF75B5D, OS:17.5, name:iPhone 15 Pro } 2024-07-21 17:41:17.609392+0200 RunnerUITests-Runner[34655:375534] [Default] Running tests... 2024-07-21 17:41:18.421238+0200 RunnerUITests-Runner[34655:375534] PatrolServer: INFO: PatrolServer constructor called 2024-07-21 17:41:18.421295+0200 RunnerUITests-Runner[34655:375534] PatrolServer: INFO: PATROL_ENABLED flag is defined 2024-07-21 17:41:18.434993+0200 RunnerUITests-Runner[34655:375534] PatrolServer: INFO: Starting server... 2024-07-21 17:41:18.437359+0200 RunnerUITests-Runner[34655:375534] PatrolServer: INFO: Server started on http://0.0.0.0:8081 2024-07-21 17:41:18.437502+0200 RunnerUITests-Runner[34655:375534] PatrolAppServiceClient: created, port: 8082 t = nans Checking existence of `"Allow" Button` t = nans Open de.hexagonstudios.moneyops t = nans Launch de.hexagonstudios.moneyops t = nans Setting up automation session t = nans Wait for de.hexagonstudios.moneyops to idle 2024-07-21 17:41:19.458703+0200 RunnerUITests-Runner[34655:375650] PatrolServer: INFO: App reported that it is ready 2024-07-21 17:41:20.241070+0200 RunnerUITests-Runner[34655:375534] PatrolAppServiceClient.listDartTests() 2024-07-21 17:41:20.291687+0200 RunnerUITests-Runner[34655:375652] [connection] nw_socket_handle_socket_event [C1.1.1:2] Socket SO_ERROR [61: Connection refused] 2024-07-21 17:41:20.310035+0200 RunnerUITests-Runner[34655:375652] PatrolAppServiceClient.listDartTests(): succeeded 2024-07-21 17:41:21.257499+0200 RunnerUITests-Runner[34655:375534] Got 1 Dart tests: ( "example_test Using Native functionality when Settings App was open before" ) 2024-07-21 17:41:21.258486+0200 RunnerUITests-Runner[34655:375534] RunnerUITests.testInvocations(): selectorName = example_test Using Native functionality when Settings App was open before, signature: Test Suite 'Selected tests' started at 2024-07-21 17:41:21.263. Test Suite 'RunnerUITests.xctest' started at 2024-07-21 17:41:21.264. Test Suite 'RunnerUITests' started at 2024-07-21 17:41:21.264. Test Case '-[RunnerUITests example_test Using Native functionality when Settings App was open before]' started. t = 0.00s Start Test at 2024-07-21 17:41:21.266 t = 0.04s Set Up t = 0.04s Open de.hexagonstudios.moneyops t = 0.04s Launch de.hexagonstudios.moneyops t = 0.04s Terminate de.hexagonstudios.moneyops:34666 t = 1.31s Setting up automation session t = 1.43s Wait for de.hexagonstudios.moneyops to idle 2024-07-21 17:41:22.891931+0200 RunnerUITests-Runner[34655:375650] PatrolServer: INFO: App reported that it is ready 2024-07-21 17:41:24.782144+0200 RunnerUITests-Runner[34655:375534] PatrolAppServiceClient.runDartTest(example_test Using Native functionality when Settings App was open before) 2024-07-21 17:41:24.796142+0200 RunnerUITests-Runner[34655:375650] [connection] nw_socket_handle_socket_event [C2.1.1:2] Socket SO_ERROR [61: Connection refused] 2024-07-21 17:41:25.091760+0200 RunnerUITests-Runner[34655:375534] PatrolServer: INFO: disabling dark mode... t = 3.83s Open com.apple.springboard t = 3.83s Activate com.apple.springboard t = 3.83s Wait for com.apple.springboard to idle 2024-07-21 17:41:25.109618+0200 RunnerUITests-Runner[34655:375652] runDartTest("example_test Using Native functionality when Settings App was open before"): call finished, test result: PASSED t = 3.85s Open com.apple.Preferences t = 3.85s Launch com.apple.Preferences t = 3.85s Terminate com.apple.Preferences:33968 t = 4.91s Setting up automation session t = 5.29s Wait for com.apple.Preferences to idle t = 6.45s Tap "Developer" Any t = 6.45s Wait for com.apple.Preferences to idle t = 6.46s Find the "Developer" Any t = 7.53s Find the "Developer" Any (retry 1) t = 8.61s Find the "Developer" Any (retry 2) t = 8.68s Requesting snapshot of accessibility hierarchy for app with pid 34684 /Users/user/IdeaProjects/MoneyOps/money_ops/ios/RunnerUITests/RunnerUITests.m:5: error: -[RunnerUITests example_test Using Native functionality when Settings App was open before] : Failed to tap "Developer" Any: No matches found for first query match sequence: `Descendants matching type Any` -> `Elements matching predicate '"Developer" IN identifiers'`, given input App element pid: 34684 t = 8.75s Find the "Dark Appearance" Any t = 9.82s Find the "Dark Appearance" Any (retry 1) t = 10.89s Find the "Dark Appearance" Any (retry 2) t = 10.96s Collecting extra data to assist test failure triage t = 10.96s Requesting snapshot of accessibility hierarchy for app with pid 34684 t = 10.99s Requesting snapshot of accessibility hierarchy for app with pid 34684 t = 11.02s Requesting snapshot of accessibility hierarchy for app with pid 34684 /Users/user/IdeaProjects/MoneyOps/money_ops/ios/RunnerUITests/RunnerUITests.m:5: error: -[RunnerUITests example_test Using Native functionality when Settings App was open before] : Failed to get matching snapshot: No matches found for first query match sequence: `Descendants matching type Any` -> `Elements matching predicate '"Dark Appearance" IN identifiers'`, given input App element pid: 34684 t = 11.05s Open com.apple.springboard t = 11.05s Activate com.apple.springboard t = 11.05s Wait for com.apple.springboard to idle t = 11.06s Terminate com.apple.Preferences:34684 t = 12.10s Open de.hexagonstudios.moneyops t = 12.10s Activate de.hexagonstudios.moneyops t = 12.13s Wait for de.hexagonstudios.moneyops to idle 2024-07-21 17:41:33.403391+0200 RunnerUITests-Runner[34655:375534] PatrolServer: INFO: done disabling dark mode 2024-07-21 17:41:33.405251+0200 RunnerUITests-Runner[34655:375534] PatrolServer: INFO: result: () t = 12.14s Tear Down Test Case '-[RunnerUITests example_test Using Native functionality when Settings App was open before]' failed (12.379 seconds). Test Suite 'RunnerUITests' failed at 2024-07-21 17:41:33.644. Executed 1 test, with 2 failures (0 unexpected) in 12.379 (12.380) seconds Test Suite 'RunnerUITests.xctest' failed at 2024-07-21 17:41:33.645. Executed 1 test, with 2 failures (0 unexpected) in 12.379 (12.381) seconds Test Suite 'Selected tests' failed at 2024-07-21 17:41:33.646. Executed 1 test, with 2 failures (0 unexpected) in 12.379 (12.383) seconds 2024-07-21 17:41:46.918 xcodebuild[34652:375395] [MT] IDETestOperationsObserverDebug: 30.070 elapsed -- Testing started completed. 2024-07-21 17:41:46.918 xcodebuild[34652:375395] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start 2024-07-21 17:41:46.918 xcodebuild[34652:375395] [MT] IDETestOperationsObserverDebug: 30.070 sec, +30.070 sec -- end Test session results, code coverage, and logs: /Users/user/IdeaProjects/MoneyOps/money_ops/build/ios_results_1721576476173.xcresult Failing tests: -[RunnerUITests example_test Using Native functionality when Settings App was open before] -[RunnerUITests example_test Using Native functionality when Settings App was open before] ** TEST EXECUTE FAILED ** Testing started ✗ Failed to execute tests of app with entrypoint test_bundle.dart for iOS simulator on simulator iPhone 15 Pro (xcodebuild exited with code 65) (32.3s) Error: xcodebuild exited with code 65 #0 throwToolExit (package:patrol_cli/src/base/exceptions.dart:7:3) #1 IOSTestBackend.execute. (package:patrol_cli/src/ios/ios_test_backend.dart:198:9) #2 DisposeScope.run (package:dispose_scope/src/dispose_scope.dart:46:7) #3 IOSTestBackend.execute (package:patrol_cli/src/ios/ios_test_backend.dart:153:5) #4 TestCommand._execute (package:patrol_cli/src/commands/test.dart:336:7) #5 TestCommand.run (package:patrol_cli/src/commands/test.dart:228:23) #6 CommandRunner.runCommand (package:args/command_runner.dart:212:13) #7 PatrolCommandRunner.runCommand (package:patrol_cli/src/runner/patrol_command_runner.dart:348:18) #8 PatrolCommandRunner.run (package:patrol_cli/src/runner/patrol_command_runner.dart:292:18) #9 patrolCommandRunner (package:patrol_cli/src/runner/patrol_command_runner.dart:70:20) #10 main (file:///Users/user/.pub-cache/hosted/pub.dev/patrol_cli-2.8.1/bin/main.dart:6:20) See the logs above to learn what happened. Also consider running with --verbose. If the logs still aren't useful, then it's a bug - please report it. $ xcrun simctl uninstall 6131C4F9-D171-4F2E-A9E7-99574EF75B5D de.hexagonstudios.moneyops $ xcrun simctl uninstall 6131C4F9-D171-4F2E-A9E7-99574EF75B5D de.hexagonstudios.moneyops.RunnerUITests.xctrunner ```

Patrol version

patrol: ^3.9.0

Patrol Doctor output

Patrol Doctor output ``` Patrol doctor: Patrol CLI version: 2.8.1 Flutter command: flutter Flutter 3.22.2 • channel stable Android: • Program adb found in /Users/user/Library/Android/sdk/platform-tools/adb • Env var $ANDROID_HOME set to /Users/user/Library/Android/sdk iOS / macOS: • Program xcodebuild found in /usr/bin/xcodebuild • Program ideviceinstaller found in /opt/homebrew/bin/ideviceinstaller ```

Flutter Doctor output

Flutter Doctor output ``` [✓] Flutter (Channel stable, 3.22.2, on macOS 14.5 23F79 darwin-arm64, locale en-DE) [✓] Android toolchain - develop for Android devices (Android SDK version 34.0.0) [✓] Xcode - develop for iOS and macOS (Xcode 15.4) [✓] Chrome - develop for the web [✓] Android Studio (version 2023.1) [✓] IntelliJ IDEA Community Edition (version 2024.1.2) [✓] Connected device (5 available) [✓] Network resources • No issues found! ```
piotruela commented 1 month ago

Hi @Friesenjung, thanks for reporting the issue. I was able to reproduce. I opened a PR with the fix.

github-actions[bot] commented 1 month ago

This issue has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar problem, please file a new issue. Make sure to follow the template and provide all the information necessary to reproduce the issue.