leancodepl / patrol

Flutter-first UI testing framework. Ready for action!
https://patrol.leancode.co
Apache License 2.0
869 stars 128 forks source link

Consider migrating from `ios-deploy` to `go-ios` #984

Open bartekpacia opened 1 year ago

bartekpacia commented 1 year ago

To run tests on a physical iOS device, patrol_cli uses ios-deploy (the user has to brew install ios-deploy. They can also run patrol doctor to see the status).

I'd call ios-deploy a fragile dependency:

Meanwhile go-ios:

I tested go-ios and it already does what we currently do with ios-deploy.

Install the app under test:

ios install --path build/ios_integ/Build/Products/Debug-iphoneos/Runner.app

Install the UI test instrumentation:

ios install --path build/ios_integ/Build/Products/Debug-iphoneos/RunnerUITests-Runner.app

Run the app under test:

ios debug build/ios_integ/Build/Products/Debug-iphoneos/Runner.app

and then run the UI test instrumentation:

ios runtest \
  --bundle-id pl.leancode.patrol.Example \
  --test-runner-bundle-id pl.leancode.patrol.Example.RunnerUITests.xctrunner \
  --xctest-config RunnerUITests.xctest

Works just fine (tested with notifications_test.dart).

bartekpacia commented 1 year ago

Additionally, I'd like patrol_cli to have a clearer lifecycle:

Of course, we'll keep the patrol test command, which does all the 3 things in a single invocation.


Notes:

The same changes should also be applied for Android, to keep behavior consistent.

bartekpacia commented 1 year ago

patrol_cli doesn't depend on ios-deploy anymore (see #1303), but this issue still stands, especially the comment above about splitting the lifecycle.

bartekpacia commented 1 year ago

Note for the future: ios runtest fails to run tests and I don't know why. Meanwhile patrol test --release works fine (it uses xcodebuild test-without-building under the hood).

Logs ``` $ ios runtest \ --bundle-id pl.leancode.patrol.Example \ --test-runner-bundle-id pl.leancode.patrol.Example.RunnerUITests.xctrunner \ --xctest-config build/ios_integ/Build/Products/Runner_iphoneos16.4-arm64.xctestrun {"level":"info","msg":"no udid specified using first device in list","time":"2023-05-23T17:08:41+02:00","udid":"00008020-001C499A3420802E"} {"bundleID":"pl.leancode.patrol.Example.RunnerUITests.xctrunner","channel_id":"com.apple.instruments.server.services.processcontrol","level":"info","msg":"Launching process","time":"2023-05-23T17:08:42+02:00"} {"channel_id":"com.apple.instruments.server.services.processcontrol","level":"info","msg":"Process started successfully","pid":714,"time":"2023-05-23T17:08:42+02:00"} {"fields.msg":"2023-05-23 17:08:43.071615+0200 RunnerUITests-Runner[714:125747] Running tests...\n","fields.time":229514261701,"level":"info","msg":"outputReceived:fromProcess:atTime:","pid":714,"time":"2023-05-23T17:08:42+02:00"} {"level":"info","msg":"device called local method","sel":"_XCT_didFailToBootstrapWithError:","time":"2023-05-23T17:08:42+02:00"} {"fields.msg":"2023-05-23 17:08:43.085256+0200 RunnerUITests-Runner[714:125747] Failed to create a bundle instance representing '/private/var/containers/Bundle/Application/6755DAAF-9F80-4F86-9DF0-412FCC173852/RunnerUITests-Runner.app.ipa.app/PlugIns/build/ios_integ/Build/Products/Runner_iphoneos16.4-arm64.xctestrun'. Check that the bundle exists on disk.\n","fields.time":229514589851,"level":"info","msg":"outputReceived:fromProcess:atTime:","pid":714,"time":"2023-05-23T17:08:42+02:00"} {"level":"error","msg":"DeviceConnection failed writing 402 bytes, only 0 sent","time":"2023-05-23T17:08:43+02:00"} {"level":"error","msg":"Failed sending: write unix -\u003e/var/run/usbmuxd: write: broken pipe","time":"2023-05-23T17:08:43+02:00"} {"channel_id":"dtxproxy:XCTestDriverInterface:XCTestManager_IDEInterface","error":"write unix -\u003e/var/run/usbmuxd: write: broken pipe","level":"info","methodselector":"_IDE_startExecutingTestPlanWithProtocolVersion:","msg":"failed starting invoking method","time":"2023-05-23T17:08:43+02:00"} {"level":"error","msg":"write unix -\u003e/var/run/usbmuxd: write: broken pipe","time":"2023-05-23T17:08:43+02:00"} {"error":"write unix -\u003e/var/run/usbmuxd: write: broken pipe","level":"info","msg":"Failed running Xcuitest","time":"2023-05-23T17:08:43+02:00"} ```