search

facebookarchive / xctool

An extension for Apple's xcodebuild that makes it easier to test iOS and macOS apps.
Apache License 2.0
6.91k stars 737 forks source link

Xcode 7 UIAutomation Support #534

Closed djrenfro closed 5 years ago

djrenfro commented 9 years ago

It would be great if you could add support for running Xcode UIAutomation using xctool. Currently, the tests are built but as they are not built into xctest they cannot be run.

ExtremeMan commented 9 years ago

We aren't currently looking into this. But we will be happy to review any PR.

ExtremeMan commented 9 years ago

Let's switch to this issue, @sampham.

The branch https://github.com/facebook/xctool/tree/nekto/support_ui_testing has all the most recent updates I have. To be honest I don't remember what exactly didn't work there. I think I stopped when I couldn't figure out why tests aren't triggered to run. I tried to repeat all the steps Xcode does when launching UI tests and turned off "wait for debugger" flag. I saw mostly the same output of Simulator in system logs as the one that is produced when tests are run from Xcode though. But the tests weren't simply starting (I've added several NSLogs statements to a sample app I was using for testing and didn't see them being executed).

If I look into it again I would try running UI tests from xcodebuild (if it supports them). Also I will probably try avoiding usage of XCTRunner. Probably it is the one which causes problems.

antranapp commented 9 years ago

yes, xcodebuild supports running UI Test. You need to create a scheme for that: https://krausefx.com/blog/run-xcode-7-ui-tests-from-the-command-line

K76154 commented 9 years ago

I believe entire iOS development community is waiting for this support to be added. We really need this!

mikelupo commented 9 years ago

FYI: It will only work with the simulator though.

On Sep 30, 2015, at 9:28 AM, K76154 notifications@github.com wrote:

I believe entire iOS development community is waiting for this support to be added. We really need this!

— Reply to this email directly or view it on GitHub.

lumos-pairstation commented 9 years ago

:+1:

ExtremeMan commented 9 years ago

I will be happy to support anyone who is ready to work on it. At Facebook we are using our own infrastructure (see https://github.com/facebook/webdriveragent) to run E2E tests and as a result we aren't going to work on this feature in the nearest future.

K76154 commented 9 years ago

Does the webdriver agent work with the new XCode 7 UI test? Or it only works with the old javascript UIAutomation? And how do you launch the simulator and the app automatically?

LegNeato commented 9 years ago

webdriveragent links with the underlying UIAutomation private objective-c framework, there is no JS involved.

K76154 commented 9 years ago

There are two ui testing frameworks provided by Apple after xcode 7. One is the new xcuiapplication based tests, the other one is the old uiautomation framework. Which one does the webdriver support? I think what people wants is the support for the new xcode 7 xcuiapplication based automation.

ExtremeMan commented 9 years ago

I would better ask these questions at the project pages. cc @mmmulani

mmmulani commented 9 years ago

we use FBSimulatorControl (also a project on GitHub) to start up the simulator and the WebDriverAgent to automate it.

we don't support straight up UIAutomation commands but instead implement the web driver spec

K76154 commented 9 years ago

@ExtremeMan I found that if I run the xcode 7 ui test with xcodebuild and custom symroot, the test will not run. It says the target app can't be found, and it really didn't install it on the simulator. I guess xctool somehow uses custom symroot implicitly, so it has the same problem. If you are seeing the no target app error, try use default symroot path.

djrenfro commented 9 years ago

@K76154 I was able to get xcode 7 ui test to run with xcodebuild and a custom symroot. 2 things I am doing that may help you. 1. In the symroot, I am putting the full path (so using ${PWD}) which it wasn't working without. The other thing is doing a build command before the test command, there is a chance in you test configuration it is not setup to build the app as well.

n4me1ess commented 9 years ago

@ExtremeMan hi. is there no news about support XC UI Testing? (Xcode7)

anton-matosov commented 8 years ago

I have dig a little bit into the problem with UI Tests. Here is what I have found:

  1. They are not 'regular' xctests, they are getting bundled into the special Runner. This is achieved by adding the user defined build settings; USES_XCTRUNNER=YES and TEST_TARGET_NAME=
  2. However xctool treats them as regular xctest and tries to read xctest bundle from the default location. This might be the main failure cause.

XCTRunner app bundle:

XctoolUITestsSupprot.app
├── Frameworks
│   └── XCTest.framework
│       ├── Headers
│       └── ...
├── Info.plist
├── PkgInfo
├── PlugIns
│   └── XctoolUITestsSupprotUITests.xctest   <--- Your real xctest bundle
│       ├── Frameworks
│       │   ├── libswiftContacts.dylib
│       │   └── ...
│       ├── Info.plist
│       └── XctoolUITestsSupprotUITests
├── XCTRunner                 <--- Runner executable
└── _CodeSignature
    └── CodeResources

I want to try to fix this issue in xctool. Can someone please give me some direction where to start with or any other suggestions?

P.S. I have collected all the possible logs and bundles and can share or help getting more info if needed

ExtremeMan commented 8 years ago

@anton-matosov, thanks for the details. Have you looked into the branch I mentioned? I've already fixed querying of test cases there (see commit https://github.com/facebook/xctool/commit/118c30fe75345df537f63860bb1c69fee5038f40).

anton-matosov commented 8 years ago

I have missed your branch @ExtremeMan, thanks for pointing it out. I will base my future work on your changes.

ExtremeMan commented 8 years ago

(y). Wondering what happens if to set USES_XCTRUNNER=NO. I stopped on trying to launch tests and might be because of I wasn't properly calling to xctrunner.

anton-matosov commented 8 years ago

I have found the the following difference between xctrunner and default/xctool custom test runners: xctrunner starts the main app bundle (TEST_TARGET_NAME) while executing first test case:

13:12:34.543 XCTRunner[83093:14343720] _IDE_startExecutingTestPlanWithProtocolVersion:16
Test Suite 'All tests' started at 2015-11-08 13:12:34.568
Test Suite 'XctoolUITestsSupprotUITests.xctest' started at 2015-11-08 13:12:34.569
Test Suite 'XctoolUITestsSupprotUITests' started at 2015-11-08 13:12:34.570
Test Case '-[XctoolUITestsSupprotUITests.XctoolUITestsSupprotUITests testExample]' started.
    t =     0.00s     Start Test
    t =     0.00s     Set Up
    t =     0.01s         Launch com.auction.XctoolUITestsSupprot      <------------ HERE
2015-11-08 13:12:34.894 XCTRunner[83093:14343707] Continuing to run tests in the background with task ID 1
    t =     1.07s             Waiting for accessibility to load
    t =     5.13s             Wait for app to idle

and continues to run in the background.

anton-matosov commented 8 years ago

So we might try to run tests hosted in the app bundle itself (not 100% sure that it will work, there might be a good reason why Apple invented new runner for this)

anton-matosov commented 8 years ago

@ExtremeMan is there a way to run xctool directly from Xcode instead of attaching to it? I get the following error if try to simply run it ERROR: Failed to launch reporter process /Users/amatosov.ctr/Library/Caches/AppCode33/DerivedData/xctool-aff12441/Build/Products/Debug/reporters/pretty: launch path not accessible

ExtremeMan commented 8 years ago

Yes, you should be able to directly run xctool from Xcode. I have never seen error you posted. May be try rebuilding/cleaning?

anton-matosov commented 8 years ago

never mind, i have opened project, not workspace 😊

anton-matosov commented 8 years ago

I have found the there is one more participant in the UI tests testmanagerd

Nov  8 15:19:50 Amatosov-MBP testmanagerd[89368]: Launched testmanagerd from /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator.sdk/usr/libexec/testmanagerd, built at Friday, October 9, 2015 at 4:42:58 PM Pacific Daylight Time
...
Nov  8 15:19:53 Amatosov-MBP XCTRunner[89409]: Running tests...
...
Nov  8 15:19:53 Amatosov-MBP testmanagerd[89368]: Received new test session connection (-[XCTestManager listener:shouldAcceptNewConnection:])
...
Nov  8 15:19:53 Amatosov-MBP testmanagerd[89368]: -[XCTestManager allowUITestControlForPID:] 89409 vs 89409
Nov  8 15:19:53 Amatosov-MBP testmanagerd[89368]: Got application state update: com.auction.XctoolUITestsSupprot (89343) -> 8
Nov  8 15:19:53 Amatosov-MBP testmanagerd[89368]: -[XCTestManager allowUITestControlForPID:] 89409 vs 89409
...
Nov  8 15:19:55 Amatosov-MBP testmanagerd[89368]: -[XCTestManager allowUITestControlForPID:] 89409 vs 89409
Nov  8 15:19:55 Amatosov-MBP testmanagerd[89368]: Terminating com.auction.XctoolUITestsSupprot
Nov  8 15:19:55 Amatosov-MBP testmanagerd[89368]: PID for com.auction.XctoolUITestsSupprot: 89343
Nov  8 15:19:55 Amatosov-MBP com.apple.CoreSimulator.SimDevice.42A954DB-BBD9-4970-8BAC-5FEF9F6B0EC8.launchd_sim[85263] (UIKitApplication:com.auction.XctoolUITestsSupprot[0x8630][89343]): Service exited due to signal: Killed: 9
Nov  8 15:19:55 Amatosov-MBP SpringBoard[85280]: Application 'UIKitApplication:com.auction.XctoolUITestsSupprot[0x8630]' exited abnormally via signal.
Nov  8 15:19:55 Amatosov-MBP testmanagerd[89368]: AX Notification 1021
Nov  8 15:19:55 Amatosov-MBP testmanagerd[89368]: Got application state update: com.auction.XctoolUITestsSupprot (89343) -> 1
Nov  8 15:19:55 Amatosov-MBP testmanagerd[89368]: AX Notification 1021
Nov  8 15:19:55 Amatosov-MBP CoreSimulatorBridge[85288]: Requesting installation of file:///Users/amatosov.ctr/Library/Developer/Xcode/DerivedData/XctoolUITestsSupprot-fqplloyqdpyjphecyzimidatfiql/Build/Products/Debug-iphonesimulator/XctoolUITestsSupprot.app/ with options: {
        CFBundleIdentifier = "com.auction.XctoolUITestsSupprot";
        PackageType = Developer;
        SimulatorRootPath = "/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator.sdk";
        SimulatorUserPath = "/Users/amatosov.ctr/Library/Developer/CoreSimulator/Devices/42A954DB-BBD9-4970-8BAC-5FEF9F6B0EC8/data";
    }
...

Nov  8 15:19:59 Amatosov-MBP testmanagerd[89368]: AX Notification 4002
Nov  8 15:19:59 Amatosov-MBP testmanagerd[89368]: Got user testing notification with payload {
        controllerClass = "XctoolUITestsSupprot.ViewController";
        event = ViewDidAppear;
    }
...

Nov  8 15:19:59 Amatosov-MBP testmanagerd[89368]: Tearing down test session <XCTestSession: 0x7c429440>.
Nov  8 15:19:59 Amatosov-MBP testmanagerd[89368]: testmanagerd exiting, idle with no test activity.
Nov  8 15:19:59 Amatosov-MBP testmanagerd[89368]: Considering exiting, 0 test sessions, 0 sessions with identifiers, 0 unidentified sessions...
anton-matosov commented 8 years ago

So it is testmanagerd who does the coordination between XCTRunner and main application bundle testmanagerd is getting launched by Xcode and xcodebuild as very first step of running tests.

It seems that it is a must have piece for the UI Tests,

anton-matosov commented 8 years ago

After several days of tries, I still can't find the way to make XCTRunner work w/o testmanagerd. And can't make testmanagerd run w/o xcodebuild.

anton-matosov commented 8 years ago

While researching the issue I have found this opensource toolkit called fastlane that beats xctool from groundup. As part of the toolkit there is tool called scan that allows to run tests including UI tests and it is 100% forward compatible as it wraps xcodebuild + xcpretty command line

K76154 commented 8 years ago

It's different, because xctool lets you to separate build and test, so you can build once and test many times. Fastlane uses xcodebuild, which has to build and test in a same command, and that sometimes is not what people want.

foulkesjohn commented 8 years ago

@anton-matosov I've been playing around with testmanagerd myself whilst trying to get https://github.com/facebook/FBSimulatorControl to launch XCTest bundles. You can start testmanagerd using simctl. I've got a SO question which has the steps I took whilst trying to get simctl to launch an XCTest bundle as a precursor to getting FBSimulatorControl to do the same thing: http://stackoverflow.com/questions/33633882/run-xctest-bundle-with-simctl

lawrencelomax commented 8 years ago

I'm somewhat aware of the challenges that testmanagerd places on running tests directly. Last time I checked you end up having to launch the tests with the DVT frameworks rather than simply launching processes like can be done with Application Tests and code injection.

I think most of this boils down to the DVT frameworks opening a socket with testmanagerd.

lawrencelomax commented 8 years ago

cc @marekcirkos who is also working on a solution for launching UITests directly.

If there is some magic that will enable us to launch UITests without going through xcodebuild/ the DVT frameworks and launch the tests directly (like with the xctest cli) that will be something that we'd want to add to FBSimulatorControl. The same applies for multisim support.

sapieneptus commented 8 years ago

@lawrencelomax Am I correct to assume that there are no plans to provide similar support for physical devices?

emoleumassi commented 8 years ago

It is possible to run the XCode UI Tests with xctool? I tried this command line:

/pathto/xctool -workspace APP.xcworkspace -scheme "APPScheme" test -only UITestsTarget:UITestsClass/UITestsMethod

and i got this error: ...ERROR: build-tests: 'E-UITestsTarget' is not a testing target in this scheme.

masterginger commented 8 years ago

Looks like AWS device farm claimed to support Xcode 7 UI Test: http://docs.aws.amazon.com/devicefarm/latest/developerguide/test-types-ios-xctest-ui.html

b1ueshad0w commented 8 years ago

@johnbill Does that really work?

marekcirkos commented 8 years ago

You can use XCTestBootstrap which is part of FBSimulatorControl to launch xcuitests. We use it to launch https://github.com/facebook/WebDriverAgent on simulators and devices. It might not report test results,but that should not be difficult to add :).

marekcirkos commented 8 years ago

You can also use fbsimctl which is also part of FBSimulatorControl to start xcuitests from command line.

lawrencelomax commented 8 years ago

Am I correct to assume that there are no plans to provide similar support for physical devices?

XCTestBootstrap in isolation should provide support for launching on Devices. XCTestBootstrap has no awareness of FBSimulatorControl but linking with some of the XCode frameworks allows it to do it's magic.

The API may not be as convenient as is currently the case for launch_xctest on fbsimctl, but we don't have anything specific to announce around Open-Sourcing of fbsimctl for devices. It is something that we want to do, but I can't make any firm commitments. It shouldn't take much to wrap up the XCTestBoostrap API in a simple CLI app for targeting Devices. cc @marekcirkos to confirm that this is feasible.

It (fbsimctl) might not report test results,but that should not be difficult to add :).

fbsimctl doesn't currently have a way of redirecting output from launched processes. If the Application into which the XCTest bundle is injected reports test results over stdout/stderr this should be easy for us to pass in a file handle and redirect output to the fbsimctl command line.

oscarmk1 commented 8 years ago

Is there any tool that currently allows to run a XCTest UI test individually, as opposed to the whole suite, like xctool does?. I looked at fastlane/scan, and while it is a very nice wrapper of xcodebuild with xcpretty that also handles many other issues, it is limited to xcodebuild capabilities, meaning you have to build and then test, and also you cannot run individual tests or classes.

lawrencelomax commented 8 years ago

Is there any tool that currently allows to run a XCTest UI test individually, as opposed to the whole suite, like xctool does

Not yet, but I've been playing with getting XCTestBoostrap working with Application tests with a full reporting interface. It connects directly to the testmanagerd service running on the Device/Simulator and receives all the progress callbacks that Xcode/xcodebuild receives. If you pour through the code there, you should see some interesting stuff.

oscarmk1 commented 8 years ago

@lawrencelomax Thanks for the update, I will take a look at XCTestBoostrap, sounds promising.

jafara commented 8 years ago

Any updates on this? Has anyone dig through to support this?

anton-matosov commented 7 years ago

xcodebuild 8+ has 2 new actions build-for-testing and test-without-building tools like plu/pxctest leverage this to allow running tests in multiple simulators at the same time @oscarmk1 you can use schemas to run individual tests based on name or class (P.S. pxctest has --only "class:test" parameter that seems to do what you are looking for, but I have not tried it)

ExtremeMan commented 5 years ago

Better late than never. But I have just pushed changes which should add UI testing support. Let me know if it doesn't work for your use case.