search

facebook / idb

idb is a flexible command line interface for automating iOS simulators and devices
https://fbidb.io
MIT License
4.52k stars 432 forks source link

Invoking test with XCTestBootstrap on real device #274

Closed marekcirkos closed 8 years ago

marekcirkos commented 8 years ago

Moved issue from https://github.com/facebook/WebDriverAgent/issues/114#issuecomment-229023492

@marekcirkos After investigate the fbsimctl sources and flows. I'm starting to port fbsimctl to fbdevicectl. However I'm stuck at FBXCTestPreparationStrategy

(instancetype)strategyWithApplicationPath:(NSString )applicationPath applicationDataPath:(NSString )applicationDataPath testBundlePath:(NSString *)testBundlePath; Could I confirm sth?

applicationPath: is path to .app which is generated by Xcode after build test applicationDataPath: is path to .xcappdata which is download by Xcode -> Window -> Devices -> DownloadContainer testBundlePath: is path to .xctest that is subfolder in *.app Otherwise, in testBundlePath for real device doesn't have .xctestconfiguration

Here is structure of *.app

WebDriverAgentRunner-Runner.app

marekcirkos commented 8 years ago

Unfortunately this is quite different. applicationDataPath requires path to xcappdata that is prepared in certain way. I got good news for you tho:

nghiadhd commented 8 years ago

Hi @marekcirkos

I tried to create xcappdata by XCode, also hardcode platformDirectory on "FBApplicationDataPackageBuilder" point to "iPhoneOS.platform"

However I still cannot connect to test bundle (I got timeout), event I tried to load private testing frameworks. Tomorrow, I'll try to resolve it and give you a log file

  if (!self.platformDirectory) {
    self.platformDirectory = @"/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform";
  }

  if (self.platformDirectory) {
    XCTestFrameworkPath = [self.platformDirectory stringByAppendingPathComponent:@"Developer/Library/Frameworks/XCTest.framework"];
    IDEBundleInjectionFrameworkPath = [self.platformDirectory stringByAppendingPathComponent:@"Developer/Library/PrivateFrameworks/IDEBundleInjection.framework"];
    testBundlePath = self.testBundle.path;
    workingDirectory = packageBundlePath;
  }
marekcirkos commented 8 years ago

Ok to give you quick unblocking information:

nghiadhd commented 8 years ago

Hi @marekcirkos

I tried to setup the paths follow your hint and do some more tricks on FBApplicationDataPackage

  self.platformDirectory = @"/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform";
  if (self.platformDirectory) {
    NSAssert(self.platformDirectory, @"platformDirectory is required to create data package");
    XCTestFrameworkPath = [self.platformDirectory stringByAppendingPathComponent:@"Developer/Library/Frameworks/XCTest.framework"];
    IDEBundleInjectionFrameworkPath = [self.platformDirectory stringByAppendingPathComponent:@"Developer/Library/PrivateFrameworks/IDEBundleInjection.framework"];
    testBundlePath = self.testBundle.path;
    workingDirectory = packageBundlePath;
  }

Now and can prepare strategy for real device. However I still cannot connect to test bundle Result:

Expected: I got message "Got proxy channel request from test bundle"

Actually: I got message "startTestManagerWithAttributes error:Error Domain=com.facebook.XCTestBootstrap Code=0 "Failed connect to test runner or test manager daemon" UserInfo={NSLocalizedDescription=Failed connect to test runner or test manager daemon}"

Log: XCBootstrap_device: https://gist.github.com/nghiadhd/e9fb82d1c0a8c07d6533fe2b65acbaa8 XCBootstrap_simulator: https://gist.github.com/nghiadhd/1695282491127b381e8a5140f78eead9

I also tried to debug on DTXTransport & DTXConnection but cause's by lacking document so I still don't know how to use them in correct way

nghiadhd commented 8 years ago

@marekcirkos I know DTXConnectionServices & XCTest are private frameworks, so no any document from Apple. But if you have any draft document, please share with me because target is use lockdown protocol base on DTXConnectionServices to communicate with testmanagerd & xctest

ledinhphuong commented 8 years ago

@nghiadhd @marekcirkos I can invoke WebDriverAgent by XCTestBootstrap on real device after some hard-codes. The necessary thing is code sign that needs to resign all bundles in .xcappdata. Let's see my workaround at line

-withCodesignProvider:self.codesignProvider]
+withCodesignProvider:[FBCodeSignCommand codeSignCommandWithIdentityName:@"My identity name"]]

More logs if you configure correct:

/path/to/your.xcappdata/AppData/tmp/WebDriverAgentRunner.xctest: replacing existing signature /path/to/your.xcappdata/AppData/tmp/XCTest.framework: replacing existing signature /path/to/your.xcappdata/AppData/tmp/IDEBundleInjection.framework: replacing existing signature

Notice: Please run security find-identity command to find your identity name.

ledinhphuong commented 8 years ago

@marekcirkos

applicationPath is a path to application you want to use to execute test runner (can't be tested application). You can use a dummy app that does nothing.

In my side, it doesn't respond when use dummy app. For example: I don't see the respond when I post curl -X POST $JSON_HEADER -d "" $DEVICE_URL/homescreen message.

It works well if applicationPath is pointed to WebDriverAgentRunner-Runner.app.

nghiadhd commented 8 years ago

Thanks @ledinhphuong

Now I can invoke WebDriverAgent on real device Notice: the applicationPath, applicationDataPath and testBundlePath must be pointed to same application (WebDriverAgent), else I got security issue 

Jul  1 13:54:26 iPhone-6 cloudd[142] <Error>: <CKDGetRecordZonesURLRequest: 0x1256bc490> Failed to get push token.
Jul  1 13:54:26 iPhone-6 securityd[98] <Error>:  securityd_xpc_dictionary_handler cloudd[142] copy_matching Error Domain=NSOSStatusErrorDomain Code=-50 "query missing class name" UserInfo={NSDescription=query missing class name}
Jul  1 13:54:26 iPhone-6 cloudd[142] <Error>:  SecOSStatusWith error:[-50] Error Domain=NSOSStatusErrorDomain Code=-50 "query missing class name" UserInfo={NSDescription=query missing class name}

Now, hopefully @marekcirkos support auto gen .xcappdata

lawrencelomax commented 8 years ago

security find-identity is something that XCTestBootstrap could run in the absence of an explicit identity passed via a constructor. Using an identity from this output should hopefully provide a 'reasonable default' for codesigning.

marekcirkos commented 8 years ago

Let me stress again that FBDeviceTestPreparationStrategy assumes that all bundles are prepared and codesigned already. Were are working on separate strategy that will take care of that part as well.

@ledinhphuong @nghiadhd Pointing applicationPath to WebDriverAgentRunner-Runner.app might not be a good idea as it will also load xctest bundle. We use stab app application that only prevents from being killed or being put into background (then WDA stops responding).

nghiadhd commented 8 years ago

@marekcirkos Regard to signCode identity, I think we can pass it via command-line arguments or sign all bundles manually in .xcappdata (we should skip prepare application bundle steps)

Otherwise If we point to dummy application, may info from .xctest and info from .xcappdata don't match (name or bundlID) and cause securityd issue. I'll continue investigate and inform you if I have any result

nghiadhd commented 8 years ago

Hi @marekcirkos

Today I tried to run XCTestBootstrap (startTestManagerWithAttributes) for multiple real devices concurrently.

However I got issue

[Error Domain=com.facebook.XCTestBootstrap Code=0 "Failed to determine test runner process PID" UserInfo={NSLocalizedDescription=Failed to determine test runner process PID}]

I debug and see that these error devices block at "testmanagerd handled session request using protcol version" message. I expected they have got "Got proxy channel request from test bundle" but not yet

Do you have any idea to resolve it?

marekcirkos commented 8 years ago

What iOS version was it?

nghiadhd commented 8 years ago

@marekcirkos

I confirm that if I use XCTestBootstrap to invoke test runner at the same time for 6 devices, only one device can be handshake with testmanagerd, other devices got exception. But if I start one by one, it works well

iPhone 6s: iOS 9.3.4 iPhone 6: iOS 9.3.2 iPhone 5s: iOS 9.3.4 iPad mini 2: iOS 9.3.4 iPad Air 2: iOS 9.0.2 iPhone 5: iOS 9.3.1

lawrencelomax commented 8 years ago

Closing due to inactivity. Feel free to reopen with more information or discussion.

aluedeke commented 8 years ago

@marekcirkos @nghiadhd @ledinhphuong it there any chance that you share your code snippets?

phoebusliang commented 7 years ago

@marekcirkos, Does FB have a plan to implement the fbsimctl on real devices?

zsol commented 7 years ago

@wacban might have some insights on this one

lawrencelomax commented 7 years ago

I've just landed some changes that make it easier to bootstrap using a swift script wrapper of the underlying API.

From this is should be pretty trivial to make a Swift script that uses the FBSimulatorControl and FBDeviceControl APIs to bootstrap a WebDriverAgent test bundle.

lawrencelomax commented 7 years ago

Here's a rough draft of what this could look like in FBSimulatorControl:

#!/usr/bin/env xcrun swift -F /usr/local/Frameworks

import Foundation
import FBSimulatorControl
import XCTestBootstrap

// Create the FBSimulatorControl Instance.
let options = FBSimulatorManagementOptions()
let config = FBSimulatorControlConfiguration(deviceSetPath: nil, options: options)
let logger = FBControlCoreGlobalConfiguration.defaultLogger()
let control = try FBSimulatorControl.withConfiguration(config, logger: logger)

// Extract the Arguments
let arguments = Array(ProcessInfo.processInfo.arguments.suffix(3))
let udid = arguments[0]
let wdaBundle = arguments[1]
let port = arguments[2]

// Get the Target
let query = FBiOSTargetQuery.udids([udid])
let results = control.set.query(query)
let simulator = results.first!
print("Using \(simulator)")

// If it is booted, keep it booted, otherwise boot it.
if (simulator.state != .booted) {
  print("Booting Simulator \(simulator)")
  try simulator.bootSimulator()
}

// List the Installed Apps and get the first installed app
let applications = simulator.installedApplications()
let application = applications.first!

// Configure an App Launch
let appLaunch = FBApplicationLaunchConfiguration(
  application: application,
  arguments: ["--port", port],
  environment: [:],
  output: FBProcessOutputConfiguration.outputToDevNull()
)
// Wrap this in a Test Launch Configuration
let testLaunch = FBTestLaunchConfiguration(testBundlePath: wdaBundle)
  .withApplicationLaunchConfiguration(appLaunch)
  .withUITesting(true)

// Start the Test Launch
print("Launching \(application)")
try simulator.startTest(with: testLaunch)
try simulator.waitUntilAllTestRunnersHaveFinishedTesting(withTimeout: 100000000)

This assumes that you have: 1) Installed fbsimctl from HEAD using brew install fbsimctl --HEAD (this path is important in the !# 2) Invoke the script with a valid UDID, WebDriverAgent.xctest bundle and a port number for WebDriverAgent to bind on.

An invocation would look something like this:

./bootstrap_wda.swift 9B865108-DF0F-4731-AEEC-01EF24146929 /path/to/WebDriverAgentRunner.xctest 8090

Which would launch WebDriverAgent for the simulator 9B865108-DF0F-4731-AEEC-01EF24146929, launching the prebuild WebDriverAgent.xctest bundle at /path/to/WebDriverAgentRunner.xctest, with WebDriverAgent binding on port 8090.

It should be possible to build something equivalent for FBDeviceControl, but with the appropriate APIs available to you.