search

facebookarchive / xctool

An extension for Apple's xcodebuild that makes it easier to test iOS and macOS apps.
Apache License 2.0
6.91k stars 738 forks source link

xctool fails to run tests in Xcode 10.2.1 after Xcode 11 has been installed ("Invalid device state") -- need to specify "--standalone" #766

Closed aaronclarke closed 5 years ago

aaronclarke commented 5 years ago

I'm on xctool version 0.3.7. My team has had problems with running xctool against older Xcode versions after we've installed Xcode 11. xctool runs just fine against Xcode 11, but trying to swap back after we're done testing some beta feature is impossible. Our understanding is that Apple changed how the simulator works in a way that breaks backwards compatibility.

/Volumes/Source/git/facebook_xctool/xctool.sh -showTasks -sdk iphonesimulator  run-tests -test-sdk iphonesimulator -logicTest /Path/to/my.xctest -only /Path/to/my.xctest:Automation_iOS_unit_tests/testRetrievedModelIsIpadOrIphone -testTimeout 20.0 -destination "platform=iOS Simulator,OS=latest,name=iPhone 8"

================================================================================
LAUNCHING TASK (finding Xcode path via xcode-select --print-path):

  /usr/bin/xcode-select \
    --print-path
================================================================================

================================================================================
LAUNCHING TASK (finding Xcode path via xcode-select --print-path):

  /usr/bin/xcode-select \
    --print-path
================================================================================

================================================================================
LAUNCHING TASK (spawning reporter task):

  /Volumes/Source/git/facebook_xctool/build/60b63dc/10E1001/Products/Release/reporters/pretty
================================================================================

================================================================================
LAUNCHING TASK (spawning reporter task):

  /Volumes/Source/git/facebook_xctool/build/60b63dc/10E1001/Products/Release/reporters/user-notifications
================================================================================

================================================================================
LAUNCHING TASK (querying available SDKs):

  /Applications/Xcode_10.2.1.app/Contents/Developer/usr/bin/xcodebuild \
    -sdk \
    -version
================================================================================

=== RUN-TESTS ===

  [Info] Collecting info for testables...
================================================================================
LAUNCHING TASK (running otest-query):

  SIMCTL_CHILD___CFPREFERENCES_AVOID_DAEMON=YES \
  SIMCTL_CHILD_DYLD_FALLBACK_FRAMEWORK_PATH=/Applications/Xcode_10.2.1.app/Contents/Developer/Library/Frameworks:/Applications/Xcode_10.2.1.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/Library/Frameworks \
  SIMCTL_CHILD_DYLD_FRAMEWORK_PATH=/Path/to \
  SIMCTL_CHILD_OTEST_QUERY_OUTPUT_FILE=/var/folders/8r/_tsq9x0n43j8g2vgrsw9n9140000gn/T/xctool_temp_99Shly/otest-query-output.ZkROV0T \
  SIMCTL_CHILD_DYLD_INSERT_LIBRARIES=/Volumes/Source/git/facebook_xctool/build/60b63dc/10E1001/Products/Release/lib/otest-query-lib-ios.dylib:/Path/to/my.xctest/Frameworks/libclang_rt.asan_iossim_dynamic.dylib \
  SIMCTL_CHILD_DYLD_LIBRARY_PATH=/Path/to \
  SIMCTL_CHILD_OtestQueryBundlePath=/Path/to/my.xctest \
  /Applications/Xcode_10.2.1.app/Contents/Developer/usr/bin/simctl \
    spawn \
    8729F887-B618-4232-A21E-31B74CF20A5A \
    /Volumes/Source/git/facebook_xctool/build/60b63dc/10E1001/Products/Release/libexec/otest-query-ios
================================================================================
  [Info] Collecting info for testables... (264 ms)
  run-test my.xctest (iphonesimulator12.2, iPhone 8, logic-test)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Failed to query the list of test cases in the test bundle: 
stdout:

stderr:
An error was encountered processing the command (domain=com.apple.CoreSimulator.SimError, code=165):
Invalid device state
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Failures:

  0) /Path/to/my.xctest (my.xctest)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Failed to query the list of test cases in the test bundle: 
stdout:

stderr:
An error was encountered processing the command (domain=com.apple.CoreSimulator.SimError, code=165):
Invalid device state
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

** RUN-TESTS FAILED: 0 passed, 0 failed, 1 errored, 1 total ** (265 ms)

According to a radar bug we filed with Apple, this is by design, and xctool needs to provide the "--standalone" flag when calling "simctl spawn"

Annoyingly, this option has no documentation other than the fact that it exists

xcrun simctl help spawn
Spawn a process by executing a given executable on a device.
Usage: simctl spawn [-w | --wait-for-debugger] [-s | --standalone] [-a <arch> | --arch=<arch>] <device> <path to executable> [<argv 1> <argv 2> ... <argv n>]

The path to the executable is searched using the following rules:
   <path> contains no / characters: search the device's $PATH. This is similar to how most shells work, but searches the device's path instead of the host's path.
   <path> starts with /: Assume a literal path to the binary. This must start from the host's root.
   <path> contains non-leading / characters: search relative to the current directory first, then relative to the device's $SIMULATOR_ROOT.

If you want to set environment variables in the resulting environment, set them in the calling environment with a SIMCTL_CHILD_ prefix.

But I feel like I've reverse engineered the meaning. Previously xctool was perfectly fine calling "simctl spawn " without a care in the world about whether or not the simulator was booted at that time. If it was booted, the process would run in that simulator. But if it wasn't booted, simctl would magically handle the lifecycle management of booting that simulator, running the process, and then shutting down afterwards. Apple specifically broke that. Now if you want that behavior, you need to specify "--standalone"

/Volumes/Source/git/facebook_xctool/xctool.sh -showTasks -sdk iphonesimulator  run-tests -test-sdk iphonesimulator -logicTest /Path/to/my.xctest -only /Path/to/my.xctest:Automation_iOS_unit_tests/testRetrievedModelIsIpadOrIphone -testTimeout 20.0 -destination "platform=iOS Simulator,OS=latest,name=iPhone 8"
=== BUILDING XCTOOL ===

  /Volumes/Source/git/facebook_xctool/scripts/build.sh
      ✓ Built xctool (4000 ms)

================================================================================
LAUNCHING TASK (finding Xcode path via xcode-select --print-path):

  /usr/bin/xcode-select \
    --print-path
================================================================================

================================================================================
LAUNCHING TASK (finding Xcode path via xcode-select --print-path):

  /usr/bin/xcode-select \
    --print-path
================================================================================

================================================================================
LAUNCHING TASK (spawning reporter task):

  /Volumes/Source/git/facebook_xctool/build/60b63dc/10E1001/Products/Release/reporters/pretty
================================================================================

================================================================================
LAUNCHING TASK (spawning reporter task):

  /Volumes/Source/git/facebook_xctool/build/60b63dc/10E1001/Products/Release/reporters/user-notifications
================================================================================

================================================================================
LAUNCHING TASK (querying available SDKs):

  /Applications/Xcode_10.2.1.app/Contents/Developer/usr/bin/xcodebuild \
    -sdk \
    -version
================================================================================

=== RUN-TESTS ===

  [Info] Collecting info for testables...
================================================================================
LAUNCHING TASK (running otest-query):

  SIMCTL_CHILD___CFPREFERENCES_AVOID_DAEMON=YES \
  SIMCTL_CHILD_DYLD_FALLBACK_FRAMEWORK_PATH=/Applications/Xcode_10.2.1.app/Contents/Developer/Library/Frameworks:/Applications/Xcode_10.2.1.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/Library/Frameworks \
  SIMCTL_CHILD_DYLD_FRAMEWORK_PATH=/Path/to \
  SIMCTL_CHILD_OTEST_QUERY_OUTPUT_FILE=/var/folders/8r/_tsq9x0n43j8g2vgrsw9n9140000gn/T/xctool_temp_fqQzem/otest-query-output.qRQ3cgH \
  SIMCTL_CHILD_DYLD_INSERT_LIBRARIES=/Volumes/Source/git/facebook_xctool/build/60b63dc/10E1001/Products/Release/lib/otest-query-lib-ios.dylib:/Path/to/my.xctest/Frameworks/libclang_rt.asan_iossim_dynamic.dylib \
  SIMCTL_CHILD_DYLD_LIBRARY_PATH=/Path/to \
  SIMCTL_CHILD_OtestQueryBundlePath=/Path/to/my.xctest \
  /Applications/Xcode_10.2.1.app/Contents/Developer/usr/bin/simctl \
    spawn \
    --standalone \
    8729F887-B618-4232-A21E-31B74CF20A5A \
    /Volumes/Source/git/facebook_xctool/build/60b63dc/10E1001/Products/Release/libexec/otest-query-ios
================================================================================
  [Info] Collecting info for testables... (917 ms)
  run-test my.xctest (iphonesimulator12.2, iPhone 8, logic-test)

================================================================================
LAUNCHING TASK (running otest/xctest on test bundle):

  SIMCTL_CHILD_HOME=/Users/aaclarke/Library/Developer/CoreSimulator/Devices/8729F887-B618-4232-A21E-31B74CF20A5A/data \
  SIMCTL_CHILD_DYLD_FALLBACK_FRAMEWORK_PATH=/Applications/Xcode_10.2.1.app/Contents/Developer/Library/Frameworks:/Applications/Xcode_10.2.1.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/Library/Frameworks \
  SIMCTL_CHILD_OTEST_SHIM_STDOUT_FILE=/var/folders/8r/_tsq9x0n43j8g2vgrsw9n9140000gn/T/xctool_temp_fqQzem/output.lG8krfj \
  SIMCTL_CHILD_DYLD_FRAMEWORK_PATH=/Path/to \
  SIMCTL_CHILD_TMPDIR=/Users/aaclarke/Library/Developer/CoreSimulator/Devices/8729F887-B618-4232-A21E-31B74CF20A5A/data/tmp/ \
  SIMCTL_CHILD_NSUnbufferedIO=YES \
  SIMCTL_CHILD_XCTestConfigurationFilePath=/var/folders/8r/_tsq9x0n43j8g2vgrsw9n9140000gn/T/xctool_temp_fqQzem/(null)-AE336268-4E67-4CB0-8304-4DEB51658A03.Yax11Ek.xctestconfiguration \
  SIMCTL_CHILD_DYLD_INSERT_LIBRARIES=/Volumes/Source/git/facebook_xctool/build/60b63dc/10E1001/Products/Release/lib/otest-shim-ios.dylib:/Path/to/my.xctest/Frameworks/libclang_rt.asan_iossim_dynamic.dylib \
  SIMCTL_CHILD_DYLD_LIBRARY_PATH=/Path/to \
  SIMCTL_CHILD_CFFIXED_USER_HOME=/Users/aaclarke/Library/Developer/CoreSimulator/Devices/8729F887-B618-4232-A21E-31B74CF20A5A/data \
  SIMCTL_CHILD_OTEST_SHIM_TEST_TIMEOUT=20 \
  /Applications/Xcode_10.2.1.app/Contents/Developer/usr/bin/simctl \
    spawn \
    --standalone \
    8729F887-B618-4232-A21E-31B74CF20A5A \
    /Applications/Xcode_10.2.1.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/Library/Xcode/Agents/xctest \
    -NSTreatUnknownArgumentsAsOpen \
    NO \
    -ApplePersistenceIgnoreState \
    YES
================================================================================
      -[Automation_iOS_unit_tests testRetrievedModelIsIpadOrIphone]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
2019-08-19 18:18:36.835 xctest[27410:29189675] model = iPhone
━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    ✓ -[Automation_iOS_unit_tests testRetrievedModelIsIpadOrIphone] (15013 ms)
    1 passed, 0 failed, 0 errored, 1 total (15014 ms)

** RUN-TESTS SUCCEEDED: 1 passed, 0 failed, 0 errored, 1 total ** (16496 ms)
aaronclarke commented 5 years ago

This may be obvious, but when I added @"--standalone", to the arguments list in xctool/xctool/SimulatorWrapper/SimulatorTaskUtils.m, everything started working from there.

qyang-nj commented 5 years ago

We're seeing this too! Before Xcode 11, simctl spawn will automatically run in standalone mode if the device is not booted. Xcode 11 starts requiring --standalone to be explicit.

ExtremeMan commented 5 years ago

Thanks for a bug report. Can someone put up a PR? Otherwise I will fix it within next several weeks

qyang-nj commented 5 years ago

Thanks for the quick response. I can have a PR this week.

ExtremeMan commented 5 years ago

I just merged a fix from @qyang-nj. Sorry for waiting for me. Thanks for jumping on the issue, reporting and fixing it! I highly appreciate your help!

moatazeldebsy commented 5 years ago

where is version 0.3.7? because I have the same problem with Xcode 11.1 and the latest releases from the tool is 0.3.6

aaronclarke commented 5 years ago

I was on 3.7 because I had pulled the repo and compiled directly from source before reporting my issue.

moatazeldebsy commented 5 years ago

ok , so for solving this issue what shall I do ?

ExtremeMan commented 5 years ago

If you cannot build from master, I will need to create a new release. I can do that

moatazeldebsy commented 5 years ago

@ExtremeMan thank you for your help and yes please if you can new release including a soltuion for this issue I really appreciate that.

moatazeldebsy commented 5 years ago

Hi @ExtremeMan any update about this issue?

aturlier-pro commented 4 years ago

Hi @ExtremeMan, @moatazeldebsy. Any news about the 0.3.7 release? I'd like to avoid building from master if possible.

ExtremeMan commented 4 years ago

New release is out - https://github.com/Homebrew/homebrew-core/pull/47718. I deeply sorry for a long delay!

ExtremeMan commented 4 years ago

@moatazeldebsy, @aturlier thanks for reminding to do it! @qyang-nj thanks for your PR!

aturlier-pro commented 4 years ago

Thanks a lot for the update !