search

aws-samples / aws-device-farm-xctest-ui-tests-for-ios-sample-app

Apache License 2.0
7 stars 13 forks source link

All tests failing but Setup and Teardown #5

Open aw-vm opened 6 years ago

aw-vm commented 6 years ago

Hello,

I have tried running my tests on device farm but all the test are failing. I have followed all the steps of the documentation and generated the files as the instructions said. I have not encountered any issue uploading these files or creating a run with them but when the run is over I get these results:

screen shot 2018-07-30 at 5 30 12 pm

I'm getting this from the automation output of one test:

User defaults from command line:
    IDEDerivedDataPathOverride = /tmp/scratchnZxqrd.scratch/xcode_screenshots7MfdjK/UITests15TwilioE4UITests/testExample
    IDETestRunOnlyIdentifiers = (
    "TwilioE4UITests/UITests15TwilioE4UITests/testExample"
)
    IDETestRunSpecificationPath = /Users/rwx/Library/Developer/Xcode/DerivedData/Build/Products/application.xctestrun

2018-07-30 09:42:09.687 xcodebuild[535:7368]  IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
/var/folders/tg/pk__v_v5785_6_qbhdg297gr0000gp/T/com.apple.dt.XCTest/IDETestRunSession-40A4F518-21AA-49FF-97D0-CDCD75DDF535/TwilioE4UITests-79D669D7-3634-4C57-8276-610333B02474/Session-TwilioE4UITests-2018-07-30_094209-hBcQkX.log
2018-07-30 09:42:09.689 xcodebuild[535:7354] [MT] IDETestOperationsObserverDebug: (032D8571-B44D-4931-A6DA-F7D1FA773B91) Beginning test session TwilioE4UITests-032D8571-B44D-4931-A6DA-F7D1FA773B91 at 2018-07-30 09:42:09.688 with Xcode 9F1027a on target <DVTiOSDevice: 0x7fc66fe14260> {
        deviceSerialNumber:         G6TVH3K9JCL8
        identifier:                 b2f96e8a3a53c1bd94d05fc1d0c06aa94954148a
        deviceClass:                iPhone
        deviceName:                 9902852853
        deviceIdentifier:           b2f96e8a3a53c1bd94d05fc1d0c06aa94954148a
        productVersion:             11.4
        buildVersion:               15F79
        deviceSoftwareVersion:      11.4 (15F79)
        deviceArchitecture:         arm64
        deviceTotalCapacity:        252371152896
        deviceAvailableCapacity:    249001328640
        deviceIsTransient:          NO
        ignored:                    NO
        deviceIsBusy:               NO
        deviceIsActivated:          YES
        deviceActivationState:      Activated
        isPasscodeLocked:           NO
        deviceType:                 <DVTDeviceType:0x7fc66fd80f20 Xcode.DeviceType.iPhone>
        supportedDeviceFamilies:    (
    1
)
        applications:              (null)
        provisioningProfiles:      (null)
        hasInternalSupport:        NO
        isSupportedOS:             YES
        developerDiskMountError:   (null)
(null)
    bootArgs:                  <unavailable>
        connected:                 yes
        isWirelessEnabled:         no
        connectionType:            direct
        } (11.4 (15F79))
MDMCreateDeltaDirectory:1920 calling MDMDirectoryDiff with:
state->old_bundle: /var/folders/tg/pk__v_v5785_6_qbhdg297gr0000gp/C/com.apple.DeveloperTools/All/Xcode/EmbeddedAppDeltas/977ae2eae09fab3f9a27a0f75731a39d/b2f96e8a3a53c1bd94d05fc1d0c06aa94954148a/TwilioE4UITests-Runner.app
state->new_bundle: /Users/rwx/Library/Developer/Xcode/DerivedData/Build/Products/Debug-iphoneos/TwilioE4UITests-Runner.app
state->dst_bundle: /var/folders/tg/pk__v_v5785_6_qbhdg297gr0000gp/C/com.apple.DeveloperTools/All/Xcode/EmbeddedAppDeltas/TwilioE4UITests-Runner.app.aYOj0V/TwilioE4UITests-Runner.app_sparse.ipa/Payload//TwilioE4UITests-Runner.app, binaryDiff flag: FALSE
    dst_ipa: /var/folders/tg/pk__v_v5785_6_qbhdg297gr0000gp/C/com.apple.DeveloperTools/All/Xcode/EmbeddedAppDeltas/TwilioE4UITests-Runner.app.aYOj0V/TwilioE4UITests-Runner.app_sparse.ipa
MDMDirectoryDiff_block_invoke:1473 calling writeDictToFile with: /var/folders/tg/pk__v_v5785_6_qbhdg297gr0000gp/C/com.apple.DeveloperTools/All/Xcode/EmbeddedAppDeltas/TwilioE4UITests-Runner.app.aYOj0V/TwilioE4UITests-Runner.app_sparse.ipa/ManifestCache.plist
writeDictToFile:1278 ==== Successfully wrote Manifest cache to /var/folders/tg/pk__v_v5785_6_qbhdg297gr0000gp/C/com.apple.DeveloperTools/All/Xcode/EmbeddedAppDeltas/TwilioE4UITests-Runner.app.aYOj0V/TwilioE4UITests-Runner.app_sparse.ipa/ManifestCache.plist
** TEST EXECUTE SUCCEEDED **

2018-07-30 11:42:16.421756-0500 TwilioE4UITests-Runner[588:26690] +[CATransaction synchronize] called within transaction
2018-07-30 11:42:16.442034-0500 TwilioE4UITests-Runner[588:26690] Running tests...
2018-07-30 11:42:17.072729-0500 TwilioE4UITests-Runner[588:26690] [AXValidations] UIAccessibility Error adding -[CAMLightingControl accessibilityLabel]
2018-07-30 11:42:17.072828-0500 TwilioE4UITests-Runner[588:26690] [AXValidations] UIAccessibility Error adding -[CAMLightingControl accessibilityValue]
2018-07-30 11:42:17.072880-0500 TwilioE4UITests-Runner[588:26690] [AXValidations] UIAccessibility Error adding -[CAMLightingControl accessibilityTraits]
2018-07-30 11:42:17.072926-0500 TwilioE4UITests-Runner[588:26690] [AXValidations] UIAccessibility Error adding -[CAMLightingControl isAccessibilityElement]
2018-07-30 11:42:17.072977-0500 TwilioE4UITests-Runner[588:26690] [AXValidations] UIAccessibility Error adding -[CAMLightingControl accessibilityIncrement]
2018-07-30 11:42:17.073017-0500 TwilioE4UITests-Runner[588:26690] [AXValidations] UIAccessibility Error adding -[CAMLightingControl accessibilityDecrement]
2018-07-30 11:42:17.073053-0500 TwilioE4UITests-Runner[588:26690] [AXValidations] UIAccessibility Error adding +[CAMLightingControl safeCategoryBaseClass]
2018-07-30 11:42:17.079126-0500 TwilioE4UITests-Runner[588:26690] [AXValidations] UIAccessibility Error adding -[PLManagedAsset accessibilityLabel]
2018-07-30 11:42:17.079200-0500 TwilioE4UITests-Runner[588:26690] [AXValidations] UIAccessibility Error adding -[PLManagedAsset accessibilityValue]
2018-07-30 11:42:17.079233-0500 TwilioE4UITests-Runner[588:26690] [AXValidations] UIAccessibility Error adding -[PLManagedAsset accessibilityTraits]
2018-07-30 11:42:17.079263-0500 TwilioE4UITests-Runner[588:26690] [AXValidations] UIAccessibility Error adding -[PLManagedAsset _accessibilityPhotoDescription]
2018-07-30 11:42:17.079292-0500 TwilioE4UITests-Runner[588:26690] [AXValidations] UIAccessibility Error adding +[PLManagedAsset safeCategoryBaseClass]
2018-07-30 11:42:17.096756-0500 TwilioE4UITests-Runner[588:26690] Continuing to run tests in the background with task ID 1
Test Suite 'Selected tests' started at 2018-07-30 11:42:17.717
Test Suite 'TwilioE4UITests.xctest' started at 2018-07-30 11:42:17.718
Test Suite 'TwilioE4UITests.xctest' passed at 2018-07-30 11:42:17.719.
     Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.001) seconds
Test Suite 'Selected tests' passed at 2018-07-30 11:42:17.720.
     Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.003) seconds

Also checking the System logs I could find this log many times:

Jul 30 11:41:56 9902015857 amfid(Security)[195] <Notice>: could not enable test hierarchy: ApplePinningAllowTestCertsiPhoneApplicationSigning not true

Watching the test videos looks like the app is not opening and nothing is happening there. Reading the logs I found this one: Unable to generate launch image for com.apple.test.TwilioE4UITests-Runner because the app has no valid launch interfaces. screen shot 2018-07-30 at 5 35 01 pm

nikhil-dabhade commented 6 years ago

@valerymadiedo : Few questions:

  1. Have you ran the same tests of a real physical device locally? The reason I ask is at times the test may have been built and run for simulator rather than a real device and error out when running on a real device.
  2. Do you have entitlements in your iOS application? If yes, these will break on Device Farm as it re-signs the app. Using private devices gives you the ability to skip re-signing of your app.
  3. I also see MDM in the logs. Do you require MDM client to be present on the device while the tests are running?
aw-vm commented 6 years ago

Hello @nikhil-dabhade Thanks a lot for your help. Answers:

  1. Yes I have run these tests in a physical device (iphone 5s - ios 11.2.1).
  2. No. The ipa I uploaded to device farm was signed as developer, which is the right way to sign it?
  3. Yes. I save some data locally using sqlite.swift.

Edit: here says I must create ad hoc profile and add the devices and I can find the uuid of the device on the device instance and add these to my provisioning profiles but these devices must be private and I dont have access to these yet.

aw-vm commented 5 years ago

Hi, I have resigned the ipa with an adhoc profile and I'm getting the same behavior. The test opens but is not able to open the app.

af-mke commented 5 years ago

Hey @valerymadiedo , did you solve the problem? I also get these errors and can't figure out what the problem is

croossin commented 4 years ago

Any updates? Facing same issue