Eruhenon commented 7 years ago

Hey,

we're trying to run EarlGrey on AWS device farm. It fails with various errors. Since I can't find any information about anyone succeeding to run this (except for https://github.com/google/EarlGrey/issues/154) I thought I'd ask here for advice :)

If anyone could guide me in the right direction, it would be very appreciated!

tirodkar commented 7 years ago

What form errors are you seeing @Eruhenon? Are you using a pre-built IPA to test or using one CocoaPods / Carthage etc?

khandpur commented 7 years ago

Adding @bootstraponline just in case he has something to share.

bootstraponline commented 7 years ago

We're also looking into running our EarlGrey tests on AWS. Step one was making the tests run on a local physical device. I'll comment once we have some tests running successfully on AWS.

bootstraponline commented 7 years ago

Amazon commented:

https://forums.aws.amazon.com/thread.jspa?messageID=772838

I'm not looking into AWS anymore so the Amazon forums are probably the best place for support.

githubist16 commented 6 years ago

Earlgrey works with XCTest and I have tested that it works. Use steps outlined here: https://docs.aws.amazon.com/devicefarm/latest/developerguide/test-types-ios-xctest.html

bootstraponline commented 6 years ago

https://firebase.google.com/docs/test-lab/ios/command-line is what I'm using now.

tirodkar commented 6 years ago

@githubist16 & @bootstraponline is there any way we could add to our docs for support for the different CI platforms?

bootstraponline commented 6 years ago

Linking to https://firebase.google.com/docs/test-lab/ios/firebase-console for firebase is sufficient I think.

githubist16 commented 5 years ago

Finally got this going again using EG 2.0 on AWS Device farm. These were the steps used (some of these steps need to be fleshed out. I will keep updating this):

Contact AWS team to add 1 or more private devices to your account:
- https://docs.aws.amazon.com/devicefarm/latest/developerguide/working-with-private-devices.html
Add the private device(s) to your team's provisioning profile:
- https://stackoverflow.com/questions/3578158/adding-devices-to-team-provisioning-profile
- https://support.smartbear.com/testcomplete/docs/app-testing/mobile/ios/preparing/registering-devices.html
  - This can also be automated using tools such as https://github.com/nomad/cupertino
  - You can also use one of these two methods:
    - (Using this you can use xcode on your local machine to register the device to your profile) https://docs.aws.amazon.com/devicefarm/latest/developerguide/direct-device-access.html
    - (Not sure yet what to with this) https://docs.aws.amazon.com/devicefarm/latest/developerguide/amazon-vpc-endpoints.html
Run steps used to build EG 2.0 tests
- You will need the .app file for your app from here. Do not use the standard method of archiving the app as that is not sufficient.
- Still create a Payload folder with the .app included
Zip up Products folder (Feel free to zip up even more files ex. a shell script you might want to access, etc.)
- ~/Library/Developer/Xcode/DerivedData//Build/Products
- Products.zip should include *.xctestrun file and Debug-iphoneos/ directory
- I called it Products.zip, it can be any name - Aragorn.zip

Products.zip content can look something like

Products
      | - *.xctestrun
      | - Debug-iphoneos/
      | - iOS_Cert.p12 file
      | - run_all_the_things.sh

On AWS Device farm:
- Create a new run
- Under choose application: Upload Payload.ipa file created from step 3. If you got it wrong, Device farm will notify you
- Under configure test: Select Appium Node.js
- Select the Products.zip file
  - Ignore the warning: 'We could not find the tarball file in the root of your test package...

Edit the yaml spec file to look like the following:

Unwanted commands were commented out

Your steps will need to also import your iOS cert (*.p12) for your codesign identity (I did not show this below, but can cleanup mine and add scripts. I did show that cert file is part of the zip file)

Also note that you can access files within your zip and call whatever terminal command you want from the root. DEVICEFARM_TEST_PACKAGE_PATH represents the root of your zip

version: 0.1

Phases are collection of commands that get executed on Device Farm.

phases:

The install phase includes commands that install dependencies that your tests use.

Default dependencies for testing frameworks supported on Device Farm are already installed.

install: commands:

By default, this testspec file uses Appium server version 1.9.1 which is a prerequisite for iOS 12

  # To use a different appium version, change 'APPIUM_VERSION' below to your desired version.
  # For convenience, we have preinstalled the following versions - 1.6.5, 1.7.1, 1.7.2, 1.8.0, 1.8.1, 1.9.1

  # By default the node version installed is 11.4.0
  # you can switch to an alternate node version using below command.
  # - nvm install 10.13.0

  # Unpackage and install the node modules that you uploaded in the test phase.

The pre-test phase includes commands that setup your test environment.

pre_test: commands:

We recommend starting appium server process in the background using the command below.

  # Appium server log will go to $DEVICEFARM_LOG_DIR directory.
  # The environment variables below will be auto-populated during run time.
  - echo "Start appium server"
  # The default WDA used is at DEVICEFARM_WDA_DERIVED_DATA_PATH_V1 (Supports versions iOS 12 and below), it is using commit f865d3. See (https://github.com/appium/appium-xcuitest-driver/tree/f865d32e78a5a8a15469bee30ed2f985d378575d)
  # If you need an older WDA version or need support for node modules, use the WDA at DEVICEFARM_WDA_DERIVED_DATA_PATH_V0. (This version does not suport iOS 12)

The test phase includes commands that run your test suite execution.

test: commands:

Go into the root folder containing your source code and node_modules

  - echo "Navigate to test source code"
  # Change the directory to node_modules folder as it has your test code and the dependency node modules.
  - cd $DEVICEFARM_TEST_PACKAGE_PATH/
  - ls $DEVICEFARM_TEST_PACKAGE_PATH
  - CUSTOM_XCUITESTRUN_FILE=$DEVICEFARM_TEST_PACKAGE_PATH/vEarlGreyTests_iphoneos12.1-arm64e.xctestrun
  - CUSTOM_DERIVED_DATA_PATH=$DEVICEFARM_TEST_PACKAGE_PATH
  - /usr/bin/xcodebuild test-without-building -destination id=$DEVICEFARM_DEVICE_UDID -xctestrun $CUSTOM_XCUITESTRUN_FILE -derivedDataPath $CUSTOM_DERIVED_DATA_PATH -only-testing:<YOURSCHEME>/<YOUR_TEST_CLASS>

  - echo "Start Appium Node test"
  # Enter the command below to start the tests . The comamnd should be similar to what you use to run the tests locally.
  # For e.g. assuming you run your tests locally using command "node YOUR_TEST_FILENAME.js.", enter the same command below:

The post test phase includes are commands that are run after your tests are executed.

post_test: commands:

The artifacts phase lets you specify the location where your tests logs, device logs will be stored.

And also let you specify the location of your test logs and artifacts which you want to be collected by Device Farm.

These logs and artifacts will be available through ListArtifacts API in Device Farm.

artifacts:

By default, Device Farm will collect your artifacts from following directories

$DEVICEFARM_LOG_DIR

Select your private device
Run test

tirodkar commented 5 years ago

That sounds awesome.

QQ -

Contact AWS team to add 1 or more private devices to your account

Could we get a link for how to do this? This would look great in a setup doc. Feel free to send it as a .md PR to us.

google / EarlGrey

Run EarlGrey on AWS #447