realm / aws-devicefarm

Github action for triggering runs on AWS devicefarm
MIT License
17 stars 10 forks source link

** Introduction This repository provides actions to interact with AWS devicefarms, implemented using the [[https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/DeviceFarm.html][AWS JavaScript SDK.]] AWS credentials need to be configured using the [[https://github.com/aws-actions/configure-aws-credentials][configure AWS credentials action]].

Due to the nature of the application this was developed for the following restrictions apply for now:

  1. The application contains the test definitions in the application binary. As AWS always expects a separate test definition a dummy test definition is provided in =sample-venv/tests=. Applications wanting to use the AWS testdefinitions should still work without problem, as long as the correct files are uploaded / correct ARNs specified. This has not been tested, though.
  2. Test results are evaluated on Github side based on files pulled from the device. Support for tests/suites an AWS is not implemented. Adding support for those APIs should be trivial, if needed. Patches are welcome.

[[https://github.com/realm/aws-devicefarm-sample-data][Sample test data and applications]] simulating this case are provided in a separate repository.

Two types of actions are provided: Simple API wrappers - useful for debugging, testing or building workflows otherwise not properly supported - and complex actions, wrapping several API calls. Typically complex actions should be used, whenever possible.

*** Examples After adding secrets for AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_DEVICE_POOL_ARN and AWS_PROJECT_ARN (see AWS setup section below if no devicefarm is configured) the following workflow will create a test run with a mix of downloaded and local data, and wait for it to execute. You may want to adjust =app_file=, =test_spec_file= and =test_package_file= before running it - as shown it will pull demo data for the run:

**** Example with remote sources and artifact pulling

+BEGIN_SRC

on: push: branches: [ master ] pull_request: branches: [ master ]

name: Test application on device farm

jobs: deploy: name: Deploy runs-on: ubuntu-latest environment: production

steps:
- name: Checkout
  uses: actions/checkout@v2

- name: Configure AWS credentials
  uses: aws-actions/configure-aws-credentials@v1
  with:
    aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
    aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
    aws-region: us-west-2

- name: Schedule test run with mixed data
  uses: realm/aws-devicefarm/test-application@master
  with:
    name: schedule-run
    file_artifacts: |
      Test spec file.yml
      Customer Artifacts.zip
    project_arn: ${{ secrets.AWS_PROJECT_ARN }}
    device_pool_arn: ${{ secrets.AWS_DEVICE_POOL_ARN }}
    app_file: https://github.com/realm/aws-devicefarm-sample-data/releases/download/0.3/fi.aardsoft.devicefarmsample-debug.apk
    app_type: ANDROID_APP
    test_type: APPIUM_PYTHON
    test_package_file: https://github.com/realm/aws-devicefarm-sample-data/releases/download/0.3/sample_env_python3.zip
    test_package_type: APPIUM_PYTHON_TEST_PACKAGE
    test_spec_file: https://github.com/realm/aws-devicefarm-sample-data/releases/download/0.3/test_spec.yaml
    test_spec_type: APPIUM_PYTHON_TEST_SPEC
    remote_src: true

+END_SRC

**** Example with remote sources, inline test specification and artifact pulling It is also possible to have the test specification inline:

+BEGIN_SRC

on: push: branches: [ master ] pull_request: branches: [ master ]

name: Test application on device farm

jobs: deploy: name: Deploy runs-on: ubuntu-latest environment: production

steps:
- name: Checkout
  uses: actions/checkout@v2

- name: Configure AWS credentials
  uses: aws-actions/configure-aws-credentials@v1
  with:
    aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
    aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
    aws-region: us-west-2

- name: Schedule test run with mixed data
  uses: realm/aws-devicefarm/test-application@master
  with:
    name: schedule-run-inline
    file_artifacts: |
      Test spec file.yml
      Customer Artifacts.zip
    project_arn: ${{ secrets.AWS_PROJECT_ARN }}
    device_pool_arn: ${{ secrets.AWS_DEVICE_POOL_ARN }}
    app_file: https://github.com/realm/aws-devicefarm-sample-data/releases/download/0.3/fi.aardsoft.devicefarmsample-debug.apk
    app_type: ANDROID_APP
    test_type: APPIUM_PYTHON
    test_package_file: https://github.com/realm/aws-devicefarm-sample-data/releases/download/0.3/sample_env_python3.zip
    test_package_type: APPIUM_PYTHON_TEST_PACKAGE
    test_spec_file: test_spec.yaml
    test_spec_type: APPIUM_PYTHON_TEST_SPEC
    test_spec: |
      version: 0.1
      phases:
        install:
          commands:
            - export PYTHON_VERSION=3

        pre_test:
          commands:
            - adb -s $DEVICEFARM_DEVICE_UDID shell pm grant fi.aardsoft.devicefarmsample android.permission.READ_EXTERNAL_STORAGE
            - adb -s $DEVICEFARM_DEVICE_UDID shell pm grant fi.aardsoft.devicefarmsample android.permission.WRITE_EXTERNAL_STORAGE

        test:
          commands:
            - adb -s $DEVICEFARM_DEVICE_UDID shell am instrument -w -r -e foo bar -e bar baz fi.aardsoft.devicefarmsample

        post_test:
          commands:
            - adb -s $DEVICEFARM_DEVICE_UDID pull /storage/emulated/0/Android/data/fi.aardsoft.devicefarmsample/files/output.txt
            - mv output.txt $DEVICEFARM_LOG_DIR/test-output.txt

      artifacts:
        - $DEVICEFARM_LOG_DIR
    remote_src: true

+END_SRC

** Available actions *** API wrappers The following actions are just wrappers around the [[https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/DeviceFarm.html][JavaScript SDK]]. In addition to the JSON returned by the API they also return the JSON fields most likely to be useful for later calls - saving some JSON parsing compared to building actions on top of the AWS CLI. For each action documentation links to both the API and the matching call in AWS CLI are provided - with the latter typically being the more useful when building a workflow on top of those actions without trying to touch the JavaScript.

**** create-upload Create an upload for a file named =name=, of type =type=, in the project specified by =project_arn=. Note that a file needs to be uploaded to the URL from the result in a second step - for an action handling the upload as well check =upload-file=.

***** inputs

***** outputs

***** documentation

***** usage example

+BEGIN_SRC yaml

A step similar to the following could be used to upload the actual file:

+BEGIN_SRC yaml

The status of the upload should be checked afterwards with =get-upload=, possibly in a loop until the status changes.

**** delete-upload Deletes a resource uploaded to a devicefarm project, specified by =resource_arn=.

***** inputs

***** outputs This method does not return data, and completes successfully even if the resource requested for deletion does not exist.

***** documentation

***** usage example

+BEGIN_SRC yaml

**** get-device-pool Get information about a specific device pool.

***** inputs

***** outputs

***** documentation

***** usage example

+BEGIN_SRC yaml

**** get-project Return details of the given project.

***** inputs

***** outputs

***** documentation

***** usage example

+BEGIN_SRC yaml

**** get-run Return details for a specific test run.

***** inputs

***** outputs

***** documentation

***** usage example Typically the ARN is obtained from an earlier step:

+BEGIN_SRC yaml

**** get-upload Return details for the upload specified by =resource_arn=.

***** inputs

***** outputs

***** documentation

***** usage example

+BEGIN_SRC yaml

**** list-artifacts List artifacts for a resource. Note that you can only list one artifact type at one time, and can't query specific artifact names. To retrieve a specific artifact loop over the returned array, check for artifact names, and GET the included URL to retrieve the artifact.

***** inputs

***** outputs

***** documentation

***** usage example

+BEGIN_SRC

**** list-device-pools Return a list of device pools configured for the specified project.

***** inputs

***** outputs

***** documentation

***** usage example

+BEGIN_SRC yaml

**** list-runs List test runs in the specified device farm project.

***** inputs

***** outputs

***** documentation

***** usage example

+BEGIN_SRC yaml

**** list-uploads Return all uploads in the project specified by =project_arn= as a JSON string.

***** inputs

***** outputs

***** documentation

***** usage example

+BEGIN_SRC yaml

The output can be referenced in following actions using the step ID:

+BEGIN_SRC yaml

with: input: ${{ steps.list-uploads.outputs.data }}

+END_SRC

**** schedule-run Schedule a test run with resources uploaded to AWS already. This action returns directly after scheduling a run - this behaviour is useful to avoid blocking a workflow if other steps can still be executed, but requires later checking with =get-run= if the run has finished. For an action capable of uploading required files as well as blocking until a test run has finished see =test-application=.

When called without a test specification the run will be triggered with the default test environment of the specified test type.

***** inputs

***** outputs

***** documentation

***** usage example This example references ARNs obtained from previous upload steps:

+BEGIN_SRC yaml

* Complex actions ** download-artifacts This action downloads one or more artifacts from a test run. Trying to download a non-existent artifact will log a warning and omit the file frem the output, but not abort.

***** inputs

***** outputs

***** usage example

This assumes the run with id =schedule-run= created customer artifacts:

+BEGIN_SRC yaml

The returned JSON looks like this - note the missing invalid file:

+BEGIN_SRC js

{ "FILE": [ "Test spec file.yml", "Customer Artifacts.zip" ], "SCREENSHOT": [ ], "LOG": [ ] }

+END_SRC

**** upload-file This action creates a file upload and then uploads a file.

***** inputs

***** outputs

***** usage example

+BEGIN_SRC yaml

**** test-application This action schedules a test run and waits for the result. It can either use already uploaded files, or upload local or remote files.

When called without a test specification the run will be triggered with the default test environment of the specified test type.

***** inputs

***** outputs

***** usage example This example pulls remote sources, and uploads them to a device farm before scheduling a run:

+BEGIN_SRC yaml

** Setting up AWS *** Create a test project In the [[https://us-west-2.console.aws.amazon.com/devicefarm/home][Device Farm console]] create a new project, and copy the displayed ARN - this is the project ARN required by some actions. Next go to =Project settings=, =Device pools= and create a new pool. Retrieve the pool ARN using the [[https://docs.aws.amazon.com/cli/latest/reference/devicefarm/list-device-pools.html][list-device-pools]] CLI command, or using the list-device-pools action if AWS CLI is not set up.

Secrets named AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_DEVICE_POOL_ARN and AWS_PROJECT_ARN or similar need to be set using the ARNs just created and account secrets.

It is recommended to use a dedicated role user for devicefarm access. the AWS role to limit access to devicefarm only is =AWSDeviceFarmFullAccess=.

** Development *** Contributing A pull request should also include build results, built with the [[https://github.com/realm/aws-devicefarm/blob/master/.github/workflows/integration-test.yml#L29][same node version as used in CI]] (at the time of writing this: 16):

+BEGIN_SRC bash

$ npm ci [...] $ npm run package

aws-devicefarm@1.0.0 package node tools/build.js

Building create-upload [...]

+END_SRC

[[https://github.com/nvm-sh/nvm][nvm]] is an easy way to add additional node versions to the local machine. After following the instructions there:

+BEGIN_SRC bash

$ nvm install 16 Downloading and installing node v16.20.2... Downloading https://nodejs.org/dist/v16.20.2/node-v16.20.2-linux-x64.tar.xz... --2023-10-02 12:00:42-- https://nodejs.org/dist/v16.20.2/node-v16.20.2-linux-x64.tar.xz Resolving nodejs.org (nodejs.org)... 104.20.22.46, 104.20.23.46, 2606:4700:10::6814:162e, ... Connecting to nodejs.org (nodejs.org)|104.20.22.46|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 22556484 (22M) [application/x-xz] Saving to: ‘/home/user/.nvm/.cache/bin/node-v16.20.2-linux-x64/node-v16.20.2-linux-x64.tar.xz’

/home/user/.nvm/.cache 100%[======================================>] 21.51M 2.70MB/s in 8.0s

2023-10-02 12:00:50 (2.71 MB/s) - ‘/home/user/.nvm/.cache/bin/node-v16.20.2-linux-x64/node-v16.20.2-linux-x64.tar.xz’ saved [22556484/22556484]

Computing checksum with sha256sum Checksums matched! Now using node v16.20.2 (npm v8.19.4)

+END_SRC

*** Local testing Local testing is possible to some extend using [[https://github.com/nektos/act][act]]. All secrets should be exported as environment variable, otherwise act will prompt for it:

+BEGIN_SRC bash

$ act -s AWS_ACCESS_KEY_ID -s AWS_SECRET_ACCESS_KEY -s AWS_DEVICE_POOL_ARN -s AWS_PROJECT_ARN

+END_SRC