POC: invoke UI test directly w/ gcloud

rpappalax commented 3 years ago

NOTE: Need to replace flank!

Flank team will soon stop supporting Flank so we'll need to switch over to a native Firebase API call. We should also determine what we'll be losing by making this switch over

FIREBASE GCLOUD BETA API https://cloud.google.com/sdk/gcloud/reference/beta/firebase/test/android/run#--results-dir

FLANK CODE https://github.com/Flank/flank

NOTE:

must modify Dockerfile to use gcloud beta

STEPS TO COMPLETE

POC run a UI test using gcloud beta - DONE
Move configuration in a yaml file
Determine how to get test run data from Firebase directly (matrix_ids, etc.)
Fix parser script so it has necessary sourcefile (current: flank.log)
Make sure we have all necessary metadata for notifications

gcloud beta artifacts

CostReport.txt FullJUnitReport.xml JUnitReport.xml MatrixResultsReport.txt android-test.apk android_shards.json app.apk matrix_0/ matrix_ids.json

To create a yaml config like flank.yaml

demo-cucumber-E2E-test:
  type: instrumentation
  app: app/build/outputs/apk/demo/debug/app-demo-debug.apk
  test: app/build/outputs/apk/androidTest/demo/debug/app-demo-debug-androidTest.apk
  device:
    - model: Nexus6
      version: 21
      locale: 'en'
      orientation: portrait
  environment-variables:
    cucumberOptions: "--tags '@demo and @e2e'"

UNKNOWNS

can we indicate number of re-runs desired? - READ
capture RPC limits?
is run time longer than with flank - TRY (MEASURE & COMPARE)
how is -1 for number of shards handled with gcloud beta - READ
try out notPackage on specific directories - TRY
how to pull down the necessary artifacts to local directory - READ

AaronMT commented 3 years ago

Notes (10/07):

Step 1:

Download the Firebase CLI
Timebox running a UI test using the Firebase CLI

Step 2:

Move the parameters for executing the test into a Flank file like file. For example, copy Flankx86.yml to firebase-x86.yml and then remove the Flank specific items at the bottom

gcloud firebase test android run \
  --type robo \
  --app app-debug-unaligned.apk \
  --device model=Nexus6,version=21,locale=en,orientation=portrait  \
  --device model=Nexus7,version=19,locale=fr,orientation=landscape \
  --timeout 90s

https://firebase.google.com/docs/test-lab/android/command-line

--num-uniform-shards=int

https://cloud.google.com/sdk/gcloud/reference/beta/firebase/test/android/run#--num-uniform-shards

AaronMT commented 2 years ago

DEVICE="model=Pixel2,version=28,locale=en,orientation=portrait"
GOOGLE_CLOUD_STORAGE_BUCKET="fenix_test_artifacts"
NUM_UNIFORM_SHARDS=1

gcloud beta firebase test android run \
    --type instrumentation \
    --app ${APK_APP} \
    --test ${APK_TEST} \
    --device ${DEVICE} \
    --timeout 30m \
    --results-bucket gs://${GOOGLE_CLOUD_STORAGE_BUCKET} \
    --results-dir ${RESULTS_DIR} \
    --num-uniform-shards ${NUM_UNIFORM_SHARDS} \
    --record-video \
    --num-flaky-test-attempts 1 \
    --no-auto-google-login \
    --test-targets "class org.mozilla.fenix.ui.NavigationToolbarTest#visitURLTest" \
    --use-orchestrator

mozilla-mobile / mobile-test-eng

POC: invoke UI test directly w/ gcloud #453