MobileNativeFoundation / bluepill

Bluepill is a reliable iOS testing tool that runs UI tests using multiple simulators on a single machine
BSD 2-Clause "Simplified" License
3.19k stars 232 forks source link

Support running arbitrary scripts on test case start and end #439

Closed bogo closed 1 year ago

bogo commented 4 years ago

Resolves #122 and #385.

As we are working on integrating Bluepill into our CI, we realized we need to get more information out of the tests than Bluepill can currently provide. In our case, those are:

  1. screen recordings of failed tests;
  2. system logs for apps under test.

Since such features are heavily use-case dependent, they probably should not be part of the main Bluepill distribution - but they are easily scriptable, given the correct hooks. This PR adds ability to execute arbitrary scripts at the beginning and end of each test case.

Since I'm pretty new to the Bluepill codebase, I am not sure I am hanging the scripts off in the best possible place. (currently the only place "aware" of individual test cases is the SimulatorMonitor) Also, would appreciate a look on whether I'm using the configuration parameters correctly. Any feedback is welcome!

For the sake of completeness, after merging this PR, one could record videos and grab os_logs as follows:

#!/bin/bash
# test_started.sh

tmp_dir=/tmp/$BP_SIMULATOR_UDID
artifact_path=../../artifacts/$BP_TEST_SUITE-$BP_TEST_CASE-$BP_ATTEMPT_NUMBER
mkdir -p $tmp_dir

xcrun_pidfile=$tmp_dir/xcrun.pid
log_pidfile=$tmp_dir/log.pid

app_name=$(
    case $BP_HOST_BUNDLE_ID in
        ("com.getdropbox.DropboxUITests.xctrunner") echo "Dropbox.app" ;;
        (*) echo $BP_HOST_BUNDLE_ID ;;
    esac
)

predicate="(processImagePath contains \"$app_name\" AND senderImagePath contains \"CoreFoundation\") OR (senderImagePath contains \"$app_name\")"

log stream --predicate "$predicate" --style=compact > $artifact_path.log &
echo $! > $log_pidfile

xcrun simctl io $BP_SIMULATOR_UDID recordVideo --codec=h264 $artifact_path.mov &
echo $! > $xcrun_pidfile
#!/bin/bash
# test_ended.sh

TMP_DIR=/tmp/$BP_SIMULATOR_UDID
PIDFILES=(xcrun.pid log.pid)
for pidfile in "${PIDFILES[@]}"; do
    pkill -SIGINT -F $TMP_DIR/$pidfile
    rm $TMP_DIR/$pidfile
done

ARTIFACT_PATH=../../artifacts/$BP_TEST_SUITE-$BP_TEST_CASE-$BP_ATTEMPT_NUMBER
if [ "$BP_TEST_STATE" = "PASSED" ]; then
    rm $ARTIFACT_PATH.*
fi
ob commented 3 years ago

Is this still a work in progress? Should we review it?