Running multiple flows in multiple iOS simulators in parallel fails

[theoriginalgri]

In our CI, we are trying to run multiple flows on multiple iOS Simulators in parallel. From #859, it looked like this would be possible, so I gave it a try and failed.

Even though starting multiple simulators works and maestro starts the app on each of the simulators, it then suddenly only controls one of them. I guess this is probably due to maestro working on a fixed port which prevents it from running multiple apps in parallel. This can also be seen from the screenshot which shows that all three parallel runs are convinced they swiped while only one of them actually happened (flow fails because of German Simulator).

In addition, random exceptions are happening since probably all instances try to use the same temporary files:

java.io.IOException: Error on ZipFile /var/folders/1f/b00_7gmx28x7pdg555wtny6w0000gn/T/maestro-driver-iosUITests-Runner.zip
    at org.apache.commons.compress.archivers.zip.ZipFile.<init>(ZipFile.java:383)
    at org.apache.commons.compress.archivers.zip.ZipFile.<init>(ZipFile.java:261)
    at org.apache.commons.compress.archivers.zip.ZipFile.<init>(ZipFile.java:230)
    at org.apache.commons.compress.archivers.zip.ZipFile.<init>(ZipFile.java:213)
    at org.apache.commons.compress.archivers.zip.ZipFile.<init>(ZipFile.java:174)
    at org.rauschig.jarchivelib.ZipFileArchiver.createArchiveInputStream(ZipFileArchiver.java:40)
    at org.rauschig.jarchivelib.CommonsArchiver.extract(CommonsArchiver.java:81)
    at xcuitest.installer.LocalXCTestInstaller.extractZipToApp(LocalXCTestInstaller.kt:164)
    at xcuitest.installer.LocalXCTestInstaller.runXCTest(LocalXCTestInstaller.kt:124)
    at xcuitest.installer.LocalXCTestInstaller.setup(LocalXCTestInstaller.kt:51)
    at ios.xctest.XCTestIOSDevice.restartXCTestRunnerService(XCTestIOSDevice.kt:39)
    at ios.xctest.XCTestIOSDevice.open(XCTestIOSDevice.kt:32)
    at ios.LocalIOSDevice.open(LocalIOSDevice.kt:24)
    at maestro.drivers.IOSDriver.open(IOSDriver.kt:75)
    at maestro.Maestro$Companion.ios(Maestro.kt:531)
    at maestro.cli.session.MaestroSessionManager.createMaestro(MaestroSessionManager.kt:198)
    at maestro.cli.session.MaestroSessionManager.access$createMaestro(MaestroSessionManager.kt:46)
    at maestro.cli.session.MaestroSessionManager$newSession$session$1.invoke(MaestroSessionManager.kt:78)
    at maestro.cli.session.MaestroSessionManager$newSession$session$1.invoke(MaestroSessionManager.kt:77)
    at maestro.cli.db.KeyValueStore.withExclusiveLock(KeyValueStore.kt:37)
    at maestro.cli.session.SessionStore.withExclusiveLock(SessionStore.kt:74)
    at maestro.cli.session.MaestroSessionManager.newSession(MaestroSessionManager.kt:77)
    at maestro.cli.session.MaestroSessionManager.newSession$default(MaestroSessionManager.kt:54)
    at maestro.cli.command.TestCommand.call(TestCommand.kt:124)
    at maestro.cli.command.TestCommand.call(TestCommand.kt:42)
    at picocli.CommandLine.executeUserObject(CommandLine.java:1933)
    at picocli.CommandLine.access$1200(CommandLine.java:145)
    at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2332)
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2326)
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2291)
    at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2159)
    at maestro.cli.DisableAnsiMixin$Companion.executionStrategy(DisableAnsiMixin.kt:22)
    at picocli.CommandLine.execute(CommandLine.java:2058)
    at maestro.cli.AppKt.main(App.kt:125)
Caused by: java.io.EOFException
    at org.apache.commons.compress.utils.IOUtils.readFully(IOUtils.java:231)
    at org.apache.commons.compress.archivers.zip.ZipFile.setDataOffset(ZipFile.java:1356)
    at org.apache.commons.compress.archivers.zip.ZipFile.resolveLocalFileHeaderData(ZipFile.java:1311)
    at org.apache.commons.compress.archivers.zip.ZipFile.<init>(ZipFile.java:378)
    ... 33 more

I've attached two shell scripts to reproduce this with the maestro examples.

prepare.sh:

#!/usr/bin/env bash

set -e

# Root simulator id is the id of the simulator that will be cloned
# Determined by: xcrun simctl list
ROOT_SIMULATOR_ID="B8C4930F-AC2E-4878-85F7-BEB413CE2A31"

xcrun simctl clone $ROOT_SIMULATOR_ID sim1 > simulator-ids.txt
xcrun simctl clone $ROOT_SIMULATOR_ID sim2 >> simulator-ids.txt
xcrun simctl clone $ROOT_SIMULATOR_ID sim3 >> simulator-ids.txt

# Start the simulator
open -a Simulator

for simulator in $(cat simulator-ids.txt); do
  xcrun simctl boot $simulator
done

# Download the samples
maestro download-samples

(cd samples && unzip sample.zip)

run-maestro.sh:

#!/usr/bin/env bash

for simulator in $(cat simulator-ids.txt); do
  xcrun simctl install $simulator samples/Wikipedia.app
  maestro --device=$simulator test samples/ios-advanced-flow.yaml &
done

echo "Waiting for all tests to finish"

wait

[kunalverma25]

Hi Team... Is there any update on this? Running tests in parallel on local and CI as well will drastically reduce the testing times..

[ArthurSav]

Hi @theoriginalgri We currently don't support parallel running via CLI.

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar problem, please file a new issue. Make sure to follow the template and provide all the information necessary to reproduce the issue. Thank you for helping keep us our issue tracker clean!