Closed mmcc007 closed 5 years ago
Created an experimental project to test signing using Fastlane's match (Match is not a requirement for managing certificates and provisioning profiles, just a convenient way to experiment for now): https://github.com/mmcc007/flutter_ios_build Build on travis: https://travis-ci.com/mmcc007/flutter_ios_build/builds/114622843
Will try to implement on Device Farm (manually via test_spec.yml). (Secure env variables can ultimately be passed via sylph -- securely? -- when automated. Though these variables will need to be declared 'locally' -- or securely on travis, etc... -- before starting a sylph run)
AFAIK there is no way to get around needing to register the Device Farm device with Apple Developer Account (barring access to Device Farm's Apple Developer Account, since there does not seem to be a secure way to do that via the API that would fit this use case. BTW: currently using public, not private devices). Looks like: https://docs.fastlane.tools/actions/register_devices/ may have potential for automating registration of devices. So, unless there are alternatives on Device Farm, will probably be looking into auto-registering using register_devices Fastlane module.
BTW: if using some parts of Fastlane makes sense, Fastlane and a Fastfile, etc.., can be added automatically during a Sylph run by adding to Appium bundle and test_spec.yml without requiring to make changes to the original app project.
I think I can output a testable app, packaged and signed as a .ipa.
If packaging the testable app as an .ipa works, we can upload it as a usual app (not as an auxiliary) and see if the flutter tools can find it. As I recall, I think I tried this and ran into some problems running flutter driver
. If I can assume the testable app gets auto-resigned with Device Farm's enterprise cert, I will try running an integration test (again) and see what happens.
I updated https://github.com/mmcc007/flutter_ios_build to publish on GitHub the .app (Runner.app) which was signed using my developer certificate: https://github.com/mmcc007/flutter_ios_build/releases This app of course will only run on my registered device(s) (I was able to get it started, using ios-deploy, but it shows a blank screen... will debug some more).
Even if I get Runner.app running on my device and then bundle it as a .ipa
and then re-signed with Device Farm's enterprise certificate, it seems likely that a test will not run on the app on Device Farm. This is because flutter tools
is expecting the app to be started in test mode. If there is a way to start an already installed app with arguments, I am currently not aware of it.
The most likely path forward at the moment is to explore the use of the register_devices
Fastlane module. This would involve, while on a Device Farm macOS, downloading the developer's certificate and the app's provisioning profile (similar to flutter_ios_build), registering the currently attached device and running with the standard command:
flutter driver test_driver/main.dart
Other paths forward do not seem that straightforward:
Extend the Device Farm API to allow starting an app with arguments.
This would involve uploading the .ipa (or possibly .app) and related start-up arguments.
flutter tools
would then need to be modified to somehow find the observatory and connect directly. Finding the observatory would involve reviewing the history of the logs from the time the app was started.
Extend the Device Farm API to allow downloading the re-signed app.
The app is uploaded, resigned with the enterprise cert. Then, while on the macOS an API call can be made to download the resigned app, install it on the device with the start-up args and run the test.
The API call could be restricted to Device Farm macOS's as a security precaution if necessary. flutter tools
would need to be modified to install the app without attempting to rebuild (there is a new '--no-build' option, but may not be sufficient in current form).
Do a design review of both Device Farm API and flutter tools
.
Establish the requirements and make the necessary changes to both Device Farm API and flutter tools
.
Other paths I'm not aware of?
For now I will continue down the register_devices
path which should not require any changes to the Device Farm API or flutter tools
. This path assumes that there are no unexpected restrictions on the macOS and the currently attached device (I'm assuming a macOS environment similar to the one provided by Travis-CI and standard access to the attached device -- no port restrictions, etc...).
Was able to download developer signing cert and provisioning profile using fastlane's match on a device farm mac, but am having problems managing the keychain (though keychain management is working locally and on travis).
I also did a quick test of fastlane's support for auto-registering devices and it seems to be working (on a local mac). I expect this should work on the device farm mac using the attached iOS device.
So it looks like getting flutter tests to run on device farm is pretty close (at least this way).
If I can resolve the keychain issue on device farm mac, and some similar minor blockers related to logging in to apple developer account, I expect to be able to run
flutter driver test_driver/main.dart
which should install the .app on the device and run the test (barring any possible currently unknown device farm mac issues).
BTW: credentials are auto-extracted from the local environment and reproduced in the device farm mac (similarly to travis).
The signing certificate and provisioning profiles are installed correctly. The problem is adding the new keychain to the keychain search path so that
security find-identity -p codesigning -v
can find it. flutter tools
expects to find the signing cert in the keychain path.
The command I am using to add the keychain to the keychain path is:
security list-keychains -s `security list-keychains | xargs` ~/Library/Keychains/fastlane_tmp_keychain-db
This works locally, but not on a device farm mac (may be a permissions problem that may require the device farm team to modify the device farm mac permissions for keychains).
The following log shows the current problem with keychain (logging into apple developer account was not tested in this run). Any suggestions would be appreciated.
According to:
5- Trust your newly created Development Certificate on your iOS device
via Settings > General > Device Management > [your new certificate] > Trust
... it looks like might have to put the kibosh on the 'run test like a developer' approach.
This probably bring us back to the drawing board.
I've since learned a little more about the device farm mac env. According to:
... the enterprise-resigned .ipa is available in on the mac.
DEVICEFARM_APP_PATH=/tmp/scratchtnt_vx.scratch/share-BbbgaB.scratch/app-mPEIn2.ipa
So if I can package a testable .app as a .ipa, I can retrieve the enterprise-resigned .ipa, unpackage it to a testable .app, and install it with ios-deploy. Then
flutter drive -no-build test_driver/main.dart
should start the .app for testing (with the start-for-testing args) and run the test.
I think this is worth trying.
Edit: So far, I'm not able to upload a testable .app packaged as a .ipa. When attempting to archive I get:
========================================================================
ERROR: Flutter archive builds must be run in Release mode.
To correct, ensure FLUTTER_BUILD_MODE is set to release or run:
flutter build ios --release
then re-run Archive from Xcode.
========================================================================
Creating a testable .ipa is currently disabled in flutter tools
. I believe this is because of issue on App Store with shipping an interpreter. There may be a way around this for device farm purposes. But for now can only use the .app.
Even if I were able to upload a testable .app via device farm api, it looks like resigning uses the resigners related provisioning profile (which is private). ios-deploy, which flutter tools
uses to install the testable .app, looks for this provisioning profile during validation (see original comment above)
One possible way around this is to package the testable .app in the Appium bundle and for device farm to provide a private call from the device farm mac to resign and install the testable .app onto the device (no change to the device farm api). Then
flutter drive -no-build test_driver/main.dart
should have everything it needs to run the test.
This is a blocker on this path.
At this point the paths described in comment above have been examined except:
3. Do a design review of both Device Farm API and flutter tools .
Establish the requirements and make the necessary changes to both Device Farm API and flutter tools.
4. Other paths I'm not aware of?
And have each resulted in a potential blocker (as described), with no discovered workarounds so far.
The most viable path at the moment is to add a feature to device farm to take the testable .app (uploaded via Appium bundle), resign it, and install it on the device (no need to start it). ios-deploy seems like the most likely way to install a .app. This should be a reasonably straightforward feature to implement. flutter tools
should then be able to start the .app and run a test using
flutter drive -no-build test_driver/main.dart
The new feature would callable from the device farm mac (exclusively).
Since activity on the mac is already authenticated, this probably doesn't need credentials.
Fixed a problem with:
flutter drive -no-build
in flutter tools
that showed-up during local testing on an attached device. This means running tests on iOS devices on device farm will no longer require a custom script.
I created https://github.com/flutter/flutter/issues/34148 with the flutter team and may submit a pull request.
Added building and publishing of non-testable .ipa, signed using my Apple Developer Account, to https://github.com/mmcc007/flutter_ios_build. This can be used with device farm api (though is not used in testing). It is what ultimately get's delivered to the app store for beta testing and release.
The https://github.com/mmcc007/flutter_ios_build project is a demo of one way a flutter build can be automated. It can be cloned and used with another developer account locally or on travis by setting-up the secure envs to produce the same artifacts with different signatures (expects a fastlane match repo).
Modified the script here to download this .ipa, together with the other artifacts, and run a test on an attached iOS device. The test runs if the matching developer accounts's cert and prov profile are installed locally. For example, the test will run on my local setup, with these artifacts, because I have the matching cert and prov profile for the testable .app installed locally.
BTW: the project and script are only intended as a convenience for experimentation with device farm to provide support for developing a solution for dealing with the testable .app.
Per the path I am currently pursuing, an installation service implemented by device farm would be called from the device farm mac. The testable .app would be sent to the installation service, re-signed, installed and started on the device (using ios-deploy). The installation service would return the observatory string found in the system log. A modification to flutter tools
would then accept the observatory string, forward the port and run the test. No change to the public-facing API.
Alternatively, the installation service could be exposed by extending the public facing API and optionally skipping the upload of the non-testable .ipa completely.
Hi Maurice, Thanks for the work on this. Your progress using register_devices from fastlane is a fascinating approach to solving the iOS resigning problem that AWS Device Farm attempts to solve using its resigning system. It's really interesting, and has the potential to help us reshape how we accept and deploy more general customer apps (beyond Flutter) in the future.
For now though, the path of least resistance seems, to my naive eyes, to be to add a parameter to flutter tools
to βuse a preinstalled driverβ (such as flutter --verbose drive --no-build --use-preinstalled-driver test_driver/main.dart
) so that users can just pass the Flutter driver into Device Farm, have it installed as a usual app (either the main test app or the auxillary app), and run their Flutter tests as usual. This is the path of least friction for a Device Farm integration from my perspective.
Adding an on-the-fly resigning option in Device Farm such that, from the macOS custom environment test spec file, you could call a command resign_my_app AppName.ipa/app/etc
and retrieve a signed app and observatory string would be too much of a one-off feature for us to consider. Could you elaborate on the "observatory string" as a concept? I'm not familiar with this an an output of codesigning.
Thanks!
@aristeia sure, it's interesting stuff, thnx!
Registering devices using developer account I think there may be a blocker on the 'run test like a developer' approach (which uses register_devices). The developer cert has to be trusted manually on the device. I don't know of any workaround for this. But if there is, it could be a useful feature in the future. Edit: maybe you can use your remote screen controller to acknowledge the cert as trusted.
Packaging a testable app for delivery to device farm
The current path I am following assumes that there is no .ipa used in testing. This is because flutter tools
runs a script during archiving that checks if a .ipa is being created for a testable .app. Archiving fails with:
========================================================================
ERROR: Flutter archive builds must be run in Release mode.
To correct, ensure FLUTTER_BUILD_MODE is set to release or run:
flutter build ios --release
then re-run Archive from Xcode.
========================================================================
I have not yet looked into this restriction to see if there is a workaround. I believe it is because of AppStore rules on embedding an interpreter.
So given that we cannot use a testable .ipa, we are using a testable .app.
flutter observatory The observatory is basically a flutter debug service that runs when you start a testable .app on a device. It attaches itself to a port on the device and declares the port number in the device log. It receives commands to move thru the app and send back results.
In theory if you know the port number you can connect to this service from the host using any program in any language. Flutter provides a test package written in dart that supports sending and receiving these commands. Developers can then write integration tests to test the behavior of the app (see example).
re-signing a testable .app Given that the testable .app is required to be running on the device, it has to be re-signed for a device farm device with the device farm enterprise cert/prov.
For the sake of experimentation, I looked around for examples of how to automate the re-signing of a .app (instead of a .ipa).
Found a script on fastlane that has a feature to re-sign a .app. Fastlane doesn't expose this feature (see https://github.com/fastlane/fastlane/issues/14023) so am using the re-signing script directly. It's not clear why they don't expose this feature. Could be because there may be a problem with resigning a .app.
Nevertheless, I updated the script to break-out commands to download, re-sign, and test a testable .app.
Don't have another developer (or enterprise) cert and prov to test with. There is a problem with the resigning when I try to re-sign the testable .app with my own cert/prov (see output below), but at least it is some way down the path of re-signing a testable .app. @aristeia, this may be useful in experimenting with re-signing a .app with enterprise cert/prov.
Here is the output from the following commands:
./script/ios_device_test.sh --download untagged-5ee732e53620ed9e3e25
./script/ios_device_test.sh --resign "iPhone Distribution: Maurice McCabe (ET2VMHJPVM)" --provisioning /Users/jenkins/Library/MobileDevice/Provisioning\ Profiles/408fa202-3212-469d-916c-c7f2ae4d083a.mobileprovision
./script/ios_device_test.sh --test
starting a testable .app Assuming the testable .app re-signing is working, the .app then has to be installed and started on the device.
This is a straightforward process in a developer environment. However, in the device farm mac environment we are assuming there is no access to the device farm's enterprise cert and provision profile.
Therefore the app has to be installed and started by device farm and the observatory port made available to the device farm mac.
Startup can be done using a modified flutter tools
or can be done with a custom script (see prototype startup for an example of how to startup a testable .app and find the observatory port). The startup process is unlikely to change for the long foreseeable future so should be very stable.
The observatory port can be made available on the device farm mac as the environment variable (for example, FLUTTER_OBSERVATORY_PORT).
flutter tools
can then attach to the .app and run the test or can be run directly using something like:
host_port=1024
iproxy $host_port $FLUTTER_OBSERVATORY_PORT
export VM_SERVICE_URL=http://127.0.0.1:$host_port
dart test_driver/main_test.dart
overall process
This requires a modification to the device farm api to accept a .app instead of a .ipa and the addition of the observatory port as an environment variable on the device farm mac.
Sylph can be modified to automate this process.
I looked into the restriction on creating a testable .ipa and found a workaround! See details below.
This means that the overall process can be further streamlined as follows:
Tests are then started in normal manner (for example, as currently works in Sylph for android)
This process requires no change to the device farm api (other than optional enhancement) and can be automated by Sylph.
minimum impact
workaround details I added the fix to the build on travis https://travis-ci.com/mmcc007/flutter_ios_build/builds/115493500 and published the testable .ipa here.
The fix involved creating a fork of flutter and making some changes: https://github.com/mmcc007/flutter This forked version of flutter is used in the travis build. Some version of this fix should probably be merged into the flutter project.
test of workaround I tested the workaround by downloading the published .ipa and tests and running them locally on an attached device. To carry out the same test, fork https://github.com/mmcc007/flutter_ios_build , publish the same artifacts via travis and run the same tests locally with an attached device.
The download of the artifacts and running of the test are automated with this script: https://github.com/mmcc007/flutter_ios_build/blob/master/script/ios_device_test.sh Test procedure:
./script/ios_device_test.sh --download <latest tag>
./script/ios_device_test.sh --install_ipa
./script/ios_device_test.sh --test
Test should succeed. No re-signing is required if using artifacts built on your fork with your developer cert/prov.
test logs
Notes about running flutter in device farm:
flutter testing The testing philosophy is somewhat different from many approaches out there. Counter-intuitively, the deliverable .ipa artifact cannot be tested, using traditional or any other method. This is because there are no native widgets in Flutter (most test tools expect access to native widgets). Plus there is no external access to the flutter widgets because everything unnecessary is stripped from the .ipa during the build in the interests of size and performance, and possibly security.
The solution is to build a testable .ipa that does have access to the flutter widgets using commands sent via a port (the observatory). Tests can then be written, in a somewhat traditional way, that results in generating these commands. The tests run on the a host and talk to the testable .ipa via a local port that is forwarded to the observatory port. Incidentally, the same test will run on an iOS or android device (a nice plus).
The closest counterpart I can think of to a testable .ipa is perhaps Selenium WebDriver with the mobile extensions, except that the WebDriver is contained in the testable .ipa. Plus there is no need for Appium-like wrappers around vendor provided automation frameworks(XCUITest, UIAutomation, etc...) since you are already talking to the 'native' flutter widgets.
flutter driver Flutter Driver is a tool that can take care of the build, connect to the running app and run the test. It is a tool that can run in multiple steps, some of which can be controlled by flags.
When running a test, Flutter Driver normally takes care of everything. Basically, it will run the build of the testable .ipa (really a .app) including signing using local developer (or enterprise, etc...) account, install the testable .app on the iOS device, start the app, which opens a random port on the device to receive commands (the observatory), finds the port, and finally forwards a port on the host to the observatory. Then it will start the test on the host and tell the test which local port to use.
Note: the test is a standalone dart program that simply talks to a remote service (the observatory). In theory, once it knowns how to access the remote service, the test could be running anywhere on any host (macOS, linux, windows), even in any language. In practice, for iOS, it is normally running on a mac with a local device attached. Further, the test has no dependency on flutter or the device type (iOS, android, etc...) or the app it is talking to (other than the wire protocol, the commands).
device farm integration The challenge when integrating with Device Farm, other than creating the testable .ipa, is to deal with the re-signing, installing and starting on a 'random' device, and establishing the observatory port number. This involves breaking up the steps that Flutter Driver supports between Device Farms in-house, and the Device Farm mac. Hence the experiments.
Note: this testable .ipa is not intended to ever be delivered. It is intended exclusively for testing only. Which is why flutter tools disables creating a testable .ipa and only builds a testable .app (hence the fix to workaround this in my experiments. BTW: it may be possible to apply this fix on the fly, locally, without having to change flutter, which would probably be preferable).
@aristeia Looks like there is a problem with installing the testable .ipa on a Device Farm device. The device farm test harness is trying to take a screenshot and erroring-out before starting the run on the mac. This means I cannot currently do test runs on the mac using the testable .ipa.
Using testable .ipa: link to job Using a non-testable .ipa does not have this problem: link to job
@aristeia I streamlined the testing of the testable .ipa artifact (from above). Instead of cloning the project and building the artifacts on travis etc..., it should now be possible to download and resign the testable .ipa artifact using your local certificate and provisioning profile (developer, enterprise, ad-hoc etc..) and run the test.
This can perhaps facilitate further in-house testing at Device Farm. In particular, experimenting with unbundling the testable .ipa, installing the testable .app on the device, and getting the observatory port (per the overall process described above).
Flutter Driver could be modified to install and start the testable .app and return the observatory port with something like:
flutter drive --no-build --startup-only test_driver/main.dart
This would be run by Device Farm and the returned observatory port exported as an env var to the mac.
Then to run a test on the mac, Flutter Driver could be modified to use the observatory to run a test with something like:
flutter drive --observatory_port=$FLUTTER_OBSERVATORY_PORT test_driver/main_test.dart
simplified testing of testable .ipa Download the updated script from: https://github.com/mmcc007/flutter_ios_build/blob/master/script/ios_device_test.sh and the dependent script: https://github.com/mmcc007/flutter_ios_build/blob/master/script/resign.sh The commands are as follows:
/script/ios_device_test.sh --download untagged-8ce77308e0d6a2165adb
./script/ios_device_test.sh --resign ipa "iPhone Developer: Maurice McCabe (5UG6C3962Q)" --provisioning /Users/jenkins/Library/MobileDevice/Provisioning\ Profiles/408fa202-3212-469d-916c-c7f2ae4d083a.mobileprovision
(after re-signing, it unbundles the re-signed testable .ipa and places is the directory expected by Flutter Driver)
(should be able to replace with enterprise cert and provisioning profile)
./script/ios_device_test.sh --test
(Flutter Driver will install the testable .app on the device, get the observatory port, forward the port and call the test)
test logs
Added a feature to test if uploaded .ipa is a Flutter testable .ipa. This can be used to start the process for handling Flutter tests in Device Farms (per overall process above). See updated: https://github.com/mmcc007/flutter_ios_build/blob/master/script/ios_device_test.sh
Usage:
$ script/ios_device_test.sh --detect_testable_ipa
Flutter testable .ipa found
$
@aristeia Looks like there is a problem with installing the testable .ipa on a Device Farm device. The device farm test harness is trying to take a screenshot and erroring-out before starting the run on the mac. This means I cannot currently do test runs on the mac using the testable .ipa.
Using testable .ipa: link to job Using a non-testable .ipa does not have this problem: link to job
Hi Maurice, This "Screenshot" error detected doesn't have to do with your IPA choice at all, but rather has to do with the device itself. Device farm simply detected that there was an error taking screenshots on that device, which usually indicates that the device needs to be re-trusted to the harness or factory reset. If you give it another try, it could work.
Thanks, Jon
@aristeia It worked on a re-try (this was the first time I had tried with the testable .ipa).
Surprisingly ios-deploy of the unpacked $DEVICEFARM_APP_PATH (testable .ipa) get's passed the verification stage and installs. This is good news, as it may make it simpler to run a flutter test.
However, I'm running into several configuration and permissions related issues.
In general, the configuration of the Device Farm mac is not standard for development and building (especially when compared to a mac on Travis-CI or a local mac):
I cannot upgrade ios-deploy using the installed brew due to permissions.
To get around this I installed a local brew.
ios-deploy is reporting an error .
[100%] Installed package build/ios/iphoneos/Runner.app
------ Debug phase ------
Starting debug of 0eaf7884daeb1fbaa0d711b212d01c461cc88124 (D22AP, D22AP, uknownos, unkarch) a.k.a. '9905944756' connected through USB...
[ 0%] Looking up developer disk image
[ 95%] Developer disk image mounted successfully
2019-06-18 01:05:20.852 ios-deploy[1787:6925] [ !! ] Unable to locate DeviceSupport directory with suffix 'Symbols'. This probably means you don't have Xcode installed, you will need to launch the app manually and logging output will not be shown!
This error may be due to missing entries in /Applications/Xcode_10.app/Contents/Developer/Platforms/iPhoneOS.platform/DeviceSupport. There may be a workaround for this but cannot be applied due to permissions on the directory.
drwxrwxr-x 22 root wheel 748 Oct 19 2018 .
drwxr-xr-x 10 root wheel 340 Oct 19 2018 ..
drwxrwxr-x 4 root wheel 136 Oct 19 2018 10.0
drwxrwxr-x 4 root wheel 136 Oct 19 2018 10.1
drwxrwxr-x 4 root wheel 136 Oct 19 2018 10.2
drwxrwxr-x 4 root wheel 136 Oct 19 2018 10.3
drwxrwxr-x 4 root wheel 136 Oct 19 2018 11.0
drwxrwxr-x 4 root wheel 136 Oct 19 2018 11.1
drwxrwxr-x 4 root wheel 136 Oct 19 2018 11.2
drwxrwxr-x 4 root wheel 136 Oct 19 2018 11.3
drwxrwxr-x 4 root wheel 136 Oct 19 2018 11.4
drwxrwxr-x 4 root wheel 136 Oct 19 2018 12.0
drwxrwxr-x 4 root wheel 136 Oct 19 2018 12.1 (16B91)
drwxrwxr-x 4 root wheel 136 Oct 19 2018 8.0
drwxrwxr-x 4 root wheel 136 Oct 19 2018 8.1
drwxrwxr-x 4 root wheel 136 Oct 19 2018 8.2
drwxrwxr-x 4 root wheel 136 Oct 19 2018 8.3
drwxrwxr-x 4 root wheel 136 Oct 19 2018 8.4
drwxrwxr-x 4 root wheel 136 Oct 19 2018 9.0
drwxrwxr-x 4 root wheel 136 Oct 19 2018 9.1
drwxrwxr-x 4 root wheel 136 Oct 19 2018 9.2
drwxrwxr-x 4 root wheel 136 Oct 19 2018 9.3
You can see some of the issues I ran into at run If mac configurations can be resolved, it may be possible to run a flutter test without adding functionality to Device Farm (as described in overall process above)
Edit: I tried with a different device (11.4), with more verbose output from ios-deploy with same result. It's having a hard time handling the DeveloperDiskImage.dmg.
[100%] Installed package build/ios/iphoneos/Runner.app
Hardware Model: D22AP
Device Name: 9902852975
Model Name: D22AP
SDK Name: uknownos
Architecture Name: unkarch
------ Debug phase ------
Starting debug of 22698ea307b124fdbe1b790154455a8b6e8a2795 (D22AP, D22AP, uknownos, unkarch) a.k.a. '9902852975' connected through USB...
Device Class: iPhone
build: 15F79
version: 11.4
DeviceSupport directory '/Applications/Xcode_9_4.app/Contents/Developer/Platforms/iPhoneOS.platform/DeviceSupport/11.4 (15F79)/DeveloperDiskImage.dmg' was located. It took 0.00 seconds
Developer disk image: /Applications/Xcode_9_4.app/Contents/Developer/Platforms/iPhoneOS.platform/DeviceSupport/11.4 (15F79)/DeveloperDiskImage.dmg
[ 0%] Looking up developer disk image
[ 90%] Mounting developer disk image
[ 95%] Developer disk image already mounted
Device Class: iPhone
build: 15F79
version: 11.4
version: 11
2019-06-18 12:07:59.805 ios-deploy[1840:7157] [ !! ] Unable to locate DeviceSupport directory with suffix 'Symbols'. This probably means you don't have Xcode installed, you will need to launch the app manually and logging output will not be shown!
See related run. Any idea what this is?
@aristeia Great news! I fixed the problem with not finding the symbols by going thru the ios-deploy source and identifying a device-specific hack.
After that ios-deploy was able to start the testable .ipa! I was then able to run a flutter test as normal.
The first Flutter test to run on an iOS device on Device Farm looks like: Related run.
This confirms that Flutter integration tests can run on Device Farm for both iOS and Android devices. This can be integrated to Sylph using the Device Farm API and can be used in a variety of CI/CD scenarios.
There is much to add to make this ready for production. But the proof of concept is there.
This is really cool Maurice, nice job! I can't wait to get a look at this!
@aristeia Implemented the fix for iOS and added support for running Sylph in a CI environment by automatically managing the developer certificate and provisioning profile on the fly (see merges #9 and #10 for details).
The resulting CI build can be seem here: https://travis-ci.com/mmcc007/sylph/builds/116272945 This build published the resulting Device Farm artifacts here: https://github.com/mmcc007/sylph/releases/tag/0.0.8 (which includes the video)
Remains to do a Flutter integration test run which includes both iOS and android pools in the same run. Then start making improvements for production quality, like speed, usability, more support for developers, etc...
Please let me know if you have any questions, suggestions, etc...
Ran a Flutter integration test on both an iOS and Android pool of devices in a single Sylph run from a local mac and downloaded the respective artifacts for each device.
Also fired off the same Sylph run on travis to demonstrate how running Flutter integration tests performs on Device Farm: https://travis-ci.com/mmcc007/sylph/builds/116507297 Artifacts: https://github.com/mmcc007/sylph/releases/tag/0.0.9 Performance can be improved by running the test in parallel on iOS and Android pools. Further performance enhancements would require more direct support for Flutter on Device Farm (similar to how Appium, etc.. is supported) and perhaps general Device Farm performance improvements.
@aristeia I notice fairly frequent failures in what appears to be the test harness and related areas on Device Farm.
For example, had to run an identical sylph on an iOS and android device three times before getting a successful run: First run failed on the android device: first sylph run: android device Second run failed on ios device:
Running flutter drive --no-build test_driver/main.dart
No devices found.
Third sylph run succeeded.
Is this rate of failure expected? If more devices are added to each device pool, should the rate of failure be expected to increase? Or is there some way to manage these failures?
On a macOS with an attached iOS device on Device Farms (via web interface) with an uploaded debug .app.
ios-deploy is not validating:
ios-deploy --bundle build/ios/Debug-iphoneos/Runner.app --no-wifi --justlaunch --args '--enable-dart-profiling --start-paused --enable-checked-mode --verify-entry-points'
``` ios-deploy --bundle build/ios/Debug-iphoneos/Runner.app --no-wifi --justlaunch --args '--enable-dart-profiling --start-paused --enable-checked-mode --verify-entry-points' [....] Waiting for iOS device to be connected [....] Using bb66dc1943570628375e82055b7770b33883dc73 (D22AP, D22AP, uknownos, unkarch) a.k.a. '9905944754'. ------ Install phase ------ [ 0%] Found bb66dc1943570628375e82055b7770b33883dc73 (D22AP, D22AP, uknownos, unkarch) a.k.a. '9905944754' connected through USB, beginning install [ 5%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/META-INF/ to device [ 5%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/META-INF/com.apple.ZipMetadata.plist to device [ 5%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/_CodeSignature/ to device [ 6%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/_CodeSignature/CodeResources to device [ 6%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/AppFrameworkInfo.plist to device [ 6%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/AppIcon20x20@2x.png to device [ 7%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/AppIcon20x20@2x~ipad.png to device [ 7%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/AppIcon20x20@3x.png to device [ 7%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/AppIcon20x20~ipad.png to device [ 8%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/AppIcon29x29.png to device [ 8%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/AppIcon29x29@2x.png to device [ 8%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/AppIcon29x29@2x~ipad.png to device [ 9%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/AppIcon29x29@3x.png to device [ 9%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/AppIcon29x29~ipad.png to device [ 10%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/AppIcon40x40@2x.png to device [ 10%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/AppIcon40x40@2x~ipad.png to device [ 10%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/AppIcon40x40@3x.png to device [ 11%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/AppIcon40x40~ipad.png to device [ 11%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/AppIcon60x60@2x.png to device [ 11%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/AppIcon60x60@3x.png to device [ 12%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/AppIcon76x76@2x~ipad.png to device [ 12%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/AppIcon76x76~ipad.png to device [ 12%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/AppIcon83.5x83.5@2x~ipad.png to device [ 13%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Assets.car to device [ 13%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Base.lproj/ to device [ 13%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Base.lproj/LaunchScreen.storyboardc/ to device [ 14%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Base.lproj/LaunchScreen.storyboardc/01J-lp-oVM-view-Ze5-6b-2t3.nib to device [ 14%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Base.lproj/LaunchScreen.storyboardc/Info.plist to device [ 15%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Base.lproj/LaunchScreen.storyboardc/UIViewController-01J-lp-oVM.nib to device [ 15%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Base.lproj/Main.storyboardc/ to device [ 15%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Base.lproj/Main.storyboardc/BYZ-38-t0r-view-8bC-Xf-vdC.nib to device [ 16%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Base.lproj/Main.storyboardc/Info.plist to device [ 16%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Base.lproj/Main.storyboardc/UIViewController-BYZ-38-t0r.nib to device [ 16%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Debug.xcconfig to device [ 17%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/embedded.mobileprovision to device [ 17%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/ to device [ 17%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/App.framework/ to device [ 18%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/App.framework/_CodeSignature/ to device [ 18%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/App.framework/_CodeSignature/CodeResources to device [ 18%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/App.framework/App to device [ 19%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/App.framework/flutter_assets/ to device [ 19%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/App.framework/flutter_assets/AssetManifest.json to device [ 20%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/App.framework/flutter_assets/FontManifest.json to device [ 20%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/App.framework/flutter_assets/fonts/ to device [ 20%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/App.framework/flutter_assets/fonts/MaterialIcons-Regular.ttf to device [ 21%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/App.framework/flutter_assets/isolate_snapshot_data to device [ 22%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/App.framework/flutter_assets/kernel_blob.bin to device [ 27%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/App.framework/flutter_assets/LICENSE to device [ 28%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/App.framework/flutter_assets/packages/ to device [ 28%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/App.framework/flutter_assets/packages/cupertino_icons/ to device [ 29%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/App.framework/flutter_assets/packages/cupertino_icons/assets/ to device [ 29%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/App.framework/flutter_assets/packages/cupertino_icons/assets/CupertinoIcons.ttf to device [ 29%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/App.framework/flutter_assets/vm_snapshot_data to device [ 30%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/App.framework/Info.plist to device [ 30%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/Flutter.framework/ to device [ 30%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/Flutter.framework/_CodeSignature/ to device [ 31%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/Flutter.framework/_CodeSignature/CodeResources to device [ 31%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/Flutter.framework/Flutter to device [ 47%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/Flutter.framework/icudtl.dat to device [ 48%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/Flutter.framework/Info.plist to device [ 48%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Info.plist to device [ 49%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/PkgInfo to device [ 49%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Runner to device [ 52%] CreatingStagingDirectory [ 57%] ExtractingPackage [ 60%] InspectingPackage [ 60%] TakingInstallLock [ 65%] PreflightingApplication [ 65%] InstallingEmbeddedProfile [ 70%] VerifyingApplication 2019-06-02 12:49:43.347 ios-deploy[544:5128] [ !! ] Error 0xe8008015: A valid provisioning profile for this executable was not found. AMDeviceSecureInstallApplication(0, device, url, options, install_callback, 0) ```Currently experimenting with installing provisioning profile and developer certificate, etc...
@tvolkert @aristeia