Run on hardware iOS device

[LeoNatan]

There are several challenges we need to overcome before we can achieve this. Please know there there will be some limitations to what will be possible on the device, such as permission changing.

[simonracz]

Here is how we achieved this recently.

• We took the "Copy Detox Framework" run script from the demo-react-native example project.
• As I ran into code signing issues, I replaced it by manually embedding the Detox.framework into the project.
• I added a new script that recursively code signs each framework and dylib inside Detox.framework.
• I copied and embedded the XCTest.framework also in the project as Earlgrey was complaining about it.
• I set up the detoxServer and detoxSessionId in the launch arguments of the scheme.
• The device and laptop was on the same network of course. The laptop's IP was set up correctly in the appdelegate and in the detoxServer launch argument.
• I set up the same port and sessionId in the detox config (in package.json)
• Also added an "ios.none" config there.
• I ran "detox run-server" manually
• Then, I ran "detox test --configuration ios.none".

And now, about the problems. After setting these all up, the app was running, it connected to the server, the tests started to run. And each of them failed. :|

During investigating what could be wrong, it suddenly started to work. Without any code change. The same pattern repeated on one of my colleague's machine. (It failed the tests first, then it passed them half an hour later.)

[rotemmiz]

@simonracz did you run a react-native app? in debug mode ? Loading the bundle from the debug node server may take time, and detox may reach the default timeout and fail.

[simonracz]

If you mean the Mocha runner time out, then no, it wasn't that.

We tried both release and debug mode. It was the demo-react-native sample. No modifications were made to it, apart from the project file modifications mentioned above.

The first test line is "await expect(element(by.id('welcome'))).toBeVisible();". This failed with an Earlgrey assertion. It found the view with the id welcome, however it failed the visibility test.

I started to debug the runtime view hierarchy. For example, I stopped the process and called "po [GREYElementHierarchy hierarchyStringForAllUIWindows]".

For this view it showed the AX.id correctly, but the AX.frame was (0,0),(0,0). Sadly, I don't have saved logs.

After it started to work, the AX.frame was correct.

I wasn't able to reproduce it since then.

[simonracz]

I was able to reproduce this "Earlgrey fails the visibility test" issue.

In fact I spent half-a-day on chasing this. I was able to narrow it down to the following.

The matched view is rejected, because it's accessibilityFrame is zero. It's frame is correct. It is an RCTView, which inherits from UIView. From to the docs "The default value for this property is CGRectZero unless the receiver is a UIView object or is a subclass of UIView, in which case the value is the frame of the view."

So, this shouldn't happen unless someone manually touches the accessibilityFrame. I looked at the react-native source-code, they never touch the accessibilityFrame. Neither does Earlgrey (except in it's tests).

I restarted, cleared, rebuilt, killed everything. Nothing helped.

I don't think this is a Detox issue. I wrote it down only that in case you encounter it, let's share our findings.

[simonracz]

After spending more than a day on this bug I found the issue.

Since some iOS version (don't know yet exactly when, but 10.3.2 and 10.3.3 are affected) Apple introduced a bug/feature around accessibility.

The accessibilityFrame is not set, nothing isAccessible etc. unless the user is really using some accessibility features, e.g. voiceover.

Earlgrey heavily depends on accessibility. I don't know why haven't anyone else found this, opened a bug about it yet.

I'll open a bug report on Earlgrey and also open radar. I want to determine first the exact iOS version when this was introduced.

[LeoNatan]

@simonracz Interesting. XCUITest also uses accessibility for its operation. It’s worth investigating how the system allows that for those cases but doesn’t for ours. Most likely an env var.

[simonracz]

Made a radar about it. http://www.openradar.me/radar?id=5043342748418048

[LeoNatan]

@simonracz This is probably an intentional change. There is no need to calculate these values on a real device, running production software (I don't mean a release build). We need to find out how the system differentiates between running from Xcode and otherwise. I will take a look soon.

[LeoNatan]

I have found a way to force accessibility. I will push soon to Detox.

[saikrishna321]

@LeoNatan is it possible to run on real device now ? I couldn't find any docs on it

[pachun]

@LeoNatan This will be so helpful to have! Thanks for putting in the work. Can't wait to get my hands on it.

[LeoNatan]

Running on devices is not supported yet.

[LeoNatan]

@pachun Please do not unassign members of the team from issues. Leave the issue management to us. Thanks.

[pachun]

@LeoNatan I have no idea how that happened. Sincerest apology. 100% unintentional. All I meant to do was thank you.

[LeoNatan]

Cheers! We plan to support Detox on device soon.

[sonianin]

@LeoNatan, are there any updates on the iOS real devices support? We are really looking forward to this functionality...

[LeoNatan]

Nothing to announce yet. It’s in our backlog and we intend to investigate this eventually, but it’s not on the top of our priorities. There are some challenges that we haven’t solved yet.

[mike623]

Really looking forward of this too. My app need turn on the real device to check the camera.

[fdnhkj]

@LeoNatan Don't you think the following note in README should be adapted then?

Runs on Devices: Gain confidence to ship by testing your app on a device/simulator just like a real user.

IMO, it is a bit misleading. I would mention it is not yet supported on iOS.

[DanielMSchmidt]

@fdnhkj I think you are right, could you go ahead and make a PR?

[fdnhkj]

@DanielMSchmidt Sure I can!

Do you know if running on physical device on Android works? I have seen android.attached config in the docs but I didn't test it so I'm not sure.

I'll adapt the README accordingly.

[DanielMSchmidt]

AFAIK it should

Fidan Hakaj notifications@github.com schrieb am Mo., 8. Jan. 2018 um 15:49 Uhr:

@DanielMSchmidt https://github.com/danielmschmidt Sure I can!

Do you know if running on physical device on Android works? I have seen android.attached config in the docs https://github.com/wix/detox/blob/master/docs/Introduction.Android.md#4-add-android-configuration but I didn't test it so I'm not sure.

I'll adapt the README accordingly.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/wix/detox/issues/95#issuecomment-355986544, or mute the thread https://github.com/notifications/unsubscribe-auth/ABRm1t2L4xtCOM1DmKekI5Qa5isrXHd0ks5tIisBgaJpZM4MVY2O .

[LeoNatan]

I think I've solved the first hurdle (but a big one), which is setting environment variables when running on device. 🎉 Using a modified ios-deploy, I was able to embed environment variables.

Steps ahead: sign Detox dylib with the same signature as the user's app and send to device storage, then try to inject it using DYLD_INSERT_LIBRARIES environment variable.

[LeoNatan]

Could anyone please walk me through how signing is managed on test clouds?

[Karniej]

Any information when the testing on real android devices will be ready?

[swaminator]

Checking in to see if there's been any progress with this?

[LeoNatan]

No new progress. We've done a POC and it's technically feasible. An actual implementation is planned, but there is no timeline.

[xzilja]

@LeoNatan since it can run on a real device will it be possible to somehow integrate Detox tests into firebase test lab? Especially now with iOS support announced, could be very powerful.

[LeoNatan]

Unfortunately we have not yet implemented real device support.

[aknackiron]

I work for Bitbar (mobile devops cloud) and we tried out Detox in our cloud successfully. We even had a post about it (link below). We believe real device execution is faster and more robust to running mobile tests. We are interested in helping to implement the cloud solution for iOS real device support once you have something working.

https://bitbar.com/its-time-to-detox-your-mobile-tests-on-bitbar-cloud/

[rotemmiz]

Hey @aknackiron, This is exciting! Please contact me at rotemm@wix.com so we can continue discussing this offer seriously.

[DimiTech]

Guys, any news on this? We're having a difficult time testing React Native \w Appium and are looking for alternative test frameworks/SaaS offerings which we can use to test our iOS/Android apps on a real device farm. Thanks

[aknackiron]

We tried out WIX's solution and our cloud supports Detox on real Android and iOS devices. Android side is out of the box, iOS needs some extra setup from our side. Not to make this into a sales post, please get in touch with me directly at niko.cankar@bitbar.com. Our cloud supports also almost any other frameworks out there, not only Detox, Appium or native frameworks.

[EdwardDrapkin]

@LeoNatan since this appears to still be in TODO, can you release the POC so we (the community) can work on it and maybe PR it back upstream without having to reinvent your wheels?

[DanielMSchmidt]

Maybe on a similar topic just to share it upfront, we are trying to move detox into a position where is it more extensible. The goal is to allow you to solve this on your own, depending on the needs of your platform. I wrote an issue for that to track the progress: https://github.com/wix/Detox/issues/1091

[LeoNatan]

@DanielMSchmidt

While extensibility is very important, the magnitude of work required for this to work on a real device is beyond the scope of plugins. It involves touching all parts of detox and some JS extensions will not suffice. This is one of these cases which should come build-in into Detox. I am not sure extensibility like that is possible or feasible. I don't think an entirely new category of device will be able to be added as an extension.

@EdwardDrapkin

My POC was to test the current delivery method of Detox on a real device.

Right now, we use an environment variable DYLD_INSERT_LIBRARIES to inject the Detox library without requiring the user's project be changed or changing it ourselves. That's good for many reasons:

• Detox should not be uploaded to the app store, so a clear solution is always better
• A lot of our users are not familiar with Xcode, so making changes to the project is difficult
• Changing the IPA on behalf of the user requires resigning of the package, which we cannot do
• The same binary can be checked with newer versions of Detox without requiring rebuild of the binary

Doing this on the simulator is relatively easy, as Xcode provides the simctl utility, which we can use to install and run binaries on the device.

For a real device, no such official utility exists, despite there being support for it (Xcode does it, obviously, as well as iTunes). So I tested a third-party utility called ios-deploy, which allows for some similar capabilities. One hurdle I found right away was providing an environment variable, which is not normally possible to do on a device. But again, Xcode is able to do it, and ios-deploy has an option to launch in a debugger mode (which means it's a controlled launch, just like Xcode does). So I was able to modify the source of ios-deploy to add the environment variable and was able to launch an unmodified app and load our framework.

This is a big step, but only the first one. There are many features that would break. applesimtutils obviously can't run on a real device, so permission dialogs cannot be handled. Going to background and foreground will probably not be possible. Since my POC, some features have been added which make assumptions that the tested process is running on the same machine as the tester and just go to the disk and take files; that will clearly not work, so all those systems need to be rewritten to use file access APIs which we don't have in Detox yet. No one has tested how Earl Grey works on arm yet with real UIKit (simulator and device UIKit have internal differences). That's a lot of hurdles to overcome before we can say we support real devices.

But, with ios-deploy, it is probably possible to achieve some sort of running POC at some capacity with relatively not much work. But from this point to a point where we can feel confident to write "we support real iOS devices" is far.

[timoisalive]

@aknackiron @LeoNatan

Out of curiosity, how is it possible Bitbar (apparently) has Detox running on iOS devices in their device farm, but Detox alone is not (even close, apparently) supporting iOS devices?

Were/are you working together on this?

[LeoNatan]

I don't see anywhere that it runs on a real device? Even if so, the project is open source and they could have done the necessary to make it work. As I have outlined, this is not that hard.

[aknackiron]

Using Leo's setup we're able to host customer dedicated iOS devices for Detox testing.

[vincenzovitale]

@aknackiron, could you share that back to the community?

[AlexK777]

I highly doubt @aknackiron will share his implementation if bitbar uses it, as that is a paid service.

Maybe I am wrong :smile:. But even if that is the case I would be personally wary to use that implementation until it was "blessed" by some of the main contributors here. It is exciting to see the ball move forward nevertheless.

[LeoNatan]

My apron is ready for blessing.

[aknackiron]

@aknackiron, could you share that back to the community?

As mentioned earlier, we have not added anything to Leo's implementation. I don't know if these steps have been shared elsewhere than in Slack.

[META-DREAMER]

Any update on this? Running on real devices is crucial for us to setup automated performance testing. If this can be working reliably on Bitbar our team would love to use it. @aknackiron Do you have a guideline for setting all this up? What are the limitations of testing on device using Bitbar?

[aknackiron]

@hammadj Detox's real iOS device support is not ready for prime time and is currently not on Detox roadmap. We have not developed anything on top of Detox.

Know that at current state (last we checked), you don't get screenshots and you can't relaunch app with Detox on real iOS devices.

If you still want to try out, we can do a project with you, but you'll have to get in touch with our sales... :-/

[LeoNatan]

Guys, please keep the discussion civil. If you'd like to see device support in the open source project, you are welcome to contribute; we will guide you. But please do not include rude comments.

[AlexK777]

@LeoNatan fair. But the unwillingness to share a repro (that many have asked for in this thread) and instead directing you to contact a sales department does not feel like "contributing to the open source project". 🤷‍♂️

[LeoNatan]

Detox is licensed under the MIT license. This is a permissive license. As long as there is attribution, anyone is free to clone the project privately and extend it as they see fit; if they decide not to share their extensions, that is part of the contract too. While I understand your frustration, your comment was out of line and unacceptable in this community.

[muraliseveneleven]

Hey Guys ..any direction would be helpful. We have a huge framework written in Java/Appium and one our goal is to execute tests on real devices. We are trying to transition everything to detox( as our new app is react native and all the backend is NodeJs). we successfully ran few tests and works well with our complete simulator setup and found it little more powerful than appium at various instances , Now we wanted to run this on real device( test devices which we have ) as cloud is out of the financial bucket to spend for this activity. Do we start and try what @simonracz had suggested or any other workable workarounds for this to make it happen today. Thanks