search

openstf / stf

Control and manage Android devices from your browser.
https://openstf.io
Other
13.16k stars 2.7k forks source link

iOS support #64

Open kwv opened 8 years ago

kwv commented 8 years ago

Awesome, inspiring stuff.

TLDR; is iOS support on your roadmap?

I've been tinkering around with a solution that works on iOS devices (or simulators). The approach streams snapshots over websockets to a canvas and forwards user-generated mouse/keyboard events to Appium which replays them against a device. For a stand alone solution, with a single device, it works.

next up on the roadmap:

I understand its hard to comment on a solution you haven't seen. Before I get too much further down this proof-of-concept, I wanted to solicit feedback Are there any plans to support iOS in STF? Have you already been down this path and abandoned it?

proof of concept approach

image capture

To capture images, I've evaluated:

Testing against a physical device is desirable however without jailbreaking the device there are some limitations (lockscreen, home button).

The iOS simulator was able to produce highest framerates, and the closest experience to what STF provides for Android, but there is a nagging feeling it's not going to scale or meet the requirements (e.g. managed in MDM, recgnoize an internal cert chain)

Shortcomings

sorccu commented 7 years ago

My understanding is that for touch events, Remote simply adds code to the app you build. I'm not sure if it can send events to any app / system.

nhanvpt102 commented 7 years ago

We need to send touch events from Mac OS to USB connected iOS devices. "http://stackoverflow.com/questions/35339953/send-touch-events-from-mac-osx-application-to-usb-connected-iphone" confirmed that they implemented this feature in their VNC Server but can't share the source code :)

nhanvpt102 commented 7 years ago

On iOS6 & 7, can use IOHIDEvents (IOKit.Framework) to send touches/swipes/buttons from MAC OS to iOS devices. Not sure if IOHIDEvents (IOKit.Framework) is still supported on iOS 10.

Ashraf-Ali-aa commented 7 years ago

Could we not use appium/WebDriverAgent to send the touch events commands to the device since it was designed to run automation tests

sorccu commented 7 years ago

No, it's super, super slow. And unreliable. Can take 5-15s for an event to register.

On Sun, 20 Nov 2016 at 1:58 Ashraf Ali notifications@github.com wrote:

Could we not use appium/WebDriverAgent to send the touch events commands to the device since it was designed to run automation tests

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/openstf/stf/issues/64#issuecomment-261725275, or mute the thread https://github.com/notifications/unsubscribe-auth/AAB-_cUX0EvPxfYueY8qY9IySPOqVx8sks5q_yqhgaJpZM4FyyUk .

azohra commented 7 years ago

@sorccu I am not sure if that is true these days, especially with the release of this library from Facebook: https://github.com/facebook/WebDriverAgent

sorccu commented 7 years ago

Facebook isn't omnipotent. We've been using WDA for touch events internally and it's the worst. Latency of touch and key events is easily several seconds, sometimes even tens of seconds. To be fair, I believe most of the issues are due to XCUITest rather WDA itself.

It's also extremely unreliable and can often go into a weird state that you can only recover from by rebooting the device.

In short: it works, but is a nightmare to work with. We will never integrate it with STF in its current state. If the situation improves we'll think again.

On Wed, Jan 18, 2017 at 6:14 Justin Watts notifications@github.com wrote:

@sorccu https://github.com/sorccu I am not sure if that is true these days, especially with the release of this library from Facebook: https://github.com/facebook/WebDriverAgent

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/openstf/stf/issues/64#issuecomment-273301639, or mute the thread https://github.com/notifications/unsubscribe-auth/AAB-_frCNEyp0a5oT4XQGHhuXXbQepERks5rTS9OgaJpZM4FyyUk .

azohra commented 7 years ago

We also use it internally, but have found it to be quite snappy and stable. Are you pushing events over Wifi or USB?

sorccu commented 7 years ago

USB. If you'd like to help, feel free to make a sample app to demonstrate the responsiveness. Based on my personal experience with WDA I really don't think it's possible to make it work well with real time control, but would be happy to be proven wrong.

On Wed, Jan 18, 2017 at 23:41 Justin Watts notifications@github.com wrote:

We also use it internally, but have found it to be quite snappy and stable. Are you pushing events over Wifi or USB?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openstf/stf/issues/64#issuecomment-273492592, or mute the thread https://github.com/notifications/unsubscribe-auth/AAB-_Zw2a3JOGxN6Cxj22l7UJAT5o5Blks5rTiSqgaJpZM4FyyUk .

crashbell commented 7 years ago

I suffered the same problem. Can't wait to see that WDA was improved. @tk8817 Please notice that it's about the touch and key. For the rest, WDA works well to me.

jayzsolt commented 7 years ago

Hi. You're doing great job guys! Now for iOS, besides screen capturing there are other problems to solve, like app deployment (install/uninstall) on the device, right? Do anyone of you guys have a solution for that?

With Android the OpenSTF is so convenient with the instant drag and drop feature, how would this look like in case of iOS? Since xcode isn't really an option for device farms, I've tried libraries like libimobiledevice, fastlane, cfgutil, SSH+ipainstaller etc, with partial success only. In the other hand, Cydia Impactor's drag-n-drop interface is quite comfy, takes care of resigning and all, it cannot be automatized for larger-scale usage. Do any of you guys have experience on this? I'm happy to contribute in this btw.

Ashraf-Ali-aa commented 7 years ago

@apprayadmin check out https://github.com/phonegap/ios-sim and https://github.com/phonegap/ios-deploy

Ashraf-Ali-aa commented 7 years ago

this would be useful for ios minicap https://github.com/andrealufino/Luminous

martinwoit commented 7 years ago

It looks like someone has figured out how to do it the STF way with ios devices ... https://kobiton.com/

sorccu commented 7 years ago

Yeah, there are many paid and/or closed source offerings out there.

On Tue, Jan 31, 2017 at 5:01 PM, Martin Woit notifications@github.com wrote:

It looks like someone has figured out how to do it the STF way with ios devices ... https://kobiton.com/

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openstf/stf/issues/64#issuecomment-276298403, or mute the thread https://github.com/notifications/unsubscribe-auth/AAB-_Y_EcwlFsMwv_HA_ogxlbXC5ErWqks5rXupjgaJpZM4FyyUk .

sorccu commented 7 years ago

That's great! Do you think you'd be able to share your example's source code?

sorccu commented 7 years ago

I've also been looking into reading data from the USB endpoint iOSScreenCapture opens. I've been able to capture some raw data from it, but haven't had time to figure out the format yet. I suspect it's deceptively simple, though. Do you know what the output format is?

The only downside of doing the capture that way is that the device menu bar switches to a static view when the endpoint is open. For example, time is always "9:41" and other icons are hidden.

saikrishna321 commented 7 years ago

@sorccu any updates on this ?

sorccu commented 7 years ago

No. This isn't a paid product, please try to learn to respect people's free time. On Tue, Mar 28, 2017 at 18:23 Sai Krishna notifications@github.com wrote:

@sorccu https://github.com/sorccu any updates on this ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openstf/stf/issues/64#issuecomment-289713329, or mute the thread https://github.com/notifications/unsubscribe-auth/AAB-_VV-IhJUC20TS8NPWNKNwsmij_vuks5rqNGGgaJpZM4FyyUk .

ch4053n61n3 commented 7 years ago

Sorry for these douchebags, we love to use openstf at my workplace and we really respect your work.

https://www.engadget.com/2014/04/14/why-9-41-am-is-the-always-the-time-displayed-on-iphones-and-ipad/

qmfrederik commented 6 years ago

I spend some time looking at how you could implement minitouch in recent iOS versions. The gist of it is: you can't, but I think you can get close.

VNC servers on iOS 5 and below use IOHIDEvents but that route is now closed for iOS apps.

The only way that I know of to inject touch events in an iOS device is via Xcode UI Test, the same framework the Facebook WebDriverAgent and Appium build upon.

Any XCUITest application interacts with a service call testmanagerd which does use IOHIDEvents to synthesize touches.

The communication between the XCUITest and testmanagerd (and eventualy IOHIDEvents) runs via the XCTestManager_ManagerInterface interface.

You can send touch events via the _XCT_performTouchGesture method.

There are limitations, though. You are required to send the entire gesture (from touch down to touch up) before it is performed. That means you can't implement a real-time touch mirroring protocol on top of it. For example, for a long touch or a swipe operation, you would first need to let the user perform the swipe before you can replay the swipe. Some UI clients solve this problem by "drawing" the gesture on the screen before it is replayed; but there will always be a lag.

That said, _XCT_performTouchGesture executes nearly instantly; there are definitive no second-long delays. I think the delays @sorccu has seen in WDA stem from the fact that WDA contains a lot of synchronisation points where WDA will wait for, for example, transitions and animations to stop before executing any operation.

Long story short, if you can get creative and can account for the fact that you can only replay a gesture on the device after the user has performed the entire gesture, you can use WDA to build a fairly decent remote control solution on top of it.

glolsh commented 6 years ago

Hello everyone! Any news about iOS support? iOS 11 introduced screenshare option. Maybe this'll help.

sorccu commented 6 years ago

I don’t think anyone is actively working on it right now. Comtributions are welcome :)

glolsh commented 6 years ago

@sorccu nobody including your team? I suppose this has least priority than ability to connect emulated devices to stf. That's bad. I can't find any solutions to create own iOS-device farm.

sorccu commented 6 years ago

People tend to think that relatively popular open source projects have some magic behind them that somehow produces funding to the team, but that’s rarely the case. As we have no business model, “the team” quite literally makes nothing from it, aside from the occasional donation. All the time we spend on the project is free, unpaid labor, that is directly out of our own leisure time. We even buy our own devices with our own money. Considering that, don’t you think you’re being a bit unfair?

glolsh commented 6 years ago

@sorccu I'm just asking, no offence. You already made very powerful tool, no doubt. I know that I'm in no position to demand something from opensource project. About funding - I thought that you have some sponsorship from Headspin. About priorities - It was a question about feature roadmap.

jayzsolt commented 6 years ago

@glolsh if you scroll up a few messages you'll see someone recommended http://kobiton.com - I've heard they provide on-prem android and ios device farm options as well, it's not free though. Wish I could figure out how they did fluent screen capturing with iPhones... :)

jayzsolt commented 6 years ago

@sorccu hi, if you come up with business model, plans and roadmap, i'm pretty sure you'll have more contribution than just a few bug ticket posters. Many of open source software succeeded that way, and your tool is awesome, why not making it even better? :)

glolsh commented 6 years ago

@apprayadmin thank you! Also I agree with you about "plans". If I (and other people too, I think) would be sure that this functionality is going to be implemented, I would donate some money. upd: these guys in kobition allow only testing and autotesting, no way to develop there.

PattabhiramaPandit commented 6 years ago

Agree with glolsh. I think this certainly deserve funding. I will talk to my organization about this.

rdhandapani88 commented 6 years ago

Team, I just want to give one heads up here. Currently I am using Macaca App inspector for object identification in IOS. The XCTestWD agent installed in the device can emulate real device screen, right now atleast we can give an option in OpenSTF to show the connected device screen. But the problem is we cannot control the device, if we have that device control option it will be great for us to continue in IOS also in Open STF. Just check through, and it may be helpful for us.

Regarding Funding, Will talk to my Company also. Lets rock this.

codeskyblue commented 6 years ago

@rdhandapani88 this method has been talked before metioned as WDA

Ashraf-Ali-aa commented 6 years ago

here is an interesting project https://github.com/kishikawakatsumi/ScreenRecorder

glolsh commented 6 years ago

Looks like it can be used for showing display, not for controlling it.

manishPatwari commented 6 years ago

https://github.com/manishPatwari/WebDriverAgent We have tried to improve upon the WebDriverAgent provided by Facebook.

Enhancements:

NOTE: This is a POC from our side and still requires more work. You would be more than welcome to contribute towards making this even better.

Special thanks to @shubhankaryash for contributing in this

guadaran commented 6 years ago

@manishPatwari

Great work! Very impressive the performance improvements that you made upon the Facebook's WebDriverAgent.

codeskyblue commented 6 years ago

can u provide some gif videos? recommended using LiceCap

manishPatwari commented 6 years ago

@codeskyblue We have recorded the video : https://www.youtube.com/watch?v=bGJAKXdfrgM

TuHuynhVan commented 6 years ago

@manishPatwari Just Awesome! Thank you so much! I was able to launch on some Simulator but not able on real ones. I got no device connected with real ones even the build and test OK.

If there is no devices, run the WebDriverAgentRunner testRunnerSocket Could you possibly tell a little more about this?

I'm sorry for asking here because there is no place on the forked project to ask about that!

TuHuynhVan commented 6 years ago

@manishPatwari The root cause (as I saw from the log) is the socket is not connected for the real ones but not sure which one triggered that...

glolsh commented 6 years ago

@manishPatwari allow Issue creation on your repo please.

manishPatwari commented 6 years ago

@TuHuynhVan You have to update the Socket path in this file : https://github.com/manishPatwari/WebDriverAgent/blob/master/WebDriverAgentLib/Routing/FBWebSocket.m ,

Replace "localhost" with the ip address where Node server is running.

sorccu commented 6 years ago

Guys thanks for your hard work but can you please take this discussion to your repo :)

manishPatwari commented 6 years ago

@glolsh I have enabled issue creation and also created reference issue in my repo . https://github.com/manishPatwari/WebDriverAgent/issues/1

glolsh commented 6 years ago

@sorccu we were trying to do so:)

rdhandapani88 commented 6 years ago

Team, Entire World is waiting for this precious solution for iOS with OpenSTF. STF team can consider manishPatwari's webdriver agent implementation of mirroring and accessing connected IOs devices. I know you guys are busy with other stuff's. But to make openSTF more power ful with IOs, we all should be united. Team need to tell, what they want actually ? what is blocking. Regards.

JamesChungZLL commented 5 years ago

I'm still working on this. The code's in much better shape, and the stakeholders are listening.

Thanks for the tip on TIFF to JPG. It shaved 100ms off the image processing.

Hi kwv, awesome work! Are you willing to share the image processing code implemented with fast ios screenshot?

qmfrederik commented 5 years ago

The Appium WebDriverAgent has evolved significantly since this work was started (over 3 years ago!). In particular, you probably want to look at:

  1. The MJPEG endpoint they have added. On recent iOS devices you can get a very high framerate (> 10 fps)
  2. They have implemented the actions API which gives you the ability to simulate multitouch gestures at a reasonable speed.

Not the same as minicap + minitouch on Android but much closer.

qmfrederik commented 5 years ago

We've released Docker images of xcuitrunner, a commercial tool which allows you to start the WebDriverAgent on iOS devices, even from Linux hosts. http://docs.quamotion.mobi/quamotion4appium/xcuitrunner-docker/ has some rough documentation on how you could use it.

I think it kind of gives you all the building blocks you need to extend OpenSTF to iOS, if you wanted to:

I'm just wondering, if someone wanted to integrate this into OpenSTF, how would that work? I assume we'd need some kind of middleware (I believe OpenSTF mainly uses gRPC?)? How would we register it & what APIs would that middleware need to expose?

histronger commented 4 years ago

Does anybody make a step to support IOS in STF?