search

openstf / stf

Control and manage Android devices from your browser.
https://openstf.io
Other
13.15k stars 2.7k forks source link

iOS support #64

Open kwv opened 8 years ago

kwv commented 8 years ago

Awesome, inspiring stuff.

TLDR; is iOS support on your roadmap?

I've been tinkering around with a solution that works on iOS devices (or simulators). The approach streams snapshots over websockets to a canvas and forwards user-generated mouse/keyboard events to Appium which replays them against a device. For a stand alone solution, with a single device, it works.

next up on the roadmap:

I understand its hard to comment on a solution you haven't seen. Before I get too much further down this proof-of-concept, I wanted to solicit feedback Are there any plans to support iOS in STF? Have you already been down this path and abandoned it?

proof of concept approach

image capture

To capture images, I've evaluated:

Testing against a physical device is desirable however without jailbreaking the device there are some limitations (lockscreen, home button).

The iOS simulator was able to produce highest framerates, and the closest experience to what STF provides for Android, but there is a nagging feeling it's not going to scale or meet the requirements (e.g. managed in MDM, recgnoize an internal cert chain)

Shortcomings

sorccu commented 8 years ago

Hi,

Thanks for the awesome work. We would like to have iOS support, yes. However we currently do not have the resources to develop it by ourselves, but if someone provides a reasonably functional version I may be able to port the rest of the functionality. I would be very interested in seeing what you've got so far.

Btw you mentioned you were hoping to perhaps compress TIFF frames into PNG. I think PNG is a lost cause for screen capture purposes as the compression is very very slow compared to JPG. JPG (w/ libjpeg-turbo) is the fastest method we've found so far, but over time we're planning on moving away from JPG frames to transferring raw x264 frames instead. As long as the device supports hardware encoding, this method should yield the highest FPS.

kwv commented 8 years ago

I'm still working on this. The code's in much better shape, and the stakeholders are listening.

Thanks for the tip on TIFF to JPG. It shaved 100ms off the image processing.

sorccu commented 8 years ago

Great, really looking forward to this :)

gunta commented 8 years ago

Amazing.

For capturing images, I get that you are using com.apple.mobile.screenshotr which firsts captures a TIFF frame and then you have to convert it to JPEG; which will not get you the max frame-rate.

Since iOS 8 there is com.apple.cmio.iOSScreenCaptureAssistant which gives you an encoded video stream. If you are able to get the stream (you may have to do some reverse-engineering) and then somehow forward that to the browser, you'll probably get the highest FPS.

On the Browser side then you could try to do any necessary processing and show it by using Media Source Extensions or WebRTC.

If you need any help in getting the approval of stakeholders, just let them know that we can add the name of your company in the credits as a big contributor on http://openstf.io, if that is the case just let us know ;)

gunta commented 8 years ago

@kwv any update on this? :)

kwv commented 8 years ago

The effort stalled. Stakeholders haven't said no, but they've shifted priorities.

I'm able to sample stills from a video stream, but it's drinking from a firehose. I was refactoring the image acquisition code to send the results back over a WebSocket (rather than diskIO), when I bumped into a hard limit of one device at a time https://github.com/WPO-Foundation/xrecord/issues/1

leandromoreira commented 8 years ago

Thanks for such amazing work ;)

PattabhiramaPandit commented 8 years ago

Hi ,

Can you please share your solution? We want to test it out for iOS devices?

PattabhiramaPandit commented 8 years ago

If anyone has a SD solution available please share the same. We are in dire need of getting this on iOS.

PattabhiramaPandit commented 8 years ago

Ooppsss!... I meant iOS.. Not SD

sorccu commented 8 years ago

If you feel like you need an iOS implementation faster, consider posting a bounty on this issue (see badge below). If the bounty gets large enough, perhaps we or someone else will take a look at it.

Bountysource

PattabhiramaPandit commented 8 years ago

Dear Sorccu, you mentioned to me that you would give me the exact quote for developing the iOS Support. Can you please give me as soon as possible?

Regards Pattabhi

Ashraf-Ali-aa commented 7 years ago

@sorccu @kwv it might be worth checking this out https://github.com/phonegap/ios-deploy and https://github.com/phonegap/ios-sim for building and deploying iOS apps to device and simulators. I know STF does not currently support iOS but it would be nice to deploy apps via the web browser. The tool also supports lldb which is useful for out putting console logs.

daluu commented 7 years ago

Curious to the original proposed solution that got stalled, as well as any other potential solutions. How far can you scale device support (or even simulator support)?

From what I encountered, unless I'm mistaken, just toying with Appium and physical iOS devices, there seemed to be a single device limitation per instance of XCode on Mac OS. Therefore, to reliably scale to more iOS devices (to work in parallel), one has to deploy 2 virtual machines (the max licensed amount allowed by Apple, on the physical Mac host) running Mac OS with the same XCode and Appium setup, etc. In that setup, assign a specific iOS device per VM via USB port/device allocation in the VM software, and the remaining (last) device to the physical Mac. Each instance (physical Mac, VMs) have their own XCode and Appium to control the device. So there's a max device limit of 3 iOS devices per Mac machine (using the VMs). This doesn't scale as well compared to Android, as more Mac machines would be needed to scale higher.

vbanthia-zz commented 7 years ago

@daluu

XCode supports testing multiple real devices using UIAutomator since v6.3.2

https://github.com/appium/appium/issues/273#issuecomment-110145744

Ashraf-Ali-aa commented 7 years ago

this might be useful for video recording https://github.com/isonic1/flick

codeskyblue commented 7 years ago

@Ashraf-Ali-aa This tools about iOS recording is using capture pictures and combine them to video file. maybe not fitted for stf.

gunta commented 7 years ago

@codeskyblue it works also on Android, should be able to be used without problems

codeskyblue commented 7 years ago

Since stf got minicap, combine the image stream into video is a better way in my view.

gunta commented 7 years ago

Well its for different use cases afaik, at least currently.

codeskyblue commented 7 years ago

@gunta I have a question, how to know ios orientation from command line.

Ashraf-Ali-aa commented 7 years ago

facebook have implemented an extension of xcuitest that allows remote interactions with ios devices and appium has adopted the code.

https://github.com/facebook/WebDriverAgent https://github.com/appium/appium-xcuitest-driver

related links: https://github.com/facebook/FBSimulatorControl https://github.com/rsms/peertalk

sorccu commented 7 years ago

I think that's probably the reason why no one has contributed a solution yet. It takes a huge amount of developer time and there are clearly financial benefits for end users. In other words it would be a viable product. Not many are willing to give that away for free.

On Tuesday, 21 June 2016, Naresh Jain notifications@github.com wrote:

any update on iOS support? If anyone has a iOS solution available for OpenSTF please share, that will help my company to reduce cost on devices.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openstf/stf/issues/64#issuecomment-227408518, or mute the thread https://github.com/notifications/unsubscribe/AAB-_bzGMKCJOp7_YKDQZARghu4O20xWks5qN8WwgaJpZM4FyyUk .

kwv commented 7 years ago

@daluu I never made it past the scaling limitation.

I started down the path of spinning up VMs for each tethered device, but I started to recognize the difference between being a trailblazer and stumbling around in the woods. Defining a repeatable approach was going to be challenging, let alone coming up with a solution that would fit back into STF.

@Ashraf-Ali-aa Capturing video and sending the stream back to a browser was a problem. @codeskyblue is right, it is easier to use images over a websocket to fake a stream. Munging the QT frames into a websocket was next, but that required stronger opinions on the architecture. (Does the app collecting the screen pipe directly to the browser or through a middle agent? If its direct, what supports discovery, or starts the app?) I wanted to leverage a lot of STF's architecture decisions for this, but the approach had already diverged.

I wonder how the commercial products work, if they use an app installed on the device which functions similar to minicap. Rather than pulling from the host machine (like the approach outlined above), push from the device itself (using private APIs?). In the near term FBSimulatorControl looks very interesting.

If nothing else, it just re-enforces that @sorccu and @gunta are doing amazing work.

HelderVSousa commented 7 years ago

Hello guys! Like @kwv i'm trying find a way to screencast the ios screen, i am currently try apply this code from @isonic1, this one https://github.com/isonic1/flick, to make a minicap for IOS. if you guys have some advise to me please tell me.

HelderVSousa commented 7 years ago

@NareshJain91 I am dividing the problem, first i want identify the device and set it visible in stf, next i want make the screencast from ios devices! that is my main work, for now, but we want keep going until get full functionality presented in the android. So far i am fighting to detect iphone and make screencast.Now with flick i am capable to detect the udid for all devices and take screenshots in all ios devices. I will try this path to make a solution.

NareshJain91 commented 7 years ago

@HelderVSousa Sure, if you need my assistance anywhere please let me know. Till now i am able to capture ScreenShots and video from iOS Devices, also i can get how many devices are connected and their UDID. But major problem i am stuck is how to give command from browser/System to iPhone, if it resolves i will be able to implement rest things, for that i have also raised the question to Developer apple forum, but most of have replied that apple do not allow it.

If you find something to give commend from mac to iOS, let me know. Thanks

sorccu commented 7 years ago

They probably meant that Apple wouldn't allow you to publish an app that does that. However, you should be able to use private APIs to your heart's content if you never publish your app in the App Store.

Ashraf-Ali-aa commented 7 years ago

@HelderVSousa you can use https://github.com/phonegap/ios-deploy to detect connected iOS devices

HelderVSousa commented 7 years ago

@Ashraf-Ali-aa thanks for the tip, for now my work is fully focused on detect and screen capture ios, my recent research and tests show me the flick can be the tool i am looking for, i will create minicapIOS based on approach in minicap. But in future i will check it to see if i can use to control the ios.

Ashraf-Ali-aa commented 7 years ago

@HelderVSousa I came across this tool https://github.com/sanekgusev/SGASScreenRecorder but it seems it does not work for iOS 9.

https://github.com/gabriel/CaptureRecord

https://github.com/JaviSoto/iOS10-Runtime-Headers/blob/master/Frameworks/MediaToolbox.framework/FigScreenCaptureController.h

Ashraf-Ali-aa commented 7 years ago

@HelderVSousa check this out https://www.macstories.net/ios/vidyo-a-screen-recorder-for-ios/

https://github.com/johnno1962/Remote

HelderVSousa commented 7 years ago

Hello people i have made some progress however this is not working like i want, now i am using libimobiledivice api to take screenshots and pass that screenshots over socket's to browser, but unfortunately i only can get 4 a 5 frames a second, that is not perfect but is a start, if someone know a lib that take screenshots faster than libimobiledevice, pls share to see what i can make improvements.

kwv commented 7 years ago

@HelderVSousa with https://github.com/WPO-Foundation/xrecord higher FPS is possible, but it's using QuickTime as a bridge.

codeskyblue commented 7 years ago

@HelderVSousa I found out that use idevicescreenshot(tools from libimobiledevice) take 0.7s, how did you make 4,5 fps.

Actually 4,5 fps is enough for the remote control. Touch swipe... basic operations can use https://github.com/facebook/WebDriverAgent (this is so far I know.)

sorccu commented 7 years ago

Note that on iOS the home screen is quite slow for some reason. You can capture screenshots faster in some other views (e.g. Safari). I can confirm that around 4-5 FPS is possible with this very basic method.

luizlohn commented 7 years ago

So, its possible get this WebDriverAgent and integrate with STF?

nhanvpt102 commented 7 years ago

Should we use libimobiledivice api or WebDriverAgent to take screenshots and pass it to browser? or do we have another better solution, please share.

sorccu commented 7 years ago

WDA screenshots suck, don't bother. It stops capturing during any animation or significant activity.

nhanvpt102 commented 7 years ago

Thanks very much for quick respond. If so, we should use QuickTime ("https://github.com/WPO-Foundation/xrecord") or libimobiledivice api (HelderVSousa's comment, 4 a 5 frames a second), right ? I'm still seeking a best solution for iOS. If you have any idea, please share.

codeskyblue commented 7 years ago

AirPlay is another way. very quickly, just do not known how to do it.

nwlunatic commented 7 years ago

I want to share a working prototype https://github.com/nwlunatic/ios-minicap. Any futher ideas/issues/pull requests are welcome.

codeskyblue commented 7 years ago

looks like the only missing stuff is ios-minitouch

nwlunatic commented 7 years ago

Well, I hope someone also will come up with some server/client solution for macOS to solve remote usb connection problem. Like usbip, or any other solutions for remote accesing usb. Or i've missed something? ) This will probably require to write some kind of virtual usb device and kext. And solve a lot of other possible usb problems )

gunta commented 7 years ago

Above repo has been transferred to the OpenSTF org. here: https://github.com/openstf/ios-minicap

aluedeke commented 7 years ago

does this work with more then one device per host?

sorccu commented 7 years ago

I believe so, but it's untested as we're improving things. Anyway it's still very much a prototype with bugs and all. It will take time to polish it. Let's not start asking too much. You can help by testing it yourself and reporting the results.

crashbell commented 7 years ago

I tested multiple devices, it seems not working. It just kept showing usage text whenever I plug 2 iOS devices. One device is good so far. I see that we use CoreMediaIO to enable DAL. That may not work with multiple devices.

nhanvpt102 commented 7 years ago

For ios-minitouch, Is it possible with IOKit framework?

nhanvpt102 commented 7 years ago

or should we use UIKit.framework ? refer to "https://github.com/johnno1962/Remote/blob/master/Classes/RemoteCapture.h" --> (void)processEvents