search

google / EarlGrey

:tea: iOS UI Automation Test Framework
http://google.github.io/EarlGrey/
Apache License 2.0
5.61k stars 740 forks source link

Video recording of individual tests #638

Open bootstraponline opened 6 years ago

bootstraponline commented 6 years ago

Debugging UI tests successfully requires video recordings of individual test runs. Are there any thoughts about how to do this properly?

I have found this API from Apple

wuhao5 commented 6 years ago

I would say it is the test runner's job to record the test and EarlGrey is a framework to help you write the test. It is really a case-by-case strategy of how you would run your tests, record them and report. It is also a fact of how and where to upload your artifacts.

What is in your mind to record EarlGrey tests?

tirodkar commented 6 years ago

I was checking this out. Does ReplayKit allow you to download and save the captured video using an easy API? You're able to save it to the Camera Gallery.

bootstraponline commented 6 years ago

I would say it is the test runner's job to record the test

EarlGrey is already recording the tests via screenshots though?

In general it looks like no one has solved this problem in open source for iOS. ReplayKit has API restrictions imposed by 🍎 that prevent seamlessly recording the screen to videos.

wuhao5 commented 6 years ago

Seems like it's not a viable option though

If the preview is accessible from the app, we could automate this to close it out, which is still a good option.

also on iOS 11.0, this seems to be possible.

bootstraponline commented 6 years ago

Interesting. I guess someone will have to investigate.

I like that EarlGrey provides screenshots on failures. I think videos would make it even better to diagnose broken UI tests.

I am working on some internal automation projects so I'm not sure when I'll have time to look into this. Just wanted to see if there was interest on the EarlGrey side.

bootstraponline commented 6 years ago

https://github.com/uxmstudio/UIScreenCapture looks like one way to solve this.

vbanthia-zz commented 6 years ago

For real device, it is possible to use MacOS's QuickTime movie recording API. With limitation of 1 device at a time because 🍎

https://github.com/openstf/ios-minicap

bootstraponline commented 6 years ago

fbsimctl can record everything but it doesn't work in a virtual machine. fbsimctl is able to record directly from the framebuffer so it's a bit nicer than quicktime.

wuhao5 commented 6 years ago

I am also curious about the performance overhead for those two tools and among others, does anyone have any clue?

bootstraponline commented 6 years ago

UIScreenCapture is pretty bad (single digit FPS). FB is awesome and has minimal overhead but it's not an option since physical hardware is required. I'd expect the official Apple API to have the smallest overhead in VMs.

KazuCocoa commented 6 years ago

I like the fbsimctl one to record but I'm not sure details for the performance because I haven't measured it carefully.