QVH and Safaridriver?

[soulgalore]

Hi @danielpaulus thanks for this great project! I've incorporated into https://github.com/sitespeedio/browsertime that is a project that measure web performance of web pages. One part of it is that we record a video of the browser screen and analyse when elements are painted on the screen and that's why I'm working on using qvh for Safari on iOS.

Internally we use WebDriver to drive the browser so for Safari we use Safaridriver. Today when I use qvh together with Safardriver there's a couple of problems when qvh shutdowns the connection to the phone so the session with Safari dies. Let me first go through the steps how we do it on a high level:

1. Start the browser
2. Navigate to a single color page (we use orange). That way its easier to know when navigation to the new page really starts
3. Start the video
4. Navigate to the new page and wait on the page to finish loading
5. Stop the video
6. Collect metrics from the page using JavaScript
7. Close the browser
8. Analyse the video

When I do it with qvh, sometimes qvh seems to kill the current Safari session in step 3 and always kills it in step 5 when the video is stopped. Maybe there's a workaround I can do? I could potentially move to stop the video after we collect all the JavaScript metrics just for Safari on iOS but maybe there's another way? Can you explain how starting of the video works, since it sometimes seems to stop the current session but not always?

[danielpaulus]

hey @soulgalore ! Thanks for using my project :-) I will try and help you with some suggestions, let me know if those are working for you.

If the device is not activated for ScreenRecording, what happens is that internally a USB command is send that causes the device to disconnect for a short amount of time. I guess that is what happens for step 3. If the device was used before and is still activated, then it would work fine but with a freshly plugged in or reset device, you would see a short disconnect. If you start with the activate command as a first step, it should work without disconnecting.

If it would help you, I could add a command to pause video recording to separate it from stopping the whole thing. So you could pause recording in step 5, run your final steps and then stop qvh.

What do you think?

[soulgalore]

I missed the activate command, thanks I'm gonna try it later today!

The pause thing would be great if it's not too much work for you.

[soulgalore]

I tried activate and that worked great!

[soulgalore]

Hi again @danielpaulus do you think you have time to try to add pause video and separate it from stop the comings months? I totally understand if you don't but I just wanted to check? :) If I understand correctly it will work like this:

1. Start video and give it a name
2. Pause
3. Restart
4. Stop and one we have one video file

The thing I'm after is that in the end, I will need multiple video files (one per start/restart). Or is it possible to create one video per start/restart? Else I can split the video by something like adding overlay on the screen with a color and split on that.

Best Peter

[danielpaulus]

Haha, @soulgalore sorry. Yeah you know how things are :-D Hmm both should be possible. So either multiple files or one file that just does not get any new frames. Probably creating multiple files makes more sense though. qvh is a command line tool and does not have any REST or similar interface right now. I was thinking of using a Linux signal to implement pause/resume. so you could do sth. like kill -s USR1 3699 to send SIG_USR to suspend and resume videos. Would that work for you? ( I think I would be able to add it this week)

[soulgalore]

Multiple files would be great and would make it super easy for my use case.

For implementation: I think we can do whatever you think is the best :)

[soulgalore]

Hi. just wanted to check in if you have time to try if it would work (totally understand if you don't) or if you have run into any issues that makes it harder? :)

[danielpaulus]

Sorry, I got super crazy about adding zipconduit to go-ios for faster app installs :-D I will add this feature to qvh in the next two weeks for sure.

[soulgalore]

Friendly ping :)

[danielpaulus]

Hey there! Sorry this is taking so long, I actually did work on it but it turned out to not be as easy as I thought. I need a little refactoring first, which is overdue anyway. I will try to work on it over the weekend.

[soulgalore]

No problem of course! Let me know if there's something I can do to help, at least I can help out with testing.

[soulgalore]

This is on my Christmas wish list of things to happen for sitespeed.io :)

[danielpaulus]

So i spent the weekend finally refactoring the code in preparation for this feature. I will ship it very soon, targeting next week.

[soulgalore]

Wow this is great news, thank you! :)

[danielpaulus]

Ok I have just executed a test run and it actually works :-D The only issue right now is that I somehow broke the gstreamer support with my changes. Are you using gstreamer to perform any kind of transcoding? Because if not, I could give you a preliminary version for testing that outputs raw h264 and wav really soon.

[soulgalore]

Oh :D We use raw h264 and then convert to a mp4.

[danielpaulus]

how are you converting it then?

[soulgalore]

ffmpeg https://github.com/sitespeedio/browsertime/blob/main/lib/video/screenRecording/ios/convertToMp4.js but maybe there's better way to do it?

[danielpaulus]

qvh is actually supposed to do this also. If you run qvh gstreamer --examples

Writing an MP4 file
    This pipeline will save the recording in video.mp4 with h264 and aac format. The default settings 
    of this pipeline will create a compressed video that takes up way less space than raw h264.
    Note that you need to set "ignore-length" on the wavparse because we are streaming and do not know the length in advance.

    Write MP4 file Mac OSX: 
    vtdec is the hardware accelerated decoder on the mac. 

    qvh gstreamer --pipeline "mp4mux name=mux ! filesink location=video.mp4 \
    queue name=audio_target ! wavparse ignore-length=true ! audioconvert ! faac ! aacparse ! mux. \
    queue name=video_target ! h264parse ! vtdec ! videoconvert ! x264enc  tune=zerolatency !  mux."

    Write MP4 file Linux:
    note that I am using software en and decoding, if you have intel VAAPI available, maybe use those. 

    gstreamer --pipeline "mp4mux name=mux ! filesink location=video.mp4 \
    queue name=audio_target ! wavparse ignore-length=true ! audioconvert ! avenc_aac ! aacparse ! mux. \
    queue name=video_target ! h264parse ! avdec_h264 ! videoconvert ! x264enc tune=zerolatency ! mux."

[danielpaulus]

I haven't done longterm tests with Gstreamer so far but it should work.

[soulgalore]

Yes I've tried with gstreamer but I had problem that the final video had too much noice/artefacts in it? We record the video when Safari access a web page and then analyse the video when the first visual changed happend (and other metrics). I tried with different setting but never got it to work +. we already use ffmpeg in the project for other things.

[danielpaulus]

hmmm yeah I was thinking about replacing gstreamer or at least adding ffmpeg support in addition out of the box. The core code of qvh is really stable and works well, but the golang gstreamer integration never seemed trustworthy/stable enough.

[soulgalore]

ffmpeg out of the box would be nice. Looking forward to try out the preliminary version!

[danielpaulus]

On branch replaceMessageProcessor-add-resume-feature you can try a first attempt. The audio files are currently broken and pause/resume only works once, but can already see how it will work and if that is ok for you. run: go run main.go record

then run kill -USR1 ${pid} for pause and kill -USR2 ${PID} for resume.

[soulgalore]

Woho @danielpaulus this seems to work for my use case! Starting, pausing and then do my webdriver stuff seems to work! I couldn't test the full flow though, specifying the path/name of the video and sound file seems removed/broken, when I try to specify it I just get the Usage output? Is that an easy fix for you?

[danielpaulus]

I am thinking about how to best do that. How about like ffmpeg does it, so you could specify qvh record video%04d.h264 and audio%04d.wav and then it would create video0001.h264 etc. ?

[soulgalore]

Yes that is great, I think it used to work like that? The binaryv0.5-beta gives me:

Usage:
  qvh devices [-v]
  qvh activate [--udid=<udid>] [-v]
  qvh record <h264file> <wavfile> [--udid=<udid>] [-v]
  qvh audio <outfile> (--mp3 | --ogg | --wav) [--udid=<udid>] [-v]
  qvh gstreamer [--pipeline=<pipeline>] [--examples] [--udid=<udid>] [-v]
  qvh --version | version

[danielpaulus]

yes, I removed those temporarily. I needed a good idea how to make multiple files work. Let me add that formating support and tinker a little on stability.

[soulgalore]

Ah I see, sorry I'm slow on Monday mornings :)

[soulgalore]

Hi @danielpaulus I'm still super interested in getting this to work. Any chance you have some time to have another try? :)

[jajo-shubham]

Hi @danielpaulus, any luck with this? I can try to help to contribute and test this if there is any way.