Coordinator startup incomplete - Video stream does not start

[pespinel]

INFO[0000] Process start - video_enabler proc=video_enabler type=proc_start INFO[0000] Process start - device_trigger proc=device_trigger type=proc_start INFO[0000] Process start - stf_provider client_hostname=mac-519802 client_ip=192.168.1.37 proc=stf_provider server_hostname=127.0.0.1 server_ip=127.0.0.1 type=proc_start INFO[0000] New Interface class=0a subclass=00 type=iface_body uuid=5324bbc2322... INFO[0000] New Interface class=02 subclass=0d type=iface_body uuid=5324bbc2322... INFO[0000] New Interface class=ff subclass=fe type=iface_body uuid=5324bbc2322... INFO[0000] New Interface class=06 subclass=01 type=iface_body uuid=5324bbc2322... INFO[0000] Device object created dev_ios_port=9240 dev_name="iPhone Boe" dev_uuid=5324bbc2322... type=devd_create vid_port=8000 vnc_port=5901 wda_port=8100 INFO[0000] Device connected dev_name="iPhone Boe" dev_uuid=5324bbc2322... type=dev_connect INFO[0000] Process start - ffmpeg ops="[-f avfoundation -i iPhone Boe -pixel_format bgr0 -f mjpeg -bsf:v mjpegadump -bsf:v mjpeg2jpeg -r 10 -vsync 2 -nostats pipe:1]" proc=ffmpeg type=proc_start uuid=5324bbc2322... INFO[0000] Process start - mirrorfeed mirrorfeed_bin=bin/mirrorfeed pipe=video_pipes/pipe_8000 port=8000 proc=mirrorfeed tunName=en0 type=proc_start uuid=5324bbc2322... WARN[0000] Process end - ffmpeg proc=ffmpeg type=proc_end uuid=5324bbc2322... INFO[0000] Process start - ffmpeg ops="[-f avfoundation -i iPhone Boe -pixel_format bgr0 -f mjpeg -bsf:v mjpegadump -bsf:v mjpeg2jpeg -r 10 -vsync 2 -nostats pipe:1]" proc=ffmpeg type=proc_start uuid=5324bbc2322...

[nanoscopic]

Quicktime Player displaying phone video now will interfere with the new streaming mechanism. You can use it to test that video will stream from your phone, but you will need to close Quicktime player for the new mechanism to be able to begin streaming.

There aren't any video permissions needed by the new method. It skips the mechanism that MacOS uses to restrict video pulling from the device.

I have split the "video pulling" part of ios_video_stream out into ios_video_pull, and made it directly use upstream quicktime_video_hack code so that it/we can benefit directly from any updates made by Daniel Paulus. This split leads to greatly increased stability. The video no longer "crashes" for me. I had it running streaming nonstop at 15fps for 2hrs and it had no issue.

It is correct that the "video" on 8100 is not related and is actually not video at all. It is sequential screenshots using the screenshot mechanism of the phone.

I've added a new "secure" option for the jpeg serving to stf_ios_support config.json. This can be used if you are using https for STF ( such as the server/ docker stuff in the repo ) You will need to generate a cert and key ( or copy the one in the server/ ) and specify them in the config also to use it. The jpeg are configured by default now to server directly without going through nginx as a proxy. This will make it easier to use the default code with a standard STF server ( such as stf local )

Just to remind; the error "handle_events: error: libusb: interrupted [code -10]" doesn't seem to affect anything. You can safely ignore it. It is not the cause of the video working or not for you.

With all of the various changes to the way the video works, I am seeing very minimal lag. It looks to be less than 1/10th of a second. I do not see any increased lag either after continuous use.

[pespinel]

@nanoscopic The process ios_video_pull dies for me after a few seconds:

INFO[0077] Process start - ios_video_pull                binary=bin/ios_video_pull proc=ios_video_pull pushSpec="tcp://127.0.0.1:7878" type=proc_start uuid="***6642"
WARN[0081] Process end - ios_video_pull                  proc=ios_video_pull type=proc_end uuid="***6642"

I can see these errors:

Attempt %!i(int=1) to start streaming
{"level":"error","msg":"failed connecting to usb - failed to claim interface 2 on vid=05ac,pid=12a8,bus=20,addr=9,config=6: libusb: bad access [code -3]","time":"2020-05-16T12:46:55+02:00"}
{"level":"error","msg":"failed connecting to usb - failed to claim interface 2 on vid=05ac,pid=12a8,bus=20,addr=9,config=6: libusb: bad access [code -3]","time":"2020-05-16T12:46:56+02:00"}
Attempt %!i(int=2) to start streaming
2020/05/16 12:46:57 handle_events: error: libusb: interrupted [code -10]
Attempt %!i(int=3) to start streaming
{"level":"error","msg":"failed connecting to usb - failed to claim interface 2 on vid=05ac,pid=12a8,bus=20,addr=9,config=6: libusb: bad access [code -3]","time":"2020-05-16T12:46:57+02:00"}
Attempt %!i(int=4) to start streaming
{"level":"error","msg":"failed connecting to usb - failed to claim interface 2 on vid=05ac,pid=12a8,bus=20,addr=9,config=6: libusb: bad access [code -3]","time":"2020-05-16T12:46:58+02:00"}
{"level":"fatal","msg":"Socket new error","time":"2020-05-16T12:46:58+02:00","type":"stream_start_failed"}
{"level":"error","msg":"failed connecting to usb - failed to claim interface 2 on vid=05ac,pid=12a8,bus=20,addr=9,config=6: libusb: bad access [code -3]","time":"2020-05-16T12:47:09+02:00"}

[nanoscopic]

@pespinel "failed to claim interface" is the error shown when another device has the USB interface "open" already by some other process. This is what happens, for example, if you have Quicktime player open when trying to start everything.

Make sure also there are no leftover processes ( such as a ios_video_pull that did not stop for some reason ). ps -Af | grep ios

If there is no quicktime, and no leftover processes, unplug and replug the USB cable for your IOS device and try again.

[Kous92]

@nanoscopic @pespinel With this new update, the situation has improved. And I have identified one important factor to ensure that the video stream process will run: the CPU (maybe the GPU also if it uses it).

For security reasons, we are using an antivirus (Symantec Endpoint Protection), and the antivirus running as Daemon consumes a lot of CPU resources due to his full access analysis. It results to a high latency and slow image stream. If I set the analysis to only one point to avoid any slow downs, the coordination brings the image around 5 second latency. If I swipe,... the changes on device are almost instant on my physical device, with latency on STF page (best: 2-3 seconds sometimes).

NOTE: It's not fully stable. Sometimes, the video output won't run and display on STF web interface.

I can say now: We're almost there !

If I go to an other webpage, I lose the video output when I come back on the page like below: -> NOTE: It keeps sometimes.

Sometimes, I got that on decoding process, below. Sometimes it's fine.

[h264 @ 0x7fe4f2010400] SEI type 128 size 896 truncated at 32
[h264 @ 0x7fe4f2010400] SEI type 128 size 896 truncated at 22
[h264 @ 0x7fe4f200e800] decoding for stream 0 failed
[h264 @ 0x7fe4f180d000] SEI type 128 size 896 truncated at 22
[h264 @ 0x7fcfde80a000] SEI type 128 size 896 truncated at 32
[h264 @ 0x7fcfde80a000] SEI type 128 size 896 truncated at 22
[h264 @ 0x7fcfde00f600] decoding for stream 0 failed
[h264 @ 0x7fcfdd008200] SEI type 128 size 896 truncated at 22
Error while decoding: Resource temporarily unavailable
Error while decoding: Resource temporarily unavailable
Error while decoding: Resource temporarily unavailable
Error while decoding: Resource temporarily unavailable

I'm also checking that it can be started when the screen is locked. It's not possible, the screen must to be unlocked to run the first time.

[nanoscopic]

@Kous92 If you are able to see video continuing without stopping ( say for 5 minutes ), then there should not be any delay. If the video is "delayed" it is because there is something causing it to be unable to be processed fast enough and the delay should increase steadily till there is no video.

The "SEI type 128" errrors indicate that corrupt data is being received by h264_to_jpeg from ios_video_pull. I haven't seen those errors myself on any of the systems I'm running the setup on since the change to use ios_video_pull.

It is true that some amount of CPU power is needed. On my macbook pro; ios_video_pull uses 8% of a core, and h264_to_jpeg uses 20% of a core. About half of what h264_to_jpeg does should be getting accelerated by the GPU on your machine. You should use top/htop and look at how much CPU is being used by those two processes. If h264_to_jpeg is using 95%+ of a core; that is a problem as it is maxed out. If that is the case, I would recommend increasing frameSkip value to 3 over the default configured 2. That will reduce video processing to around 10fps instead of the default 15. ( 30fps is received; 2 means process half of them. 3 means process 1/3 of them ) https://github.com/tmobile/stf_ios_support/blob/master/coordinator/proc_h264_to_jpeg.go#L28

I am also, by the way, running Symantec on the company macbook. I don't know and/or can't configure how it works, but it doesn't seem to affect the video at all.

I am currently testing by using port 8000 directly instead of going through STF just to minimize components to diagnose. Do you see issues with the video through port 8000? ( go to port 8000, and click 'open' to begin streaming video )

[pespinel]

@nanoscopic I'm getting error when running make on a fresh installation:

git clone https://github.com/nanoscopic/ios_video_pull.git repos/ios_video_pull
Cloning into 'repos/ios_video_pull'...
remote: Enumerating objects: 13, done.
remote: Counting objects: 100% (13/13), done.
remote: Compressing objects: 100% (10/10), done.
remote: Total 13 (delta 2), reused 10 (delta 2), pack-reused 0
Unpacking objects: 100% (13/13), done.
/Applications/Xcode.app/Contents/Developer/usr/bin/make -C repos/ios_video_pull
go build  -o ios_video_pull .
# github.com/nanoscopic/ios_video_pull
./main.go:158:11: device.Close undefined (type screencapture.IosDevice has no field or method Close)
make[1]: *** [ios_video_pull] Error 2
make: *** [repos/ios_video_pull/ios_video_pull] Error 2

[nanoscopic]

@pespinel Latest updates to ios_video_pull fix that. Pull again and retry.

[pespinel]

@nanoscopic

git clone https://github.com/nanoscopic/ios_video_pull.git repos/ios_video_pull
Cloning into 'repos/ios_video_pull'...
remote: Enumerating objects: 17, done.
remote: Counting objects: 100% (17/17), done.
remote: Compressing objects: 100% (14/14), done.
remote: Total 17 (delta 5), reused 11 (delta 2), pack-reused 0
Unpacking objects: 100% (17/17), done.
/Applications/Xcode.app/Contents/Developer/usr/bin/make -C repos/ios_video_pull
make[1]: *** No rule to make target `vendor/github.com/danielpaulus/quicktime_video_hack/screencapture/usbadapter.go', needed by `ios_video_pull'.  Stop.
make: *** [repos/ios_video_pull/ios_video_pull] Error 2

[Kous92]

@nanoscopic @pespinel Same situation for me and @chriiis78 after pulling the update of ios_video_pull.

[nanoscopic]

@pespinel @Kous92 @chriiis78 Minor glitch with the Makefile rule I added. It doesn't function correctly till after running go get . on that repo. I've removed the rule to prevent it from causing anyone an issue.

I only ever added the rule to make it easier to tinker with an alter the upstream component, but I am finished with that for now, so the rule isn't needed.

[nanoscopic]

I'm once again closing this ticket. Feel free to create a ticket for each specific issue encountered and they will be addressed where time is available. I appreciate greatly any problem encountered being reported, but it is not helpful for problems to be buried in a continuous rambling chain of posts.

It will be very helpful to have the different issues all of you are encountering in their own tickets, so that I can give a proper response as to what will be done or how to address those specific issues. The way this particular issue is going I don't see that it will help others understand the current state or see what issues currently exist.

As for the topic of this ticket "Video stream does not start", it certainly does, and you all seem to have it in some state of starting, so this ticket is now resolved as far as what was described.