Appium iOS 13.3 mjpeg-stream read ECONNRESET via ffmpeg

brummetj commented 4 years ago

The problem

I'm using the driver.start_recording_screen() functionality of a remote appium server to get a ffmpeg stream to the localhost:9100

such as what you guys provided...

[ffmpeg] ffmpeg version 4.2.1 Copyright (c) 2000-2019 the FFmpeg developers
[ffmpeg]   built with Apple clang version 11.0.0 (clang-1100.0.33.8)
[ffmpeg]   configuration: --prefix=/usr/local/Cellar/ffmpeg/4.2.1_2 --enable-shared --enable-pthreads --enable-version3 --enable-avresample --cc=clang --host-cflags='-I/Library/Java/JavaVirtualMachines/adoptopenjdk-13.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/adoptopenjdk-13.jdk/Contents/Home/include/darwin -fno-stack-check' --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libmp3lame --enable-libopus --enable-librubberband --enable-libsnappy --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librtmp --enable-libspeex --enable-libsoxr --enable-videotoolbox --disable-libjack --disable-indev=jack
[ffmpeg]   libavutil      56. 31.100 / 56. 31.100
[ffmpeg]   libavcodec     58. 54.100 / 58. 54.100
[ffmpeg]   libavformat    58. 29.100 / 58. 29.100
[ffmpeg]   libavdevice    58.  8.100 / 58.  8.100
[ffmpeg]   libavfilter     7. 57.100 /  7. 57.100
[ffmpeg]   libavresample   4.  0.  0 /  4.  0.  0
[ffmpeg]   libswscale      5.  5.100 /  5.  5.100
[ffmpeg]   libswresample   3.  5.100 /  3.  5.100
[ffmpeg]   libpostproc    55.  5.100 / 55.  5.100
[ffmpeg] 
[Xcode] 2019-12-26 13:30:38.266967-0700 WebDriverAgentRunner-Runner[46161:2586046] Got screenshots broadcast client connection at 127.0.0.1:54595
[Xcode] 
[Xcode] 2019-12-26 13:30:38.268144-0700 WebDriverAgentRunner-Runner[46161:2586356] Starting screenshots broadcast for the client at 127.0.0.1:54595
[Xcode] 
[ffmpeg] Input #0, mjpeg, from 'http://localhost:9100':
[ffmpeg]   Duration: N/A, bitrate: N/A
[ffmpeg] 
[ffmpeg]     Stream #0:0: Video: mjpeg (Baseline), yuvj420p(pc, bt470bg/unknown/unknown), 1125x2436 [SAR 216:216 DAR 375:812], 25 tbr, 1200k tbn, 25 tbc
[ffmpeg] Stream mapping:
[ffmpeg]   Stream #0:0 -> #0:0 (mjpeg (native) -> mjpeg (native))
[ffmpeg] Press [q] to stop, [?] for help
[ffmpeg] 
[ffmpeg] Output #0, mp4, to '/var/folders/8n/5q15vd794gb0lnf4l8cjc38m0000gn/T/20191126-55838-14vpgk7.j9r8h/appium_652612.mp4':
[ffmpeg]   Metadata:
[ffmpeg]     encoder         : Lavf58.29.100
[ffmpeg] 
[ffmpeg]     Stream #0:0: Video: mjpeg (mp4v / 0x7634706D), yuvj420p(pc), 1125x2436 [SAR 216:216 DAR 375:812], q=2-31, 200 kb/s, 25 fps, 12800 tbn, 25 tbc
[ffmpeg]     Metadata:
[ffmpeg]       encoder         : Lavc58.54.100 mjpeg
[ffmpeg]     Side data:
[ffmpeg]       cpb: bitrate max/min/avg: 0/0/200000 buffer size: 0 vbv_delay: -1
[ffmpeg] 
[XCUITest] Starting screen capture on the device '4f88ec2e5785fc7a4d167855745e3db543029fb5' with command: 'ffmpeg -f mjpeg -i http://localhost:9100 -vcodec mjpeg -y /var/folders/8n/5q15vd794gb0lnf4l8cjc38m0000gn/T/20191126-55838-14vpgk7.j9r8h/appium_652612.mp4'. Will timeout in 180000ms

I use this to increase the screenshot time, but I take it a step further instead of calling driver.get_screenshot_as_png(), I remote ssh into the box and run a script that calls

ffmpeg -i http://localhost:9100/?action=stream -r 1 -vframes 1 -f image2pipe -hide_banner -loglevel panic on a 5 second interval inside a loop for a certain duration...

I grab that png buffer on stdout in python so I can reduce the image size down to half... e.i... from 15mb to 700kb, which will greatly increase the response time to get the image back to the client.

but what I'm seeing that the connection gets reset every time I trigger that action.

I didn't see it crash but I just want to understand why this stream should be getting disconnected from the client every time I interact with the stream via ffmpeg, and then will have to reestablish a new connection on an incremented port number?...

This might be something worth checking out! This might lead to future potential failures for our automation...

Environment

Appium version (or git revision) that exhibits the issue: 1.16-beta.3
Last Appium version that did not exhibit the issue (if applicable): 1.13
Desktop OS/version used to run Appium: 10.14.6
Node.js version (unless using Appium.app|exe): 6.11.3
Npm or Yarn package manager:
Mobile platform/version under test: iOS
Real device or emulator/simulator: real!
Appium CLI or Appium.app|exe: cli

Details

If necessary, describe the problem you have been experiencing in more detail.

Appium logs

[Xcode] 2019-12-26 13:45:08.017539-0700 WebDriverAgentRunner-Runner[46161:2590415] Got screenshots broadcast client connection at 127.0.0.1:54620
[Xcode] 
[Xcode] 2019-12-26 13:45:08.018367-0700 WebDriverAgentRunner-Runner[46161:2590286] Starting screenshots broadcast for the client at 127.0.0.1:54620
[Xcode] 
[iProxy@4f88ec2e] read ECONNRESET
[Xcode] 2019-12-26 13:45:08.217809-0700 WebDriverAgentRunner-Runner[46161:2590421] Disconnected a client from screenshots broadcast
[Xcode] 
[Xcode] 2019-12-26 13:45:13.043453-0700 WebDriverAgentRunner-Runner[46161:2590421] Got screenshots broadcast client connection at 127.0.0.1:54621
[Xcode] 
[Xcode] 2019-12-26 13:45:13.044560-0700 WebDriverAgentRunner-Runner[46161:2590184] Starting screenshots broadcast for the client at 127.0.0.1:54621
[Xcode] 
[iProxy@4f88ec2e] read ECONNRESET
[Xcode] 2019-12-26 13:45:13.224442-0700 WebDriverAgentRunner-Runner[46161:2590421] Disconnected a client from screenshots broadcast
[Xcode] 
[Xcode] 2019-12-26 13:45:18.013060-0700 WebDriverAgentRunner-Runner[46161:2590184] Got screenshots broadcast client connection at 127.0.0.1:54622
[Xcode] 
[Xcode] 2019-12-26 13:45:18.013999-0700 WebDriverAgentRunner-Runner[46161:2590398] Starting screenshots broadcast for the client at 127.0.0.1:54622
[Xcode] 
[iProxy@4f88ec2e] read ECONNRESET
[Xcode] 2019-12-26 13:45:18.228970-0700 WebDriverAgentRunner-Runner[46161:2590421] Disconnected a client from screenshots broadcast
[Xcode] 
[Xcode] 2019-12-26 13:45:23.010304-0700 WebDriverAgentRunner-Runner[46161:2590398] Got screenshots broadcast client connection at 127.0.0.1:54623
[Xcode] 
[Xcode] 2019-12-26 13:45:23.010952-0700 WebDriverAgentRunner-Runner[46161:2590415] Starting screenshots broadcast for the client at 127.0.0.1:54623
[Xcode] 
[iProxy@4f88ec2e] read ECONNRESET
[Xcode] 2019-12-26 13:45:23.234892-0700 WebDriverAgentRunner-Runner[46161:2590415] Disconnected a client from screenshots broadcast
[Xcode] 
[Xcode] 2019-12-26 13:45:28.027723-0700 WebDriverAgentRunner-Runner[46161:2590398] Got screenshots broadcast client connection at 127.0.0.1:54624
[Xcode] 
[Xcode] 2019-12-26 13:45:28.028199-0700 WebDriverAgentRunner-Runner[46161:2590415] Starting screenshots broadcast for the client at 127.0.0.1:54624
[Xcode] 
[iProxy@4f88ec2e] read ECONNRESET
[Xcode] 2019-12-26 13:45:28.241088-0700 WebDriverAgentRunner-Runner[46161:2590184] Disconnected a client from screenshots broadcast
[Xcode] 
[Xcode] 2019-12-26 13:45:33.022904-0700 WebDriverAgentRunner-Runner[46161:2590184] Got screenshots broadcast client connection at 127.0.0.1:54625
[Xcode] 
[Xcode] 2019-12-26 13:45:33.023870-0700 WebDriverAgentRunner-Runner[46161:2590415] Starting screenshots broadcast for the client at 127.0.0.1:54625
[Xcode] 
[iProxy@4f88ec2e] read ECONNRESET
[Xcode] 2019-12-26 13:45:33.223404-0700 WebDriverAgentRunner-Runner[46161:2590415] Disconnected a client from screenshots broadcast
[Xcode] 
[Xcode] 2019-12-26 13:45:38.025578-0700 WebDriverAgentRunner-Runner[46161:2590421] Got screenshots broadcast client connection at 127.0.0.1:54626
[Xcode] 
[Xcode] 2019-12-26 13:45:38.026814-0700 WebDriverAgentRunner-Runner[46161:2590184] Starting screenshots broadcast for the client at 127.0.0.1:54626
[Xcode] 
[iProxy@4f88ec2e] read ECONNRESET
[Xcode] 2019-12-26 13:45:38.203910-0700 WebDriverAgentRunner-Runner[46161:2590398] Disconnected a client from screenshots broadcast
[Xcode] 
[Xcode] 2019-12-26 13:45:43.024101-0700 WebDriverAgentRunner-Runner[46161:2590421] Got screenshots broadcast client connection at 127.0.0.1:54627
[Xcode] 
[Xcode] 2019-12-26 13:45:43.025349-0700 WebDriverAgentRunner-Runner[46161:2590184] Starting screenshots broadcast for the client at 127.0.0.1:54627
[Xcode] 
[iProxy@4f88ec2e] read ECONNRESET
[Xcode] 2019-12-26 13:45:43.193075-0700 WebDriverAgentRunner-Runner[46161:2590184] Disconnected a client from screenshots broadcast

Code To Reproduce Issue [ Good To Have ]

start appium driver and run

driver.start_recording_screen()

Create a python script and run

import sys
from PIL import Image
from io import BytesIO

command = ['/usr/local/bin/ffmpeg', '-f', 'MJPEG',
           '-i', 'http://localhost:9100/?action=stream',
           '-r', '1',
           '-vframes', '1',
           '-f', 'image2pipe',
           '-hide_banner',
           '-loglevel', 'panic',
           '-']

def get_screenshot():
    image_size = 600
    pipe = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    img = pipe.communicate()[0]
    img = Image.open(BytesIO(img))
    byte_array = BytesIO()
    basewidth = image_size
    wpercent = (basewidth / float(img.size[0]))
    hsize = int((float(img.size[1]) * float(wpercent)))
    img = img.resize((basewidth, hsize), Image.ANTIALIAS)
    img.save(byte_array, format='PNG', quality=20, optimize=True)
    byte_array = byte_array.getvalue()
    sys.stdout.buffer.write(byte_array)

if __name__ == '__main__':
    get_screenshot()

python3 script.py

mykola-mokhnach commented 4 years ago

WDA has built-in settings for image scaling and quality control. Check the documentation on mjpegScalingFactor, mjpegCompressionFactor, mjpegServerFramerate and mjpegServerScreenshotQuality items.

brummetj commented 4 years ago

These must be new... can you point me to a link where this is all documented ?

Are these effecting the stream or the image response itself when using the appium server to get the image ?

E.i.. what’s the most optimal settings to set these at?.. and why would I want to use these settings instead of just running ffmpeg on my own and do my own image scaling / compression.

mykola-mokhnach commented 4 years ago

These must be new... can you point me to a link where this is all documented ?

http://appium.io/docs/en/advanced-concepts/settings/

Are these effecting the stream or the image response itself when using the appium server to get the image ?

These settings affect the WebDriverAgent server itself. E.g. image scaling is done by the mobile phone without affecting the receiving side.

E.i.. what’s the most optimal settings to set these at?.. and why would I want to use these settings instead of just running ffmpeg on my own and do my own image scaling / compression.

There is no such thing as optimal there. Settings depend on the particular use-case the streaming is used for.

and why would I want to use these settings instead of just running ffmpeg on my own and do my own image scaling / compression.

Of course you are not limited in the way how to configure the client of the server. It is open source ;) I only mentioned the things that have been tested. Your scenario is totally valid, but is "custom". Which means it has not been tested and issues are quite possible there. Although, I cannot neither confirm nor reject they are bugs or not.

brummetj commented 4 years ago

Perfect well thanks for answering my questions here!

I'll just need to do more testing for finding out what provides the highest performance, custom or the solution you provide. As well as most reliable...

Happy holidays

mykola-mokhnach commented 4 years ago

Consider moving the topic to Appium forum if you have further questions

appium / appium