Closed totaam closed 7 years ago
Updates:
Still todo:
Updates:
r14387 enables scrolling, but this won't fire as it is because h264 is still disabled by default, maybe mpeg4 video needs doing first? (#1107)
html5-mpeg4video.patch
(4.6 KiB)try to use the mpeg4 video from #1107
html5-mpeg4video-v2.patch
(7.3 KiB)updated patch
Useful pointers for html5 video:
mp4info audio.mp4
after saving the sound stream using XPRA_SAVE_TO_FILE=audio
Not video related: browser feature compat table.
html5-mpeg4video-v3.patch
(44.7 KiB)this works! (on Firefox, with one frame of latency, no cleanup code, etc..)
html5-mp3.patch
(8.1 KiB)support mp3 sound via MediaSource (google chrome only...)
html5-sound.patch
(6.2 KiB)try more codecs to attempt to play in all browsers
html5-mediasource.patch
(16.2 KiB)video and audio through mediasource api
html5-mediasource-v2.patch
(22.3 KiB)gives 3 working audio codecs with google-chrome, still none with Firefox
flush-video-frame.patch
(3.9 KiB)ugly PoC hack for flushing frames from the ffmpeg encoder
html5-mediasource-v4.patch
(75.4 KiB)updates: buffer audio (now works with firefox), flush video (video shown is up to date)
As of the patch above:
Still TODO:
PS: using XPRA_SAVE_TO_FILE=1
then using ffmpeg -i test-vp8+webm-4.webm out%03d.png
I can inspect individual frames as PNG and see that the stream is always up to date. So the problem is due to video buffering on the client side...
Large update for handling sound via mediasource with aurora as fallback done in r14412 + r14413 + r14414 (mostly svn subclipse client f--- up!)
The video still lags, but this is not a problem with the video, the same happens with jpeg or png with Firefox! (the patch above just happened to work most of the time by repainting unnecessarily)
html5-mpeg4video-v4.patch
(59.3 KiB)patch updated for already merged audio bits
Improvements and fixes:
About the status of the Broadway decoder:
RangeError: invalid array length
Decoder/onPicFun<() Decoder.js:138
bound () self-hosted
_broadwayOnPictureDecoded() Decoder.js:63
kb() Decoder.js:69
decode() Decoder.js:329
paint() Window.js:931
XpraClient.prototype._process_draw() Client.js:1377
XpraClient.prototype._route_packet() Client.js:339
XpraProtocolWorkerHost.prototype.open/<() Protocol.js:39
Decoder.js:138 Uncaught RangeError: Source is too large
at Uint8Array.set (native)
at Decoder.<anonymous> (http://localhost:14500/js/lib/broadway/Decoder.js:138:25)
at _broadwayOnPictureDecoded (http://localhost:14500/js/lib/broadway/Decoder.js:63:53425)
at Object.kb [as _broadwayPlayStream] (http://localhost:14500/js/lib/broadway/Decoder.js:69:3425)
at Decoder.decode (http://localhost:14500/js/lib/broadway/Decoder.js:329:16)
at XpraWindow.paint (http://localhost:14500/js/Window.js:912:25)
at XpraClient._process_draw (http://localhost:14500/js/Client.js:1377:7)
at XpraProtocolWorkerHost.XpraClient._route_packet [as packet_handler] (http://localhost:14500/js/Client.js:339:3)
at Worker.<anonymous> (http://localhost:14500/js/Protocol.js:39:9)
About the mediasource video api:
Still TODO:
Lots more improvements in r14434:
Notes:
Important update: r14474 (+ r14476 + r14477 derp fixup) worker stuff, see commit message for details.
Still TODO:
websocket.onmessage
?Uint8Array
in the protocol receive codepath, this doesn't solve the special cases in the message queue handlers... - but this can be used already with XPRA_INLINE_SIZE=0 xpra start ..
, works ok and removes one special case.. (but would also cause backwards compatibility issues with older servers)process_interval
better?Make sure we empty the queues fix: r14480.
Maybe process_receive_queue
should be using a while loop too rather than using recursion.
PS: new task: #1372, the html5 client should live in its own package.
More thoughts on the byte-copying: I'm just guessing that this is costly because it was the case with the python code when I had profiled it, many years ago now. Maybe the javascript engines are really really clever and optimize it somehow? But I doubt it can be as fast as memcpy.. which is already a penalty. It may be that most clients have enough CPU power to spare to do all this unnecessary copying, but every little saving helps.
That's why I had added code to send the packet header and data to the socket separately when the packets are big enough (r246 - 5 years ago!, 64KB is the threshold in the current code), this avoids concatenating them before sending which causes memory churn. And that's using a fast memcpy for doing the copying!
One way of dealing with this would be to queue the "Uint8Array"s directly, and then work a bit harder to extract the header from them. It sounds complicated, but in almost all cases, the header will be contained in a single array, with the data either appended to it or in the next array element(s). I'm sure concatenating multiple arrays into one big array is going to be faster than byte-copying. I'm not sure how much TCP packet aggregation websockets end up doing. It probably depends on all sorts of things from OS network configuration, congestion, browser, etc. We may need to re-queue left-over data from each process_receive_queue iteration, this can be stored in a dedicated variable.
Another note: do we really need to run the "process_xxxx_queue" timers all the time? Seems wasteful. Can't we just start scheduling the timers when we add elements to their queue (if not scheduled already) and stop them when the queue becomes empty?
Tested some with 1.0 r14570 fedora 23 server - using both Chrome (55) on windows 8.1 and... well, Opera (41) on OSX 10.12.1.
With the default settings this html5 client is really really working well.
There are some probably not related to this ticket issues with scrolling events being lost when the session is shifted to a browser that's significantly smaller than the originally connected one (anecdotally, re-sizing the browser so that the canvas was large enough that the server-side chrome browser's scroll bar was visible, the scroll events began being caught); OSX laptop tracpad events result in dizzyingly & unmanageably fast scrolling (something about the tracpad sending more events with a smaller delta, apparently); and the fact that one has to remember that browser shortcuts, like control-t, will be captured by the local browser rather than being passed through to the server-side applications.
On to more relevant issues to this ticket...
Still seeing that
Decoder.js:138 Uncaught RangeError: Source is too large
error mentioned above when video is enabled... had the server-side browser at about 1600x575px and was apparently still too large a source when scrolling.With video enabled it looks like the h264 events start to back up, and it's not too long before I start seeing
Warning: delayed region timeout
s on the server side.2016-12-21 14:51:21,509 client display size is 1749x1106 with 1 screen: 2016-12-21 14:51:21,509 HTML (463x293 mm - DPI: 95x95) 2016-12-21 14:51:21,509 Canvas 2016-12-21 14:51:24,735 Error: expiring 6 missing damage ACKs, 2016-12-21 14:51:24,736 connection may be closed or closing, 2016-12-21 14:51:24,736 sequence numbers missing: 199, 200, 201, 203, 213, 217 2016-12-21 14:51:32,740 Error: expiring 1 missing damage ACK, 2016-12-21 14:51:32,740 connection may be closed or closing, 2016-12-21 14:51:32,740 sequence numbers missing: 220 2016-12-21 14:51:48,959 Warning: delayed region timeout 2016-12-21 14:51:48,960 region is 15 seconds old, will retry - bad connection? 2016-12-21 14:51:48,961 30 late responses: 2016-12-21 14:51:48,961 234 h264: 66s 2016-12-21 14:51:48,961 245 h264: 65s 2016-12-21 14:51:48,961 272 h264: 58s 2016-12-21 14:51:48,961 276 h264: 58s 2016-12-21 14:51:48,961 280 h264: 58s 2016-12-21 14:51:48,961 281 h264: 58s 2016-12-21 14:51:48,961 282 h264: 58s 2016-12-21 14:51:48,961 283 h264: 57s 2016-12-21 14:51:48,961 287 h264: 55s 2016-12-21 14:51:48,962 295 h264: 55s 2016-12-21 14:51:48,962 543 h264: 52s 2016-12-21 14:51:48,962 754 h264: 51s 2016-12-21 14:51:48,962 1022 h264: 49s 2016-12-21 14:51:48,962 1062 h264: 46s 2016-12-21 14:51:48,962 1068 h264: 42s 2016-12-21 14:51:48,962 1083 h264: 39s 2016-12-21 14:51:48,962 1103 h264: 37s 2016-12-21 14:51:48,962 1119 h264: 36s 2016-12-21 14:51:48,962 1120 h264: 36s 2016-12-21 14:51:48,963 1124 h264: 32s 2016-12-21 14:51:48,963 1129 h264: 27s 2016-12-21 14:51:48,963 1134 h264: 27s 2016-12-21 14:51:48,963 1139 h264: 27s 2016-12-21 14:51:48,963 1142 h264: 25s 2016-12-21 14:51:48,963 1147 h264: 24s 2016-12-21 14:51:48,963 1222 h264: 23s 2016-12-21 14:51:48,963 1233 h264: 22s 2016-12-21 14:51:48,963 1259 h264: 22s 2016-12-21 14:51:48,963 1284 h264: 21s 2016-12-21 14:51:48,963 1318 h264: 18s 2016-12-21 14:52:04,050 Warning: delayed region timeout 2016-12-21 14:52:04,051 region is 15 seconds old, will retry - bad connection? 2016-12-21 14:52:04,052 30 late responses: 2016-12-21 14:52:04,052 234 h264: 81s 2016-12-21 14:52:04,053 245 h264: 80s 2016-12-21 14:52:04,053 272 h264: 73s 2016-12-21 14:52:04,053 276 h264: 73s 2016-12-21 14:52:04,054 280 h264: 73s 2016-12-21 14:52:04,054 281 h264: 73s 2016-12-21 14:52:04,054 282 h264: 73s ...
... the painting at this point becomes extremely choppy and the entire session often becomes non-responsive for as much as 10 long seconds (occasionally something like a minute). I'll post a screenshot of a point I got it frozen at.
Trying the mp3, it is stable, but does seem to run further and further out of sync, reaching some 20 seconds off after 10-15 minutes of playing (it took about 20 seconds for sound to stop after pausing a video... and then it took about 10-15 seconds to re-start again once the video was started again... at which point sync was just an abstract concept).
Trying the vorbis sound I find that the sound only lasts a minute or two before the audio queue apparently starts overflowing and the media stream becomes "undefined" client side.
Client console logs:
startup complete Client.js:123 audio media source open Client.js:123 using audio codec string for vorbis+mka: audio/webm; codecs="vorbis" Client.js:123 audio: requesting vorbis+mka stream from the server Client.js:588 server connection is OK Client.js:123 audio start of mediasource vorbis+mka stream Client.js:958 audio play! Client.js:123 close_video: video_source_buffer=undefined, media_source=undefined, video=undefined
Server logs:
2016-12-21 12:00:04,075 sound source using audio codec vorbis 2016-12-21 12:00:04,075 sound source using container format matroska [31605:31656:1221/120010.466497:ERROR:browser_gpu_channel_host_factory.cc(113)] Failed to launch GPU process. [31605:31656:1221/120010.468056:ERROR:browser_gpu_channel_host_factory.cc(113)] Failed to launch GPU process. [31605:31656:1221/120010.468322:ERROR:browser_gpu_channel_host_factory.cc(113)] Failed to launch GPU process. [31605:31656:1221/120010.468485:ERROR:browser_gpu_channel_host_factory.cc(113)] Failed to launch GPU process. [31605:31656:1221/120010.468614:ERROR:browser_gpu_channel_host_factory.cc(113)] Failed to launch GPU process. [31605:31656:1221/120010.468749:ERROR:browser_gpu_channel_host_factory.cc(113)] Failed to launch GPU process. 2016-12-21 12:00:11,156 client 1: close_video: video_source_buffer=undefined, media_source=undefined, video=undefined [1:1:1221/120201.455646:ERROR:KeyboardEventManager.cpp(424)] Not implemented reached in static bool blink::KeyboardEventManager::currentCapsLockState() 2016-12-21 12:03:42,130 client 1: close_video: video_source_buffer=undefined, media_source=undefined, video=undefined 2016-12-21 12:03:53,866 client 1: audio queue overflowing: 250, stopping 2016-12-21 12:03:53,867 client 1: close_audio: audio_source_buffer=[object SourceBuffer], media_source=[object MediaSource], video=[object HTMLAudioElement] 2016-12-21 12:03:53,873 sound source stopping 2016-12-21 12:03:54,842 client 1: audio error 2016-12-21 12:03:54,843 client 1: close_audio: audio_source_buffer=null, media_source=null, video=null [31605:31656:1221/120408.173414:ERROR:browser_gpu_channel_host_factory.cc(113)] Failed to launch GPU process. [31605:31656:1221/120408.173835:ERROR:browser_gpu_channel_host_factory.cc(113)] Failed to launch GPU process. [31605:31656:1221/120408.173895:ERROR:browser_gpu_channel_host_factory.cc(113)] Failed to launch GPU process. [31605:31656:1221/120408.173930:ERROR:browser_gpu_channel_host_factory.cc(113)] Failed to launch GPU process. [31605:31656:1221/120408.173970:ERROR:browser_gpu_channel_host_factory.cc(113)] Failed to launch GPU process. [31605:31656:1221/120408.174002:ERROR:browser_gpu_channel_host_factory.cc(113)] Failed to launch GPU process. 2016-12-21 12:04:08,835 client 1: close_video: video_source_buffer=undefined, media_source=undefined, video=undefined 2016-12-21 12:04:59,822 client 1: close_video: video_source_buffer=undefined, media_source=undefined, video=undefined
Refreshing the browser tab that is connecting to the server, the sound will restart... but it crashes soon after again.
Trying with the opus codec I'm promptly informed in server logs that opus isn't supported on my server and it fails over to vorbis, and behaves as above.
Didn't try wav yet.
Not sure if you'd like any particular logs from the server side for any of these issues... or if you're ready to start making tickets for things like tracpads or non-responsive drop menu buttons in a javascript dialog box.
I'll re-assign this to you to look at and let me know what you want to pursue here and what you might want broken out into a more specific ticket.
ticket1341_chrome-with-video-enabled_scrolling-arstechnica_1600x575px-server-side-browser.PNG
(479.0 KiB)Screenshot of scrolling with video
Uncaught RangeError: Source is too large
is only a problem with the non-native decoding - and I'm still not sure what can be done about this, r14617 will catch those errors and report them to the server which will then re-send the frame as png or jpeg, so at least the screen updates won't freeze - this probably explains the delayed region timeouts
Managed to make some time to test with both OSX 10.12.1 and Windows 8.1 against a 1.0.2 14823 fedora 25 server.
As mentioned in #845, the sound seems to have no real problems aside from the poor network I'm using.
Testing with Chrome 55 and Opera 42 on the OSX and Chrome 56 (beta) on the Windows 8.1, I saw about the same performance on all. Wav would cut out within seconds (not hugely surprisingly); while the opus and vorbis both performed similarly, remaining stable until the client hit spinners and a ping timeout after about 3-5 minutes; and the mp3 (no mention of legacy) seemed to just run and run and clients didn't ping out for 10-20 minutes with that sound running.
As a comparison, I connected an OSX 1.0.1 14723 client (with default presumably vorbis codec) and it took about the same 3-5 minutes to ping timeout.
I did run into a not-at-all-verbose sound error with the Windows 8.1 & Chrome 56, but wasn't able to reproduce at all.
2017-01-24 15:11:35,121 client 14: audio queue overflowing: 250, stopping 2017-01-24 15:11:35,126 client 14: close_audio: audio_source_buffer=[object SourceBuffer], media_source=[object MediaSource], video=[object HTMLAudioElement] 2017-01-24 15:11:35,129 client 14: audio error 2017-01-24 15:11:35,131 client 14: close_audio: audio_source_buffer=null, media_source=null, video=null 2017-01-24 15:11:35,143 sound source stopping
The scrolling, meanwhile, was very good with the defaults, but once I enabled experimental video (h264 I presume) and scrolled the canvas seemed to 'break up into little boxes', each of which seemed to be updating separately... presumably the scrolling code having issues... and I began seeing the 'delayed region timeout' errors. No sign of any new errors though.
I'll try to get a 2.0 build working so I can test that a bit too.
Ok, with a little help from Smo, got the 2.0 (14909) server running and, as a bonus, seem to have upgraded to a less bandwidth-light network.
With that said, made some time to test sound with OSX 10.12.1 and Chrome 56 (the beta).
Looks like it's much more stable with all the codecs (well, didn't make time to try wav yet).
Vorbis and opus seemed about as stable as the (non-legacy) mp3 (vorbis didn't even timeout, must've found a hole in the bandwidth usage crowds).
Didn't manage to try with other browsers on OSX or any on windows ... but I'd be very surprised, after what I saw, to have them behave any differently (well, except maybe better).
Watching my ping times as I was testing, it looked like most of the issues I did see corresponded to super-jitter spikes and timeouts... like:
64 bytes from 10.0.32.138: icmp_seq=517 ttl=64 time=20.302 ms Request timeout for icmp_seq 518 64 bytes from 10.0.32.138: icmp_seq=519 ttl=64 time=24.950 ms 64 bytes from 10.0.32.138: icmp_seq=520 ttl=64 time=21.777 ms 64 bytes from 10.0.32.138: icmp_seq=521 ttl=64 time=22.435 ms 64 bytes from 10.0.32.138: icmp_seq=522 ttl=64 time=22.710 ms 64 bytes from 10.0.32.138: icmp_seq=523 ttl=64 time=25.820 ms 64 bytes from 10.0.32.138: icmp_seq=524 ttl=64 time=29.762 ms 64 bytes from 10.0.32.138: icmp_seq=525 ttl=64 time=22.227 ms Request timeout for icmp_seq 526 64 bytes from 10.0.32.138: icmp_seq=526 ttl=64 time=1012.982 ms 64 bytes from 10.0.32.138: icmp_seq=527 ttl=64 time=50.985 ms 64 bytes from 10.0.32.138: icmp_seq=528 ttl=64 time=231.962 ms
So that seems stable as is.
Turning the video encoding on, I'm still seeing (expected I expect) the
RangeError
issue, though rather pretty and regular.2017-01-30 17:16:33,314 Warning: client decoding error: 2017-01-30 17:16:33,315 2017-01-30 17:16:33,315 client 1: error painting 2017-01-30 17:16:33,316 client 1: h264 2017-01-30 17:16:33,316 client 1: RangeError: Source is too large
I did manage to trigger one interesting error, but otherwise the behavior seems to be exactly as with the 1.0.2 server (canvas breaking up into little boxes when scrolling, etc).
2017-01-30 17:16:33,337 Warning: client decoding error: 2017-01-30 17:16:33,338 2017-01-30 17:16:33,338 Error: failed to encode x264 video frame: 2017-01-30 17:16:33,338 image pixels does not have 3 planes! (found 921600) 2017-01-30 17:16:33,338 source: XShmImageWrapper(BGRX: 20, 131, 640, 360) 2017-01-30 17:16:33,339 options: {'av-sync': True, 'scroll': True} 2017-01-30 17:16:33,339 encoder: 2017-01-30 17:16:33,339 profile : baseline 2017-01-30 17:16:33,339 generation : 247 2017-01-30 17:16:33,339 delayed : 0 2017-01-30 17:16:33,339 height : 360 2017-01-30 17:16:33,340 max-size : (8192, 4096) 2017-01-30 17:16:33,340 preset : medium 2017-01-30 17:16:33,340 frames : 0 2017-01-30 17:16:33,340 quality : 58 2017-01-30 17:16:33,340 lossless : False 2017-01-30 17:16:33,340 frame-types : {} 2017-01-30 17:16:33,340 width : 640 2017-01-30 17:16:33,340 speed : 34 2017-01-30 17:16:33,341 source : unknown 2017-01-30 17:16:33,341 version : 148 2017-01-30 17:16:33,341 tune : zerolatency 2017-01-30 17:16:33,341 src_format : YUV420P 2017-01-30 17:16:33,341 formats : ['YUV422P', 'RGB', 'BGRX', 'BGR', 'YUV420P', 'BGRA', 'YUV444P'] 2017-01-30 17:16:33,341 b-frames : 0 2017-01-30 17:16:33,354 client 1: error painting 2017-01-30 17:16:33,355 client 1: h264 2017-01-30 17:16:33,355 client 1: RangeError: Source is too large 2017-01-30 17:16:33,356 Warning: client decoding error: 2017-01-30 17:16:33,356 2017-01-30 17:16:33,356 client 1: error painting 2017-01-30 17:16:33,357 client 1: h264 2017-01-30 17:16:33,357 client 1: RangeError: Source is too large
I suspect that, once that new issue is looked at, this ticket might be able to be closed and replaced by specific tickets for the video and such... unless you'd like me to check with firefox as a client first?
RangeError: Source is too large
stuff is expected - maybe Dave can help us fix that?Error: failed to encode x264 video frame:
had been seen before (see #1346#comment:9), it looks like a race condition caused by the client decoding error - maybe r14916 will help us avoid it (but maybe not in all cases..), either way - let's not worry too much about this one unless we start hitting it under "normal" conditionsLet's follow up in #1424, #1463
See also:
1300
1298
1232
1107
1194
890