search

Xpra-org / xpra-html5

HTML5 client for Xpra
Mozilla Public License 2.0
214 stars 55 forks source link

fix broadway software h264 decoding #68

Closed totaam closed 3 years ago

totaam commented 6 years ago

Issue migrated from trac ticket # 1839

component: html5 | priority: major | resolution: fixed

2018-05-13 06:38:13: antoine created the issue

Split from Xpra-org/xpra-html5#25, the error that comes up is RangeError: Source is too large.

totaam commented 6 years ago

The error RangeError: Source is too large turned out to be trivial to fix: r19294, just needed a "new" pair of eyes, found it when looking into mpeg1 decoding (#1816).

So r19295 enables broadway software decoding by default, again.

What we may still want to do here:

totaam commented 6 years ago

2018-05-13 08:01:40: antoine commented

The html5 client already sets the correct h264 encoding properties (see also Xpra-org/xpra#1840):

"encoding.h264.YUV420P.profile"       : "baseline",
"encoding.h264.YUV420P.level"     : "2.1",
"encoding.h264.cabac"         : false,
"encoding.h264.deblocking-filter" : false,

And these are correctly applied by the x264 codec (as seen with "-d x264"):

x264 context=0x7f08788859c0, YUV420P  640x356  quality=40, speed=40, source=unknown
 preset=medium, profile=baseline, tune=zerolatency
 me=DIA, me_range=16, mv_range=-1, weighted-pred=0
 b-frames=0, max delayed frames=0
 vfr-input=False, lookahead=-1, sync-lookahead=0, mb-tree=False, bframe-adaptive=FAST
 open-gop=True, bluray-compat=False, cabac=False, deblocking-filter=False
 intra-refresh=False, interlaced=False, constrained_intra=False
 threads=auto, sliced-threads=True
x264 encode YUV420P frame     0 as  IDR slice with 8 nals, tune=zerolatency, total   10776 bytes, keyframe=True , delayed=0

Note: nvenc doesn't have such controls, so it may or may not produce a stream which is more CPU intensive to decode.

So the only remaining issue is the blacklisting, which we can look at after some testing. This:

Decoder.js:450 Invalid asm.js: Unexpected token

could slow things down quite a bit - not sure how to fix that (building it from source is a pain).

totaam commented 6 years ago

Updates:

Which gives us:

window.5.encoder=x264
window.5.encoder.b-frames=0
window.5.encoder.delayed=11
window.5.encoder.fast-decode=True
window.5.encoder.formats=('BGRX', 'YUV422P', 'YUV420P', 'BGRA', 'YUV444P')
window.5.encoder.fps=16
window.5.encoder.frame-types.IDR=1
window.5.encoder.frame-types.P=7
window.5.encoder.frames=8
window.5.encoder.generation=9
window.5.encoder.height=432
window.5.encoder.lossless=False
window.5.encoder.max-size=(8192, 4096)
window.5.encoder.ms_per_frame=1
window.5.encoder.params.bframe-adaptive=NONE
window.5.encoder.params.bluray-compat=False
window.5.encoder.params.cabac=False
window.5.encoder.params.constrained_intra=False
window.5.encoder.params.deblocking-filter=False
window.5.encoder.params.interlaced=False
window.5.encoder.params.intra-refresh=False
window.5.encoder.params.lookahead=0
window.5.encoder.params.mb-tree=False
window.5.encoder.params.me.me-range=16
window.5.encoder.params.me.mv-range=256
window.5.encoder.params.me.type=HEX
window.5.encoder.params.me.weighted-pred=0
window.5.encoder.params.open-gop=False
window.5.encoder.params.sliced-threads=False
window.5.encoder.params.threads=12
window.5.encoder.params.vfr-input=False
window.5.encoder.pixels_per_second=159132896
window.5.encoder.preset=veryfast
window.5.encoder.profile=baseline
window.5.encoder.quality=58
window.5.encoder.source=video
window.5.encoder.speed=82
window.5.encoder.src_format=YUV420P
window.5.encoder.total_time_ms=13
window.5.encoder.tune=film
window.5.encoder.version=148
window.5.encoder.width=640

Still TODO:

totaam commented 6 years ago

why am I seeing many delayed frames?

I was and now I'm not. No idea why.

some corrupted frames too?

I believe those were jpeg frames, there were more of those because of this bug: r19579

Just like Xpra-org/xpra#1816, this is ready for testing. The quality is better than mpeg1, the bandwidth is slightly lower but the CPU usage is higher.

totaam commented 6 years ago

2018-06-12 17:51:51: maxmylyn commented

Quick question:

How do I force it to use h264 over mpeg-1? In Xpra-org/xpra#1816 you mentioned that adding --video-encoders=ffmpeg will force mpeg-1. At least that's how I understood it. Running without that flag definitely has a quality increase. I also tried --video-encoders=x264 and nothing yelled at me so I assume (probably incorrectly) that it worked.

totaam commented 6 years ago

How do I force it to use h264 over mpeg-1?

h264 will be used ahead of mpeg1 by default. If you really want to ensure that mpeg1 will not be used, add --video-encoders=x264,vpx

You can always check what is actually being used using the paint boxes and -d compress.

totaam commented 6 years ago

2018-06-12 21:06:49: maxmylyn commented

Okay that did it.

The h264 decoding looks really good - it's seriously impressive.

Is there anything in particular I should look for? As far as I can tell it's working great.

I'll use it some more today and see if I can't break something. But until then - it's pretty fantastic. I can actually play FTL ([https://www.gog.com/game/faster_than_light]) at a decent framerate (almost the same as the Python2 client) and is actually playable. That's totally awesome.

totaam commented 6 years ago

2018-06-12 21:44:13: maxmylyn commented

Playing around with the HTML5 client some more I am fairly confident I've found a memory leak. If you open up the Chrome task manager (or whatever it's called) with shift+escape in Google Chrome, you can sort tabs by memory. Simply watching a video file with the HTML5 client causes the memory to increase steadily up to over 3GB after a few minutes. Disconnecting causes the memory to go away. Reconnecting with video decoders disabled does not cause the memory to climb.

EDIT: I should make it clear that I also started my server with --video-encoders=x264,vpx, so it's definitely running with h.264 and not mpeg-1.

I'm not sure what logs to get for you, but it's very obvious there's a memory leak with video enabled.

totaam commented 6 years ago

I can only reproduce this by having the javascript console activated. In this case, the memory is used by the "GPU Process" which climbs at the same rate as the xpra html5 client. When I do see the memory leak, it is reproducible without h264 enabled, so it would not belong in this ticket. Note: there was a bug which prevented the server from suspending the video stream when the browser tab is not active, that's fixed in r19631. This does mean that the tab may have to be shown / active to reproduce the bug, as otherwise the screen updates stop.

There are lots of details missing, just to name a few:

Assuming that there is a leak without the debug console enabled, here are some links that may be relevant:

totaam commented 6 years ago

2018-06-13 18:19:52: maxmylyn commented

Okay, in that case, I will follow up in a new ticket.

totaam commented 6 years ago

2018-06-20 18:09:31: maxmylyn commented

Since the memory leak isn't related to the h264 decoding, is there anything left on this ticket or should we close it?

totaam commented 6 years ago

Okay, in that case, I will follow up in a new ticket.

The html5 client memleak ticket is Xpra-org/xpra#1879.

Since the memory leak isn't related to the h264 decoding, is there anything left on this ticket or should we close it?

I would have expected you guys to investigate bandwidth savings, CPU usage and such (this could be done using the automated tests). But this works well enough for me.

totaam commented 5 years ago

Missed a glaring bug: Xpra-org/xpra#2060.