Colorspace appears to change frequently

[EmanH]

I'm running a java app and launching/streaming it with xpra from within an Ubuntu 20.04/22.04 docker container (using the html5 client).

It works ok, except for a troublesome change of color depth every time we interact with the app.

See the example here: https://youtu.be/Yxq6-KH2AOc

Observe the light-blue flickering before changing to darker blue. This is something XPRA is doing. The darker blue is the true color, and the lighter blue seems to be a compressed image where some of the pixel depth information is lost maybe? It also effects light greys in other parts of the UI.

I've tried playing around with various encodings, compression settings, quality settings, with opengl, without opengl, pixel-depth settings, etc. Nothing I can try, changes this.

I've tried a couple different base docker images, current one is based on:

FROM nvidia/opengl:1.0-glvnd-runtime-ubuntu20.04

# Set environment variables
ENV DEBIAN_FRONTEND=noninteractive \
    JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

RUN apt-get update && \
    apt-get install -y software-properties-common apt-transport-https wget && \
    wget -qO - https://xpra.org/gpg.asc | gpg --dearmor -o /usr/share/keyrings/xpra-archive-keyring.gpg && \
    echo "deb [signed-by=/usr/share/keyrings/xpra-archive-keyring.gpg] https://xpra.org/ $(lsb_release -cs) main" | tee /etc/apt/sources.list.d/xpra.list && \
    apt-get update && \
    apt-get install -y xpra xpra-html5 xvfb xterm libjs-jquery libjs-jquery-ui fonts-material-design-icons-iconfont && \
    apt-get clean && \ 
    rm -rf /var/lib/apt/lists/*

RUN apt-get update && \
    apt-get install -y openjdk-8-jre openjdk-8-jdk

After trying every parameter I could think of that might be relevent, I've come to the conclusion it might be an XPRA Bug.

@totaam, we'd be stoked if you could comment on what you think this might be, and any ideas we could try to resolve it? Thanks so much!

[totaam]

Does this also happen with the python client? Is the java application relevant here? Can you reproduce with just an image viewer showing the same picture?

[EmanH]

@totaam I can't reproduce when streaming over a VNC connection, so I don't think the java application is relevant. Will try the python client.

[EmanH]

Launching the application using xpra launcher from the host machine works a charm. No color depth issues. So it seems the issue relates to the HTML client. Earlier I was using v3 of XPRA and was getting a bandwidth streaming warning until switching to the latest version. Perhaps that is still the root cause.

Not sure why there would be a bandwidth issue, as everything is running on localhost. Maybe it's related to docker?

[EmanH]

I tried running the html client from the host machine, but the color depth quality issue came back.

[totaam]

from the host machine works a charm. No color depth issues

Right, so the problem is with the html5 client

Not sure why there would be a bandwidth issue, as everything is running on localhost. Maybe it's related to docker?

The bandwidth warnings show up when there are latency spikes - docker containers can end up introducing measurable latency. If connecting locally, you should be using mmap to bypass this issue.

Perhaps that is still the root cause.

Not really, only indirectly.

This sounds like the browser is decoding jpeg or webp screen updates with a different colorspace / color range than what it should be using. What browsers have you tried? Do they all have this issue?

[EmanH]

Might be onto something with the browser decoding. I was using chrome/edge. Firefox doesn't have the colorspace/range issue, but it shows a significantly reduced quality image for those renders. https://youtu.be/dKOlNEK-cEk

[EmanH]

Despite setting --quality=100 --encoding=h264, Firefox still shows pixelated clusters when interacting with the app. Do you think this is still a browser decoding issue?

[totaam]

Do you think this is still a browser decoding issue?

If it only happens with the html5 client, then yes that's where the problem is.

[EmanH]

@totaam ok, any tips to debugging further? I'll try playing around with the html5 client source.

[totaam]

@EmanH run your server with -d compress and see from the server log which screen updates correspond to the colour issues.

[EmanH]

@totaam Setting an encoding on the client-side has solved the issue. Previously I was just setting the encoding server-side. h264 and webp are both working well.

[totaam]

@EmanH that is very much the wrong thing to do. But since you didn't figure out which ones cause problems... you will be stuck with a sub-optimal solution.

[EmanH]

@totaam I wonder why it doesn't have the effect when I set h264 or webp from the server, but it does when setting it in the client?

[EmanH]

@totaam Setting video encoders explicitly to nvec also fixes the issue. This works:

xpra start --start=bash /app/start.sh --video-encoders=nvenc --bind-tcp=0.0.0.0:14501 --clipboard=yes --clipboard-direction=both --daemon=no --file-transfer=on

[totaam]

None of that helps, figuring out which specific encoding(s) display with colorspace issues would.

[EmanH]

@totaam Yeah fair enough. -d compress or -d encoder doesn't seem to log anything additional for me, idk? But intercepting packets in the html client, I can see the encodings are coming through as a mix of webp, rgb24 and h264. When the color goes off I'm pretty certain it is the h264 packets.

[totaam]

-d compress will definitely send a message to the server log for every screen update that is sent.

If it is h264 packets that cause the problems then we need to know if all server encoders are affected (ie: --video-encoders=x264 vs --video-encoders=nvenc - in which case the Python client would also be affected by this bug and the solution would likely be server side) and if the reverse is True with other browsers.

Perhaps we need to give a hint when calling VideoDecoder.configure to enable full-range colorspace. Either the defaults are different with Firefox, or the offscreen decode worker makes it behave differently.

[EmanH]

@totaam The issue was only occuring with nvenc x264, and have confirmed that passing in colorSpace: new VideoColorSpace({fullRange: true}) to the videoDecoder.configure() call resolves the issue. I tested and was able to replicate the issue and resolution on Chrome, Firefox and Edge.

Have opened this PR: https://github.com/Xpra-org/xpra-html5/pull/248

[totaam]

Is it the case that different browsers use different defaults or that nvenc uses "full-range" whilst other encoders do not? If the former then fine, if the latter then we either need to also fix the encoders to use full-range, or pass a flag down to the client so that it knows what range to use when decoding. (I haven't noticed this with the Python client but perhaps I wasn't careful enough when running the color tests)

[EmanH]

My bad, I mean x264 was the one causing trouble

[EmanH]

Firefox was still having an issue, but appears to handle it differently. Yeah, I'd guess x264 uses the full range while nvenc doesn't?

[totaam]

It could also be related to the colorspace conversion module used on Debian, which only recently got a libyuv package, so xpra on Ubuntu 20.04 is probably using swscale instead. Until I know more about exactly where the color range is being misinterpreted, I cannot merge your PR as this could make things worse in terms of fixing this properly. You can see the video encoding settings used for each window with:

xpra info | grep client.window

(look for csc and encoder)

[totaam]

I've merged the PR, but I am still hoping to enhance the draw packets with better information about colourspace so that the server can specify which type of range to use client-side.

[totaam]

We still have work to do on colourspaces. Here's a link to some docs:

• webgpu color-spaces
• imagebitmapoptions colorspaceconversion
• imagebitmapoptions premultiplyalpha

[totaam]

I know what we're doing wrong, we're not populating "Annex E" data so the decoders are guessing what colorspace is used, some get it wrong. The xpra client knows what settings are usually used, but this isn't correct either. We should provide the correct metadata values, both in the stream itself and in the xpra video stream metadata.

[totaam]

Much improved by the commit above.

Still TODO:

• review all the tickets related to colorspace
• figure out how to set / get the range for openh264 encoder and decoder
• nvenc, nvjpeg, nvdec
• test with older versions

[totaam]

Not sure how I missed it before, but with my current test config I can see colourspace issues with webp (ie: running VirtualBox on Fedora 40). Easily reproducible with:

xpra control :6 request-update png all "*"
xpra control :6 request-update webp all "*"

All the other encodings look OK, but webp is washed out. Like it's painting studio range colours without stretching them.