on scaling

[Lanchon]

thank you all for this project.

i have only used xpra for a couple emergency situations. my experience with and knowledge of it are virtually nonexistent. i had issues with scaling and read around the issue tracker a bit, and found some confusion around scaling.

i'll write down here what i thought as i read around in the hope that some of it might help the devs. please mind my complete ignorance of the project take it all with copious amounts of salt.

there is some scaling implementation. i think the problem with it might be that it was not clear exactly which issues(s) that implementation was meant to fix. scaling can contribute at least to these next issues, and some of these affected the implementation:

• bug fixing: make a large remote shadow desktop usable in a smaller client window.
• efficiency: scaling down images before transmission.
• usability: adjust for differences in local vs remote DPI.
• usability: multi-monitor support.

as a result of this possible lack of focus, you may want to change the meaning of the current --desktop-scaling flag down the road.

server scaling

• the presence of sever scaling is a performance/implementation detail that should in no way change the way the client behaves. the client should always compensate for what the server is doing (i think this is already done).
• client requesting some sever scaling should be considered a hint; the server gets to choose its scaling factor. eg: if a client requests 60% the server may choose to provide a 50% feed, and the client will scale that feed by 120%. or it may choose to provide a 100% feed that the client will scale by 60%.
• server should not scale unless it can do it with good quality. server scaling will always be <= 100%, so zero-order interpolation (nearest neighbor) should in general not be used as it can generate crippling visual artifacts.
  • the above rule can be relaxed for scaling factors much less than 1. eg, if a 1:8 (12.5%) scaling factor is requested, simple interpolation would do little to add quality (however, a proper bidimensional low-pass antialias filter would).
  • a non-interpolating, non-filtering server could choose to do 1:n scaling for integer n. the aliasing generated in this way is more tolerable. in fact, for a client requested x scaling it could choose 1:floor((1+k)/x), where k is a small value (say 0.001 to accommodate rounding errors, or say 0.201 to switch to 1:2 scaling bellow requested 60%).
• another reason to disallow server scaling near 1:1 might be efficiency: a particular server implementation might need to switch from hardware- to software-based compression to scale prior to compression (i think i read something in this regard).

asymmetric scaling

i've read issues mentioning the possibility of stretching a remote desktop to fill a client window of a different aspect ratio. the good way of doing this would be to support 2 different scaling factors for X and Y in the server. the best way IMHO would be to not implement any of this at all. the old days of non-square pixels is long gone. we've all switched to 1:1 pixels and no modern desktop that i know supports asymmetric scaling. it is a feature nobody wants, there is no need to implement it in xpra just because "we can". fill the client window with margins instead.

scaling for efficiency

it could be argued that in some situation you could want to scale on both sides only for efficiency sake. eg, in a 4K shadow displayed on a 4K client you might want to scale down 2:1 on the server, then scale back up 1:2 on the client. for data compression sake, you should get better quality just by asking for more compression for the algorithm rather than doing this trick. but for CPU efficiency, this could be desired.

in any case, this could be handled entirely by the client, and the server only needs to scale when in shadow mode. both in app and desktop mode the client can simply pretend to have smaller display windows and get on with life as usual: open remote app windows with less size and request desktop sessions with less size than actually displayed. this is kind of an obscure feature, and if implemented at all it should probably be implemented as a static setting for windows at creation time, and maybe even for the whole server connection if that is easier. a 4K user could simply say then can't really see their desktop pixels and ask xpra to scale everything up by 2:1.

client scaling

• should be dynamic and independent of server support. it should send hints to the server but always work no matter what the server ends up doing.
• shadow client windows could have a (dynamic) zoom setting where one could select a specific % value, resulting in scrollbars or unused margin area in the client window, or an AUTO setting where it maximizes the zoom to fit the window, typically resulting in margin area in one axis but not both.
• if the above is too much to ask, shadow client windows should simply behave as having AUTO zoom enabled at all times.
• i read about scaling down/up a shadow desktop when maximizing the client window, and IMHO this is undesirable. maximizing a window should have no unexpected semantics than changing its size (eg, it should not force AUTO zoom).
• in contrast, a client full screen window could have some additional semantics, this is typically accepted. if really desired, i would argue for a single "AUTO zoom in full screen mode" checkbox in the title bar menu.
• desktop client windows could have zoom settings, but if the desktop can be remotely dynamically resized (i don't know if this is supported in xpra), this is probably unwanted feature creep.
• margins could at first be added right and top/bottom only to avoid coordinate offsets in mouse events. but given that events have to be coordinate-transformed due to scaling, adding translational components to the required transform is easy enough.
• shadow client windows could also have a button to resize their local size so that the internal area of the windows matches the remote desktop (watch for the case of trying to resize beyond the local desktop size). this button should probably just re-run the window sizing code executed at window creation.
• for the previous bullet, 100% zoom is assumed. this 100% value should probably actually be whatever the user passed though the --desktop-scaling flag. the actual scaling requested from the server changes dynamically with current zoom level, which can change when the user alters the zoom setting or when they resize the local window while the zoom is set to AUTO.
• the default for --desktop-scaling could be 100%... or it could depend on the ratio of local vs remote DPI, especially when that ratio is far away from 1:1.

well those are my $0.02 to add. i'm sure most if not all of it has been thought before, but you never know.

thanks for xpra!

[totaam]

fill the client window with margins instead

That's hard because now you need to center the window and manage the offsets. And you may still have some scaling used, to complicate those calculations.

adding translational components to the required transform is easy enough

Feel free to help.

in a 4K shadow displayed on a 4K client you might want to scale down 2:1 on the server, then scale back up 1:2 on the client...

That's what --desktop-scaling does for values greater than 1:1.

you should get better quality just by asking for more compression for the algorithm

No, scanning a 4K bitmap takes huge amounts of memory bandwidth. Downscaling before compressing reduces compression time significantly. At least for software compression. Hardware compression generally uses less aggressive scaling heuristics.

both in app and desktop mode the client can simply pretend to have smaller display windows and get on with life as usual: open remote app windows with less size and request desktop sessions with less size than actually displayed

The position of all windows could be messed up. The server's virtual screen(s) geometry needs to match the one that the client claims to have. (and if the client is going to upscale things, then the virtual screen needs to be downscaled to match)

if implemented at all it should probably be implemented as a static setting for windows at creation time

It is dynamic, because that's not too hard to do: the client already synchronizes its monitor geometry with the server - it has to. We tried (and failed) to make this per-window. The is an open ticket for that: #3454

but if the desktop can be remotely dynamically resized

You mean the virtual screen running on the server right? It does, this is fundamental to how xpra works.