LibVNC / libvncserver

LibVNCServer/LibVNCClient are cross-platform C libraries that allow you to easily implement VNC server or client functionality in your program.
GNU General Public License v2.0
1.11k stars 485 forks source link

libvncclient: fix tight decoding endianness issues #605

Closed jknockel closed 8 months ago

jknockel commented 8 months ago

When the host byte order of a client is different from that of the RFB pixel format it is using, we must swap bytes before delivering them to the client. This commit addresses places where we were not properly doing that during tight decoding, manifesting in issues in which solid filled rectangles and gradients could be decoded to the wrong color. As the RFB protocol spec says: "Swap the pixel value according to big-endian-flag (e.g. if big-endian-flag is zero (false) and host byte order is big endian, then swap)."

I have tested these changes on all combinations of (host byte order, pixel format byte order), namely (LE, LE), (LE, BE), (BE, LE), and (BE, BE).

Here are examples of the output before before and after after this change on a LE client using a BE pixel format.