When the host byte order of a client is different from that of the RFB pixel format it is using, we must swap bytes before delivering them to the client. This commit addresses places where we were not properly doing that during tight decoding, manifesting in issues in which solid filled rectangles and gradients could be decoded to the wrong color. As the RFB protocol spec says: "Swap the pixel value according to big-endian-flag (e.g. if big-endian-flag is zero (false) and host byte order is big endian, then swap)."
I have tested these changes on all combinations of (host byte order, pixel format byte order), namely (LE, LE), (LE, BE), (BE, LE), and (BE, BE).
Here are examples of the output before and after this change on a LE client using a BE pixel format.
When the host byte order of a client is different from that of the RFB pixel format it is using, we must swap bytes before delivering them to the client. This commit addresses places where we were not properly doing that during tight decoding, manifesting in issues in which solid filled rectangles and gradients could be decoded to the wrong color. As the RFB protocol spec says: "Swap the pixel value according to big-endian-flag (e.g. if big-endian-flag is zero (false) and host byte order is big endian, then swap)."
I have tested these changes on all combinations of (host byte order, pixel format byte order), namely (LE, LE), (LE, BE), (BE, LE), and (BE, BE).
Here are examples of the output before and after this change on a LE client using a BE pixel format.