kvmd vnc server is not compatible with client talking RFB Protocol 3.3

[zhangyoufu]

Describe the bug RFB Protocol 3.7/3.8 negotiate security types. Server sends an array and client choose one. Security type is encoded as u8.
RFB Protocol 3.3 does not negotiate security types. Server decides the security type and sends a single u32.

To Reproduce Steps to reproduce the behavior, like:

1. Capture VNC packet between client and server
2. Connect to kvmd vnc server using a VNC client that only supports RFB Protocol 3.3
3. Wireshark dissector shows that server sent a malformed Security types supported

Expected behavior kvmc vnc server is expected to downgrade protocol version to RFB Protocol 3.3

Screenshots

Desktop (please complete the following information):

• OS: macOS
• Version: Sonoma 14.5 (23F79)
• VNC client: Screen Sharing

PiKVM info:

• PiKVM platform: v3-hdmi
• KVMD version: 4.2-1

[zhangyoufu]

I understand that PiKVM requires a VNC client that supports either Tight(JPEG) or H.264. Both encoding are not available for RFB Protocol 3.3 clients.

Please simply drop support for RFB protocol < 3.7 and reject connection earlier.

[mdevaev]

Hello. Tight can be used with 3.3, he version only applies to the protocol, not extensions. I think we can just add the correct handling of the 3.3 handshake.

[zhangyoufu]

3.3 spec defined raw(0), copy rectangle(1), RRE(2), CoRRE(4), hextile(5) without mentioning Tight though.

[zhangyoufu]

I opened a pull request for this issue, please checkout. I don't have a working RFB 3.3 client to perform e2e test. But my client passed security type handshake and shows PiKVM.

[mdevaev]

Merged, thank you.