Can't connect using macOS Screen Sharing or Apple Remote Desktop

[tmcintos]

Is it possible to make this work with the built-in Screen Sharing app in macOS, and/or Apple Remote Desktop?

Screen Sharing hangs after prompting for a password (no way to disable passwords with Screen Sharing, as far as I know).

ARD hangs waiting to connect, whether I leave the username/password blank or not.

In both cases, VNC on the A/UX machine is logging the expected message about accepting a connection, and logs about EOF on socket when cancelling at the VNC client.

[tmcintos]

It looks like this means supporting VNC authentication, at least going through the motions.

Note:

• macOS VNC/ARD server advertises VNC version 003.889 with Apple-specific auth types.
• If the A/UX VNC server advertises 003.003, 003.007, or 003.008, macOS Screen Sharing falls back to 003.003.
• If the A/UX VNC server advertises 003.889 with security type [1, 1] (no auth), macOS Screen Sharing fails saying the viewer & server are incompatible—this is where I tried hacking the A/UX server to do this:

[tmcintos]

I faked authentication and got further with Screen Sharing:

diff --git a/app/session.c b/app/session.c
index 0d7d5d1..a1c9633 100644
--- a/app/session.c
+++ b/app/session.c
@@ -152,8 +152,34 @@ void vnc_requestClientProtocolVersion(session* sess) {

 void vnc_sendSecurityHandshake(session* sess) {
    long vncSecurityHandshake = 1;
+   int fd = open("/etc/vncpasswd", O_RDONLY);
+
    sess->last_checkpoint = "vnc_sendSecurityHandshake";
-   sendw(sess, (char*) &vncSecurityHandshake, sizeof(long), 0);
+
+   if ( fd > -1 ) {
+       char password[8] = {};
+       int n;
+
+       n = read(fd, password, sizeof password);
+
+       if ( n > 0 ) {
+           char challenge[16] = {};
+           char response[16] = {};
+           long vncSecurityResponse;
+
+           vncSecurityHandshake = 2;
+
+           sendw(sess, (char*) &vncSecurityHandshake, sizeof vncSecurityHandshake, 0);
+           sendw(sess, challenge, sizeof challenge, 0);
+           // TODO: 3DES encrypt challenge using password, see https://github.com/kanaka/libvncserver/blob/master/common/d3des.c
+           recvw(sess, response, sizeof response, 0);
+           // TODO: compare response to challenge, set vncSecurityResponse = 1 if failed
+           sendw(sess, (char*) &vncSecurityResponse, sizeof vncSecurityResponse, 0);
+       }
+   }
+
+   if ( vncSecurityHandshake == 1 )
+       sendw(sess, (char*) &vncSecurityHandshake, sizeof(long), 0);
 }

 void vnc_requestClientInit(session* sess) {
@@ -556,4 +582,4 @@ void vnc_evt_update_request(session* sess, VNCFBUpdateReq* evt) {

    vnc_upd_send_hdr_raw(sess, &evt->incremental);
    vnc_send_body_raw(sess, evt->incremental);
-}
\ No newline at end of file
+}

But then got into a failure loop on Set Pixel Format—looks like macOS wants 32bpp (24bpp RGB truecolor right-justified in 32 bits), which doesn't look feasible without major changes to the current scheme based on 8-bit indexed color:

Here are the supported encoding types, besides the required raw encoding: