Regarding client-to-server messages

[sibeshkar]

Hello, thank you very much for building this. I had a query - how do I explicitly separate client-to-server messages recorded, from server-to-client messages? I have a setup where I need to convert mouse and keyboard messages recorded from the client into pyautogui events. Any help appreciated, thanks.

[amitbet]

The way this project is built is that you have a server side connection, which is created when a vnc client connects to the vncProxy, and a client side connection that is created when the proxy connects to the vnc server of your choice, each of them has an array of listeners, that you can add to in order to listen in on the events sent by the remote side (client or server). The recorder (recorder.go) listens to both sides in order to know about everything, but it doesn't record client mouse & keyboard messages. Of the client messages, it only responds to SetPixelFormatMsgType, as you can see in the switch case for SegmentFullyParsedClientMessage on recorder.go, this is since the current recorder implementation records the events needed for screen replay, which means: mainly the frame buffers returned from the server.

You can modify it to listen and record additional client messages, although I don't know if writing them to file will keep in line with the standard fbs file specification (for replay in the tightVnc player). You can also create your own recorder and save whatever events you want in a format of your choosing.

[sibeshkar]

Thanks, I tried this my modifying the code in recorder.go and then logged the output in debug mode. On interacting with the vncserver via a different vnc client, the recorder didn't detect any client Messages from my client, and hence didn't log any of the debug comments.

case common.SegmentFullyParsedClientMessage:
        clientMsg := data.Message.(common.ClientMessage)

        switch clientMsg.Type() {
        case common.SetPixelFormatMsgType:
            clientMsg := data.Message.(*server.MsgSetPixelFormat)
            logger.Debugf("Recorder.HandleRfbSegment: client message %v", *clientMsg)
            r.serverInitMessage.PixelFormat = clientMsg.PF
        case common.KeyEventMsgType:
            logger.Debugf("Key Event Received")
        case common.PointerEventMsgType:
            logger.Debugf("Pointer Event Received")

        default:
            //return errors.New("unknown client message type:" + string(data.UpcomingObjectType))
        }

While I understand how proxy.go records client messages, how does recorder.go also do it? Does the vncserver broadcast messages received from any one client, to all connected clients? Thanks.

[amitbet]

recorder.go is used by proxy.go to record messages, is is inserted as listener to the client-conn and server-conn, but If I understand you correctly - you may have been using "recorder/cmd/main.go", in that case you will not have any client messages since you have no real client since main.go is your client, and it doesn't send any key/mouse messages, just buffer requests. if you have been using proxy/cmd/main.go you should have a server-conn which accepts a connection from a real vnc client - in this case the ServerConn's handle function should be called on client messages and these lines should create a client message segment: seg := &common.RfbSegment{ SegmentType: common.SegmentFullyParsedClientMessage, Message: parsedMsg, } err = c.Listeners.Consume(seg)

[sibeshkar]

The code snippet you gave I can see here already in the ServerConn handle function, https://github.com/amitbet/vncproxy/blob/master/server/server-conn.go#L206. As I see in the debug mode, proxy/cmd/main.go seems to be detecting the Pointer and KeyEvents from the attached client but isn't recording them. How do I both write them to file and read them later? I assumed by including them as cases here : https://github.com/amitbet/vncproxy/blob/master/recorder/recorder.go#L166 . Tried this:

        switch clientMsg.Type() {
        case common.SetPixelFormatMsgType:
            clientMsg := data.Message.(*server.MsgSetPixelFormat)
            logger.Debugf("Recorder.HandleRfbSegment: client message %v", *clientMsg)
            r.serverInitMessage.PixelFormat = clientMsg.PF
        case common.KeyEventMsgType:
            clientMsg := data.Message.(*server.MsgKeyEvent)
            logger.Debug("Recorder.HandleRfbSegment: writing bytes for KeyEventMsgType", *clientMsg)
            r.writeToDisk()
        case common.PointerEventMsgType:
            clientMsg := data.Message.(*server.MsgPointerEvent)
            logger.Debug("Recorder.HandleRfbSegment: writing bytes for PointerEventMsgType", *clientMsg)
            r.writeToDisk()

While I think it is detecting the EventMsgTypes, I don't think it's writing to file. How to fix this? thanks!

P.S : also puzzlingly, FramebufferUpdateRequestMsgType doesn't seem to be accounted for by the above switch-case condition in recorder.go, yet it is still recorded (and is detected by player/cmd/main.go).

[amitbet]

Some clarifications:

• It is not a code snippet, just a part of server-conn that I copied to show you where these messages come from.
• The client's buffer requests are never recorded since there is no purpose in doing so - the recorder is trying to create a file that can be replayed, so it needs the actual frame buffers sent from the server, that contain the screen images, it may also include some mouse positions since mouse can be drawn on the client side (instead of having the mouse cursor as part of the image) in some scenarios.
• r.writeToDisk is just a private function that flushes the member buffer to disk, you would need to fill the buffer with what you want to write in order for it to be written, this is done since the protocol doesn't include an indicator for frameBuffer size, forcing the proxy to understand the frame encoding and parse the rects to keep up with the stream positions. so: in the case of screen buffers the data is written to r.buffer and flushed at the end of the segment by using the writeToDisk function.
• Finally, in order to write a client message you should just use clientMsg.Write(r.writer) or substitute r.writer with a writer of your choosing, the client message built-in serialization routines will take care of the rest. (and again - using r.writer may change the fbs file in a way that will be incompatible with the file format)

[sibeshkar]

Got it. Thank you so much! I will record the client messages into a separate writer.

P.S. One unintended consequence I noticed of doing r.writeToDisk() on detecting KeyEventMsgType and PointerEventMsgType was that playback got much smoother when I played back the file using player/cmd/main.go, as opposed to before, when it was abrupt and jerky. Not sure why this is happening, but am very happy with the result!