scrcpy

[connglli]

Overview

scrcpy is used and extended by many manufactures, such as XIAOMI, which they call "多屏协同". This blog then introduces the basic scrcpy tool.

Android Side (AS); Desktop Side (DS)

The approach for scrcpy is that, AS plays a role of server, and DS plays a role of client. In such C/S arch,

• AS keeps encoding the video frame of the device and sending that to DS
• DS accepts that frame, decode it, and render it to the desktop Additionally, DS will also send user inputs to AS, and AS will also send device-specific system events to users (such as clipboard data for ease of use)

Therefore, AS opens three threads

• main: encoding and sending video frames
• control (controller): accepting user inputs
• message (sender): sending device-specific non-GUI events

DS opens four threads

• main: listening to user inputs
• control (controller): sending user inputs
• stream: accepting video frames
• message (receiver): accepting device-specific system events

Implementation

Estanblish Connections

AS/DS plays a server/client role conceptually, which may be not in implementation.

At before, scrcoy lets DS opens a server socket, adb reverses that socket to AS, and AS connects to that socket. That said, in implementation, AS plays a role of client, whereas DS a server. However, due to a bug of adb reverse, scrcpy enables a forward mode.

Option 1: Forward Mode

In adb forward mode, the AS opens a server socket, whereas the DS uses a client socket to connect

• adb forward tcp:port localabstract:scrcpy
• in such mode
  1. AS opens to a unix domain server socket named scrcpy,
  2. adb forwards tcp:port to that server socket, and
  3. DS connects to that tcp:port
• on connect, AS waits for the first incomming socket as the video socket, and the second as the control sockets

Option 2: Reverse Mode

In adb reverse mode, the AS opens a client socket, whereas the DS uses a server socket to connect

• adb reverse localabstract:scrcpy tcp:port
• in such mode
  1. DS opens to a tcp server socket on port,
  2. adb forwards localabstract:scrcpy to that server socket, and
  3. AS conects to localabstract:scrcpy
• on connect, DS waits for the first incomming socket as the video socket, and the second as the control sockets

Android side:

public final class DesktopConnection implements Closeable {
    public static DesktopConnection open(Device device, boolean tunnelForward) throws IOException {
        // Forward Mode
        if (tunnelForward) {
            // opens a unix domain server socket
            LocalServerSocket localServerSocket = new LocalServerSocket(SOCKET_NAME);
            try {
                // waits for the first connection from DS as video socket
                videoSocket = localServerSocket.accept();
                // send one byte so the client may read() to detect a connection error
                videoSocket.getOutputStream().write(0);
                try {
                    // waits for the second connection from DS as control socket
                    controlSocket = localServerSocket.accept();
                } catch (IOException | RuntimeException e) {
                    videoSocket.close();
                    throw e;
                }
            } finally {
                localServerSocket.close();
            }
        }
        // Reverse Mode
        else { 
             // use first connection to DS as video socket
            videoSocket = connect(SOCKET_NAME);
            try {
                 // use second connection to DS as control socket
                controlSocket = connect(SOCKET_NAME);
            } catch (IOException | RuntimeException e) {
                videoSocket.close();
                throw e;
            }
        }
    }
}

Desktop side:

bool server_connect_to(struct server *server) {
    // Reverse Mode
    if (!server->tunnel_forward) {
        // waits for the first connection from AS as control socket
        server->video_socket = net_accept(server->server_socket);
        if (server->video_socket == INVALID_SOCKET) {
            return false;
        }

        // waits for the second connection from AS as control socket
        server->control_socket = net_accept(server->server_socket);
        if (server->control_socket == INVALID_SOCKET) {
            // the video_socket will be cleaned up on destroy
            return false;
        }
    } 
    // Forward Mode
    else {
        // use first connection to AS as video socket
        server->video_socket =
            connect_to_server(server->local_port, attempts, delay);
        if (server->video_socket == INVALID_SOCKET) {
            return false;
        }

        // use second connection to AS as video socket
        server->control_socket =
            net_connect(IPV4_LOCALHOST, server->local_port);
        if (server->control_socket == INVALID_SOCKET) {
            return false;
        }
    }
}

Note that, after above, AS and DS are connected through a video socket and a control socket. Once connected, AS sends the device info (i.e., device name/width/height) to DS through video socket

Threading

After the connection is established, AS:

1. creates a ScreenEncoder for capturing and encoding video frames,
2. starts the control thread and message thread, and
3. stream the video in main thread

public final class Server {
    private static void scrcpy(Options options) throws IOException {
        try (DesktopConnection connection = DesktopConnection.open(device, tunnelForward)) {
            // 1. creates the encoder according to bitrate, framerate, and codec options
            ScreenEncoder screenEncoder = new ScreenEncoder(options.getSendFrameMeta(), options.getBitRate(), options.getMaxFps(), codecOptions,
                    options.getEncoderName());

            // 2. starts the control and message thread
            final Controller controller = new Controller(device, connection);
            controllerThread = startController(controller);
            deviceMessageSenderThread = startDeviceMessageSender(controller.getSender());

            // 3. stream the video in main thread synchronously
            screenEncoder.streamScreen(device, connection.getVideoFd());
        }
        // ...
    }
}

On the other side, DS:

1. initializes the decoder, creates the stream thread for accepting incomming frames
2. initializes the controller, creates the control thread for sending user inputs, and creates the message thread for receiving events
3. falls into the main thread event loop to listen user inputs

bool scrcpy(const struct scrcpy_options *options) {
  // 1. initializes the decoder
  decoder_init(&decoder, &video_buffer);
  // 1. creates and start streaming
  stream_init(&stream, server.video_socket, dec, rec);
  if (!stream_start(&stream)) { // it creates the stream thread
      goto end;
  }
  stream_started = true;

  // 2. initializes the controller
  if (!controller_init(&controller, server.control_socket)) {
      goto end;
  }
  controller_initialized = true;
  // 2. creates the control thread for sending user inputs, and creates the message thread for receiving events
  if (!controller_start(&controller)) {
      goto end;
  }
  controller_started = true;

  // 3. listening to user inputs
  ret = event_loop(options);
}

Video Streaming

Prepare main loop????

To stream video, scrcpy record current screen, and stream to the DS. To record, scrcpy first creates a logic (virtual) display with a custom surface. Then with help of Android framework, the display('s surface) would be rendered with content of current screen.

After obtaining the screen record (mirror), scrcpy feeds that to a codec for encoding, and stream it to DS through video socket.

Screen Record (Mirror)

Option1: MediaProjection

Typically, a virtual display can be created with help of MediaProjection#createVirtualDisplay(), which captures a screen projection to a given Surface.

val display = mediaProjection.createVirtualDisplay("mpdisplay", WIDTH, HEIGHT, DPI, DisplayManager.VIRTUAL_DISPLAY_FLAG_PUBLIC, surface, null, null)

To see details of the above workflow, see ScreenRecorder or [MediaDemo]().

Option1: SurfaceControl

Alternatively, a virtual display can also be created via SurfaceControl#createDisplay(), and configured lately by methods of SurfaceControl.

IBinder display = SurfaceControl.createDisplay("scdisplay", true);
SurfaceControl.openTransaction();
try {
    SurfaceControl.setDisplaySurface(display, surface);
    SurfaceControl.setDisplayProjection(display, orientation, deviceRect, displayRect);
    SurfaceControl.setDisplayLayerStack(display, layerStack);
} finally {
    SurfaceControl.closeTransaction();
}

However, above SurfaceControl APIs are hidden by Android framework, and should be accessed by reflection. Furthermore, the layerStack is a field of DisplayInfo, however, this field is not only hidden by also sealed by the Android framework. Hence, to use DisplayInfo.layerStack in a common app (not a system-level app like scrcpy), one need to unseal the reflection using library like tiann/FreeReflection. Additionally, the createDisplay() method needs ACCESS_SURFACE_FLINGER permission that can only be granted to a system app.

To see details of the above workflow, see scrcpy/ScreenRecorder.

Encoding

For streaming, we need to encode the screen mirror. Considering the surface used by above display can be filled by display, we can directly encode the surface.

Contribute to the Andorid multimedia framework, surface can be directed created as the input buffer of a codec by MediaCodec#createInputSurface().

val encoder = MediaCodec.createCodecByType("video/avc")
...
val surface = encoder.createInputSurface() // use the surface as input instead of the input buffer
encoder.start()

Above code encodes the

Decoding

see stream.{h,c}, decoder.{h,c}, recorder.{h,c}

Control Thread and Message Thread

We do not introduce the implementation if these threads because they are much more easier to implement. See Controller.java, DeviceMessageSender.java for AS, and controller.{c,h}, receiver.{c,h} for DS.

[tiennguyen12g]

very nice guideline. thank bro

[JohnodonCode]

my comment was making fun of the scammers :(

[connglli]

@JohnodonCode Sorry for deleting your comment bro. I could get you. I deleted it as it is related to the scam. I should have let you know; sorry, bro.

[JohnodonCode]

Nah I get it dont worry about it

[bradleydowse777]

It's ok man anything that a scam gotta go ✊✊

On Tue., 20 Feb. 2024, 1:25 pm John Bostick, @.***> wrote:

Nah I get it dont worry about it

— Reply to this email directly, view it on GitHub https://github.com/connglli/blog-notes/issues/115#issuecomment-1953385411, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWNWCUWAEYOIHOGCHN2Y4TLYUQCR3AVCNFSM4WCJ5OA2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOJVGMZTQNJUGEYQ . You are receiving this because you were mentioned.Message ID: @.***>