How is the data of the scrcpy server displayed on HTML?

[ITBOYLIN]

I created a websocket using nodejs and connected it to the scrcpy server for forwarding. The HTML used jsmpeg, but it had no effect. The canvas was black.

This is the scrcpy script

 adb shell CLASSPATH=/data/local/tmp/vlc-scrcpy-server.jar app_process / com.genymobile.scrcpy.Server 1.25 tunnel_forward=true control=false cleanup=false max_size=1920 raw_video_stream=true

[nnnpa31]

The scrcpy video stream is a h264/h265 encoded bare stream, not an mjpeg stream. to play the bare stream, you should refer to wfs.js or jmuxer.js, not jsmpeg.

[ITBOYLIN]

@nnnpa31

Thank you for your answer. I have tried jmuxer.js and it is working properly. But the delay is very high, is there a better way?

[nnnpa31]

I use these parameters:

const jmuxer = new JMuxer({
  //...
  debug: false,
  flushingTime: 1, // This option significantly reduces latency.
  maxDelay: 0,
  fps: 60
});

The latency can be reduced to 50ms in a local loopback network.

[ITBOYLIN]

@nnnpa31

nice！ The video playback has become faster.

But there is still a problem. After refreshing the page, reconnect to the node's websocket, and the video will no longer play. must restart the scrcpy server. Why is this?

[nnnpa31]

This is a characteristic of h264 encoding: the player must parse the NALU frame by frame. for the Scrcpy server, only the first frame of the whole stream is an I-frame (a complete keyframe), all subsequent frames are P-frames (forward prediction frames).

Server generates I-frames only about every ten seconds, and when you re-establish the connection, the player may only receive P-frames, but more likely: it starts receiving from the middle of a NALU packet, causing him to fail to parse any data at all, causing confusion.

To solve this problem you have to delete the raw_video_stream option on your command and enable the following three options:

send_dummy_byte=false
send_device_meta=false
send_frame_meta=true
// When all three options are false, it is equivalent to raw_video_stream.

Then, refer to #3995 to manually parse the "frame meta" on the node side for the purpose of sending the full packet every time on the websocket.

Next, you need to wait for 10 seconds until the websocket receives the new I-frame and starts rendering properly.

If you want to resume the video as soon as possible, then you will have to resort to restarting the server. Alternatively, you can modify this value ScreenEncoder.java#L21 to reduce the I-frame generation interval, but this will lead to higher bandwidth consumption and thus poorer picture quality.