search

Genymobile / scrcpy

Display and control your Android device
Apache License 2.0
113.49k stars 10.81k forks source link

How to reduce input delays #3275

Open pokeks19 opened 2 years ago

pokeks19 commented 2 years ago

The latency of scrcpy is very low. However, it is not sufficient for playing real-time games. I understand about the inevitable delay in mirroring the screen. However, I believe that the input delay can be reduced. What do you think is the bottleneck for input delay? In fact, OTG mode has almost no delay. If we assume that the latency due to communication is significant, what if the raw input, like in OTG mode, is converted on the smartphone side? Or can you think of a way to reduce the input delay?

rom1v commented 2 years ago

However, I believe that the input delay can be reduced.

It's very low for me.

What do you think is the bottleneck for input delay?

Probably the time for Android to inject and the app to process the event. Especially if it works correctly in OTG mode.

See #1013.

If the computer triggers event at a higher rate that Android could inject them (typically for touch events), this introduces an increasing delay.

If we assume that the latency due to communication is significant

It's not.

what if the raw input, like in OTG mode, is converted on the smartphone side?

Converting few bytes of data and serializing it is very quick. That's not the bottleneck.

pokeks19 commented 2 years ago

I tried it on several devices and found that the response speed depends on the performance of the device. With a margin of performance, good results were obtained.

By the way, is it possible to use only the scrcpy server for input? In other words, can I use the scrcpy server for input from my own software?

rom1v commented 2 years ago

By the way, is it possible to use only the scrcpy server for input?

You can use only the scrcpy server, but currently it necessary sends the video stream, you could not request only input handling.

GPlays09 commented 7 months ago

What sort of an impact on latency do adb forward and sockets have, do you think the latency would improve if we, for example, (on a device that supports usb programming) opened a comport at bitrate of lets say 115200 or higher and communicate with PC through that.

I know that the latency is already low, but can we go lower? @rom1v

Also, is there a good way to determine the highest rate that the device is able to inject input events so that they don't stack up, which would cause further latency?

I would imagine it to be the screen's own touch refresh rate?

rom1v commented 7 months ago

What is called the input latency is the duration between the time you press a key and the time you get the result on your screen, so it is a full round trip.

This includes the time to forward and inject the event to the device, but also the time for the screen to be captured, encoded, transmitted, decoded and displayed. The time to forward the event is insignificant.

Btw, this is confirmed by:

In fact, OTG mode has almost no delay.

Because in OTG mode, you look at the device. If you use normal input injection but look at the physical device (not the mirrored window), the input latency should also be low.

What sort of an impact on latency do adb forward and sockets have, do you think the latency would improve if we, for example, (on a device that supports usb programming) opened a comport at bitrate of lets say 115200 or higher and communicate with PC through that.

I don't think so, what takes time is mostly encoding/decoding. Increasing the transmission bit rate (socket or not) might also help (the bit rate of a video stream is quite high).

Also, is there a good way to determine the highest rate that the device is able to inject input events so that they don't stack up, which would cause further latency?

I don't know.

GPlays09 commented 7 months ago

Ok, I've realised I worded the question badly.

I was talking about the latency between the time the event gets sent from the PCvia USB to the time it is registered on the device.

How would we go about further reducing that?

wisplite commented 7 months ago

If it doesn't use ADB forwarding, how does this work? I have Andronix installed on my phone and I can forward the TigerVNC port over USB using ADB and I get incredibly low-latency, low enough for it to feel like Linux is running natively on the client device. If it is possible to achieve similar latency using scrcpy, I would love to know what would need to be done.