search

Genymobile / scrcpy

Display and control your Android device
Apache License 2.0
105.45k stars 10.24k forks source link

Create MJPEG Server using scrcpy #3680

Open ghost opened 1 year ago

ghost commented 1 year ago

Is there any way to create MJPEG Server using scrcpy

Tryanks commented 1 year ago

Hi, I am using Go language to implement a similar requirement to yours. Based on the learning progress so far, I can provide my answer: it's not possible. The server side of scrcpy uses the native encoders provided by google, and on the vast majority of phones they are OMX.google.h264.encoder. This means that it can only provide video data encoded in h264. However, my idea was to re-implement the scrcpy client using the Go language. Use the libav library to decode h264 and transcode it to MJPEG in order to serve MJPEG streams. If you're interested in developing it yourself, I'd be happy to share all the details except transcoding (since I haven't implemented transcoding yet), as long as this doesn't violate the scrcpy maintainer's comments.

wchensc commented 1 year ago

Hi @Tryanks , I am also interested in the topic. I want to decode h264 and transcode it to JPEG in the Golang client side. But I am very new to media/video programming. Would you mind sharing some ideas or tutorials? I didn't find much useful information online and got stuck right now. Thank you so much.

Tryanks commented 1 year ago

Would you mind sharing some ideas or tutorials?

@wchensc I have not had much time to continue this research, but I still have some progress available here.

First, I modified the Server code to force the encoder to encode every frame as an IDR frame. This had some effect on latency and packet loss issues, but the phone became noticeably laggy.

I'm also a beginner when it comes to video codecs, so my first thought was to use ffmpeg for conversion. At first, I tried to use goav, but it was unmaintained for years and the API implementation was too ugly. Then I found a much better ffmpeg binding: go-astiav, where I asked my question and got help and a quick reply. Now I have proceeded to the stage of re-encoding MJPEG, but no further progress has been made so far.

When I'm done with these features, I'll post my solution here for reference, but not yet :)

wchensc commented 1 year ago

@Tryanks thanks for the response. Btw how did you force the encoder to encode every frame as an IDR frame. I tried to set the iframe interval to 1 sec but it didn't work. https://github.com/Genymobile/scrcpy/blob/master/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java#L24

Tryanks commented 1 year ago

@Tryanks thanks for the response. Btw how did you force the encoder to encode every frame as an IDR frame. I tried to set the iframe interval to 1 sec but it didn't work. https://github.com/Genymobile/scrcpy/blob/master/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java#L24

I also tried many values, however none of them worked except for 0. The server still behaves as if it only has the initial I-frame, followed by all P-frames, and cannot wait for a new I-frame. Among other ISSUEs, I saw that @rom1v said that normally, I-frames should appear about once every ten seconds. But what I have observed is that I-frames never appear.