[Work in progress] Adding an option --serve tcp:localhost:port

Darkroll76 commented 4 years ago

Hi everyone,

Following this post: #1073 and suggested by @rom1v, I started to work on an option --serve. This option would allow forwarding the video stream to any clients connected. It could be a C# Unity App, Java App or C# UWP App or whatever.

The command would be --serve tcp:localhost:1234

It could be a great solution for people like me, who wants to display the video stream inside a desktop application without redeveloping and rebuilding the scrcpy client in the desired language.

I create this post to give some updates and request some help in C development.

Following these indications :

Basically, when you receive a packet, it is decoded + recorded (if necessary): https://github.com/Genymobile/scrcpy/blob/39356602ed472cc3f533e36ae04a110b247c29e0/app/src/stream.c#L81-L97

We can, in addition, write it to a socket.

But in fact, it could even be written earlier (without even parsing it):

https://github.com/Genymobile/scrcpy/blob/39356602ed472cc3f533e36ae04a110b247c29e0/app/src/stream.c#L229

Originally posted by @rom1v in https://github.com/Genymobile/scrcpy/issues/1073#issuecomment-577636642

So, we create another socket (for testing, we're using localhost and port 27015) In stream.c -> run_stream(void *data)

//open another socket on port 27015
socket_t Listensocket;
Listensocket = net_listen(IPV4_LOCALHOST, 27015, 1);
if (Listensocket == INVALID_SOCKET) {
    LOGI("Listen Error");
    net_close(Listensocket);
    return false;
}

socket_t ClientSocket;
ClientSocket = net_accept(Listensocket);
if (ClientSocket == INVALID_SOCKET) {
    LOGI("Client Error");
    net_close(Listensocket);
    return false;
}

net_close(Listensocket);

// We must only pass complete frames to av_parser_parse2()!
// It's more complicated, but this allows to reduce the latency by 1 frame!
stream->parser->flags |= PARSER_FLAG_COMPLETE_FRAMES;

int cpt = 0;

for (;;) {
    AVPacket packet;
    bool ok = stream_recv_packet(stream, &packet);

    if (!ok) {
        // end of stream
        break;
    }

    ok = stream_push_packet(stream, &packet);

    cpt++;
    net_send(ClientSocket, packet.buf, packet.size);
    printf("Send : %d", cpt);

        av_packet_unref(&packet);
    if (!ok) {
        // cannot process packet (error already logged)
        break;
    }
}

We 're not sure that we are sending the right thing yet but this is our first step. Everyone who wants to help is welcomed 😃 !

Thanks,

Darkroll76 commented 4 years ago

Is it possible to send frames already decoded by FFmpeg instead of forwarding packet? If yes, any idea in which function I should take it and write it to a socket?

rom1v commented 4 years ago

(sorry, I don't have much time right now to review)

Is it possible to send frames already decoded by FFmpeg instead of forwarding packet?

That's a bad idea, the decoded frames are way bigger than the encoded ones. You want to transmit the encoded stream, and decode on the client.

¹ For example, in YUV 4:2:0 at 8 bit/color, each pixel takes 16 bits. In 1920×1080, that's 4Mb/frame. At 60fps, 237Mb/s.

Darkroll76 commented 4 years ago

(sorry, I don't have much time right now to review)

Don't be sorry! You're very helpful with everyone so thank you 😉.

That's a bad idea, the decoded frames are way bigger than the encoded ones. You want to transmit the encoded stream, and decode on the client.

Ok haha, I asked because I'm having trouble decoding H264 stream on my client because I'm using C# with Unity 3D and, by this fact, it's more complicated to use FFmpeg library to decode since it's less documented and have some compatibility issues that's requiring additional development.

rom1v commented 4 years ago

Note that if you just want to test your implementation of the --serve option, you can just use VLC as client:

vlc --demux h264 tcp://localhost:1234

(the latency will be higher because VLC bufferizes, but it should work)

The behavior of the --serve option should be ~ the same as:

# listen on localhost:1234 and stream the ouput of screenrecord
adb exec-out screenrecord --output-format h264 - | nc -l -p 1234

dedosmedia commented 4 years ago

@Darkroll76 I hope you make progress with this important feature! I don't have so advanced C knowledge, so I won't be able to help you. Anyway, Good Luck!

dedosmedia commented 4 years ago

@Darkroll76 I am also working on this feature... https://github.com/dedosmedia/scrcpy/tree/h264 This is what I have till now, it's almost the same as you.

@rom1v When I launch scrcpy, it will wait for a connection to port 27184 (the default, but can be overwritten by --serve PORT). Once that I launch:

vlc --demux h264 tcp://localhost:27184

scrcpy continues its process and launch SDL_Window, where I can see my phone screen, however, VLC does not play anything.

When I do the same, but using Putty (raw mode, port 27184). I can see that I receive a lot of garbage, much more when I handle my screen's phone.

Maybe we are not sending the right thing. Excuse but I don't understand anything about AVFrame, AVPacket, frame, pts, header... etc I am using the code @Darkroll76 used for doing it:

ok = stream_push_packet(stream, &packet);
net_send(ClientSocket, packet.buf, packet.size);
av_packet_unref(&packet);

Maybe we are not sending the right thing... I know you are busy with other matters, however, if you have a few free minutes, please give us a hand to get this working. Thanks in advanced

Darkroll76 commented 4 years ago

[UPDATE]

Hi,

I almost did it. @dedosmedia I changed to packet.data instead. I'm about to create a fork today. For now, only TCP, "localhost" and the port are taken as parameters.

I gave the possibility the display the screen (or not) independently from --serve option. So, in order to only forward the stream, we should specify scrcpy --serve tcp:localhost:1234 --no-display.

@rom1v What about allowing recording and serve at the same time?

TODO:

Need to convert the IP address as char *IP to uint32_t (Currently just checking if it's "localhost", then I set value to IPV4_LOCALHOST)
[update] it's a Windows issue: Handle stopping the process (like doing Ctrl+C) => I handle when the client is lost, it will stop the serve, but when I stop the process by force, MSYS is crashing. I guess it's because It's not closing properly.
Handle more than one client (accept() is called only one time for testing)

IDEAS:

Change --serve to --forward-to or --forward-on could be more clear about what it's doing ?

rom1v commented 4 years ago

I changed to packet.data instead

Good catch :)

What about allowing recording and serve at the same time?

Yes, serve will not prevent recording.

(and it should be done from another thread, not to block the stream on blocking calls).

Allow also UDP

Nope, UDP does not guarantee packet ordering and does not retransmit lost packets, so the stream will be corrupted as soon as one of these events occur.

Darkroll76 commented 4 years ago

[UPDATE] Here is the fork and branch: https://github.com/Darkroll76/scrcpy/tree/serve

Yes, serve will not prevent recording.

Ok, I was wondering if it should be allowed or not.

Nope, UDP does not guarantee packet ordering and does not retransmit lost packets, so the stream will be corrupted as soon as one of these events occur.

So, we can remove tcp as parameter: --serve IP:PORT

rom1v commented 4 years ago

So, we can remove tcp as parameter: --serve IP:PORT

For now, we can. In theory, we could also serve over other protocols (http://, rtsp://…).

npes87184 commented 4 years ago

Very interesting idea!

I simply viewed the code, there are several coding style needs to be refined.

For example,

We use space not tab
The left parentheses should the same line as if, while etc.

After feature completed, I suggest to fix coding style before sending PR.

Overall, I like this idea, we can build some application after supporting this feature. And it can be a temporary workaround before libscrcpy finishing.

rom1v commented 4 years ago

It's a great feature for streaming to OBS without opening a scrcpy window too :wink:

Darkroll76 commented 4 years ago

@npes87184 Are you talking about the code shown in my first post? Because it was just for testing. Thanks for this quick review 😉

dedosmedia commented 4 years ago

It works now! Although sometimes after some seconds the video on VLC becomes "corrupted", anyway the video on scrcpy window is always fine. I suppose it's due to VLC because of its buffering.

You both Rock guys!

rom1v commented 4 years ago

It works now!

:+1:

It's a great feature for streaming to OBS without opening a scrcpy window too :wink:

Arf, it does not work: OBS supports opening a stream from VLC, but not adding extra parameters (--demux h264). :disappointed:

solnyshok commented 4 years ago

hi, thanks a lot for your work. can I ask for a feature package this (tcp) h264 stream into a v4l-like video capture device which can then be used by any application like skype, zoom, obs, etc.

rom1v commented 4 years ago

@solnyshok Yes, I thought there was already a feature request for that, but I didn't find it.

Could you create one please?

Maciejszuchta commented 4 years ago

I'm really looking forward to this feature. Can't wait to use it on my website :D

AVTurovskiy commented 4 years ago

Ok haha, I asked because I'm having trouble decoding H264 stream on my client because I'm using C# with Unity 3D and, by this fact, it's more complicated to use FFmpeg library to decode since it's less documented and have some compatibility issues that's requiring additional development.

Hello

I am trying to implement a scrcpy in c# to extend some functionality in a familiar environment. But, there are some problems with video decoding. Have you found a good solution to decode and show raw h264 in c#? I've tried https://github.com/qaisbayabani/SCRCPY-C-Sharp-Client, but it doesn't seem stable enough. Perhaps you could share some examples, or thoughts about this?

cuntoulishifu commented 3 years ago

Ok haha, I asked because I'm having trouble decoding H264 stream on my client because I'm using C# with Unity 3D and, by this fact, it's more complicated to use FFmpeg library to decode since it's less documented and have some compatibility issues that's requiring additional development.

Hello

I am trying to implement a scrcpy in c# to extend some functionality in a familiar environment. But, there are some problems with video decoding. Have you found a good solution to decode and show raw h264 in c#? I've tried https://github.com/qaisbayabani/SCRCPY-C-Sharp-Client, but it doesn't seem stable enough. Perhaps you could share some examples, or thoughts about this?

I have the same problem as you, did you resolve this? I mean to decode and show raw h264 in c#.

AVTurovskiy commented 3 years ago

Ok haha, I asked because I'm having trouble decoding H264 stream on my client because I'm using C# with Unity 3D and, by this fact, it's more complicated to use FFmpeg library to decode since it's less documented and have some compatibility issues that's requiring additional development.

Hello I am trying to implement a scrcpy in c# to extend some functionality in a familiar environment. But, there are some problems with video decoding. Have you found a good solution to decode and show raw h264 in c#? I've tried https://github.com/qaisbayabani/SCRCPY-C-Sharp-Client, but it doesn't seem stable enough. Perhaps you could share some examples, or thoughts about this?

I have the same problem as you, did you resolve this? I mean to decode and show raw h264 in c#.

No. I am still using ffmpeg. But it sometimes crashes and so I have to reset the connection. All the wrapper libraries I've tried have similar problems and none have proven to be stable enough. Reducing the frame rate slightly improves the situation. Maybe you need to understand the structure of the incoming H264 and write your own wrapper or decoder. If you or someone else comes up with a solution, please let me know.

JMLX42 commented 3 years ago

Hello,

Would this work help for https://github.com/Genymobile/scrcpy/issues/1939#issuecomment-811193380 ?

Regards,

nguyenviettuan96 commented 2 years ago

Hi, Could you tell me how to build from your source on mac ? Thanks

qaisbayabani commented 1 year ago

https://github.com/qaisbayabani/SCRCPY-C-Sharp-Client is 100% stable now and also without adb server and client updated on git

qaisbayabani commented 1 year ago

Ok haha, I asked because I'm having trouble decoding H264 stream on my client because I'm using C# with Unity 3D and, by this fact, it's more complicated to use FFmpeg library to decode since it's less documented and have some compatibility issues that's requiring additional development.

Hello

I am trying to implement a scrcpy in c# to extend some functionality in a familiar environment. But, there are some problems with video decoding. Have you found a good solution to decode and show raw h264 in c#? I've tried https://github.com/qaisbayabani/SCRCPY-C-Sharp-Client, but it doesn't seem stable enough. Perhaps you could share some examples, or thoughts about this?

UPDATE now its Stable

qaisbayabani commented 6 months ago

yup

Genymobile / scrcpy

[Work in progress] Adding an option --serve tcp:localhost:port #1159