search

Genymobile / scrcpy

Display and control your Android device
Apache License 2.0
110.84k stars 10.62k forks source link

Help with Streaming H.264 from Scrcpy Server to React App #5361

Open georgefbarbosa opened 2 weeks ago

georgefbarbosa commented 2 weeks ago

Subject: Trouble Displaying H.264 Stream from Scrcpy in React App

Hello everyone,

I'm facing some challenges consuming the H.264 video stream from the scrcpy server in my setup, and I'm looking for advice.

Context: I have a Java application that installs and starts scrcpy on multiple phones connected via USB. After scrcpy starts, I open a socket connection to receive the video stream, and then forward the stream over WebSocket to a React app for display. However, I'm having trouble rendering the video on the UI.

What I've Tried:

Streaming raw H.264 I tried streaming raw H.264 data directly, but the React side fails to display the video properly.

Parsing frames with send_frame_meta=true I parsed each frame on the Java side and forwarded it to the React app. I then tried updating a element with each new frame. However, I'm running into issues parsing frames and rendering them correctly on the UI.

Goal: The ultimate goal is to allow users not only to view the stream but also to interact with the device (e.g., touch inputs, gestures). For this, I believe using a would be more appropriate than a

Details:

Scrcpy server version: 2.7 Phones are correctly streaming video to the Java application. Any guidance, code samples, or library recommendations that could help with displaying the stream in the React app and enabling device interaction would be greatly appreciated!

Thanks in advance for any suggestions!

georgefbarbosa commented 1 day ago

I found the issues in my implementation and now it's working just fine using WebCodecs API. I will post a PoC here soon.