Sending opencv frame to webrtc

[nihalar]

Hi,

This is my current procces:

1. Read webcam using opencv as frame (DONE)
2. Do some modifications to the frames (DONE)
3. Send the modified frame to a peer through aiortc (NOT DONE)
4. Repeat

5. I cant find a way, on how to send opencv frame (after encoding) to the peer through aiortc. Any help please? Thanks.

NOTE: The example below shows the opposite i.e sends the video stream to the peer, the peer accepts it as opencv frames and does processing on it. I want to do the opposite. https://github.com/jlaine/aiortc/tree/master/examples/server

[kueblert]

I created a gist to do something very similar: https://gist.github.com/kueblert/fc1517fc9254e9a3cb0add7795c337f4 It currently generates an opencv image via numpy and rotates it in OpenCVPlayerStreamTrack. But you could probably insert your webcam frame here as well and read from the VideoCapture on each call to recv. The data channel is used to send a command to the video to turn the image a bit darker. Hope that helps others as well, as it took me quite a while to figure out how to handle things correctly without requiring any stream from the client side.

[jlaine]

Take a look at the videostream-cli example, it produces and transforms video frames via OpenCV and sends them using aiortc.

[q920980002]

class VideoTransformTrack(VideoStreamTrack):

def __init__(self):
    super().__init__()  # don't forget this!

    video = cv2.VideoCapture(0)
    video.set(cv2.CAP_PROP_FRAME_WIDTH, 320)
    video.set(cv2.CAP_PROP_FRAME_HEIGHT, 240)

    self.video = video

async def recv(self):
    pts, time_base = await self.next_timestamp()
    res, img = self.video.read()
    frame = VideoFrame.from_ndarray(img, format="bgr24")
    frame.pts = pts
    frame.time_base = time_base
    return frame

[jlaine]

The example above does not look good:

• it will probably hold up the event loop
• the timing information is probably wrong, OpenCV doesn't tell you when the frame was captured

[jonatanalimin]

The example above does not look good:

• it will probably hold up the event loop
• the timing information is probably wrong, OpenCV doesn't tell you when the frame was captured

It's there any good way to send OpenCV frame (read from VideoCapture) using aiortc?