search

elsampsa / valkka-core

Valkka - Create video surveillance, management and analysis programs with PyQt
GNU Lesser General Public License v3.0
186 stars 35 forks source link

Question: Server side using libValkka on linux and Client side using other library on Windows #30

Closed idonahum1 closed 2 years ago

idonahum1 commented 2 years ago

Hi,

After reading most of the lessons and trying them, first of all I must say that this is a great library, and well documented. Second, I will explain the use case that made to test libValkka: I developed a client-server with UI to control a PTZ Camera using onvif zeep. The UI consist of control buttons and video preview. For getting a video preview, I used onvif GetStreamURL method and Opencv to read frames from it in a seperate thread, the thread signals every frame to the main thread which pushes it to Queue object, from the queue object I pull those frames and update the image preview label in the UI. This method works most of the time, but when I am trying to connect to an IP camera that located very far from the viewer location, I get errors like:

[h264 @ 0x7fd8807d8fc0] error while decoding MB 72 1, bytestream -14
[h264 @ 0x7fd880024b80] error while decoding MB 59 24, bytestream -11
[h264 @ 0x7fd880782500] error while decoding MB 129 56, bytestream -5

Which freezes the video preview for good on the client side Just to make it clear, to access those cameras which located on a different network I use VPN and forwarding the camera ip in order to connect it. Currently the code is server-client code but its tightly coupled and runs on the same machine. I guess that the reason im getting those kind of errors is because OpenCV is not synced with the RTSP stream and reads incomplete frames that results in corrupted frames. I could not find a way to catch those corrupted frames and throw them using opencv.

So I decided to search for another way to read frames from RTSP url, and found libValkka, which works perfectly, and knows when to throw unnecessary information. The problem is that it only works on linux OS. The UI and the client side is meant to be used on Windows OS, while the server will be running on a remote machine which runs linux OS. So, no problem with the server side, I can use libValkka there. The problem is with the client side, thats where my question comes. Is there any way to consume the frames from valkka pipeline on the server side without using any valkka api on the client side? or any other way that I have not though of?

Thanks.

elsampsa commented 2 years ago

Hi,

I'm not sure if I caught all details of your setup, but if I understood correctly:

Server:

Client/UI:

In the following I assume that the client receives H264 & does it's own decoding

If in the windows client you want to use Qt:

If you want to use a browser UI instead:

Hope this helps.

idonahum1 commented 2 years ago

Hey, sorry for the delay. Thanks for your answer.

Actually at the moment there is no server-client architecture, everything sits on the same machine, but I would like to change it.

Server:

reads streams from an RTSP camera does the server need the actual images (decoded stream) for something? say, opencv analysis, etc.? server forwards the stream - H264 or decoded bitmap images? - to the UI

Server: Yes, it reads streams from an RTSP camera, it does not need to decode them, just forward the stream to the client. The server sits on a linux machine.

Client/UI:

In a separate windoze machine Only needs to show the live stream?

Client: Sits on a windows machine, its shows the live stream and also have buttons to move the camera.

If in the windows client you want to use Qt:

I'd recommend PyQt & libVLC python bindings & visualize the video using the libVLC bindings You can arrange libValkka server to forward the original RTSP/H264 stream to the client ..or you can tell libVLC to connect directly to the original RTSP source (no need for libValkka) warning: libVLC sucks somewhat when it comes to live video streaming - especially if frames arrive in bursts.

At the moment the windows client uses PyQt, if you say that libVLC sucks when it come to live video streaming, I would like to know how to code the second option. Do you have an example?

If you want to use a browser UI instead:

You can arrange libValkka server to encapsulate the original H264 into frag-mp4 and send it through websockets to the web UI ref: https://github.com/elsampsa/websocket-mse-demo

This is also a great option, I will check it out, since in the future I would like the client to be a web UI.

Thanks a again, it helps me alot!

elsampsa commented 2 years ago

I would like to know how to code the second option. Do you have an example?

You mean libVLC python bindings connecting to the RTSP camera? Google is your friend. :)

Good luck! :)

idonahum1 commented 2 years ago

Nope, I meant this option :)

You can arrange libValkka server to forward the original RTSP/H264 stream to the client

Do you have an example for this option?

elsampsa commented 2 years ago

RTSP Server:

https://elsampsa.github.io/valkka-examples/_build/html/lesson_5.html#using-the-rtsp-server

Also of interest: heavy forking of streams & sharing it via multicast:

https://elsampsa.github.io/valkka-examples/_build/html/lesson_7.html

A word of warning about the RTSP server: https://github.com/elsampsa/valkka-core/issues/24

idonahum1 commented 2 years ago

Thank you, I read both lessons, but I want to understand what is the benefit in using RTSP server if I already consume data from an RTSP stream, the result wont be the same? RTSP -> Client compare to RTSP -> libValkka -> RTSP server -> Client, arent they equivalent?

elsampsa commented 2 years ago

Here is some ascii art:

case (1)

camera ---rtsp connection---> [ libValkka .. python code where you do analysis or whatever] 
   |
   +---another rtsp connection---> your libVLC client running in windows (or any other program that uses RTSP)

case (2)

camera ---rtsp connection---> [ libValkka .. python code where you do analysis or whatever] 
                                                            |
                                                            |
   +----------------------------  rtsp connection from libValkka   
   |
   +-------> your libVLC client running in windows (or any other program that uses RTSP)

I think (1) sufficient for your case. You might want to do something like (2) if your rtsp camera gives only a limited number of rtsp connections: in that case you just need a single rtsp connection to the camera. You might also want case(2) and just a single rtsp connection to the camera due to network topology: in case (1) you have double the bandwith coming from the camera compared to case (2).