video sending using multithreading

[julio16101]

In context, I transfer video from a camera on a raspberry pi 3, to a local computer. The local network is only used for this purpose so there is no congestion on the network, besides I am connected by cable, on the computer and on the raspberry, (LAN). The shipping is slow and I had an idea to increase the transfer using double wire delivery

Question

I have a question, to verify if it is possible or someone else has tried to do it.

In context, I transfer video from a camera on a raspberry pi 3, to a local computer. The local network is only used for this purpose so there is no congestion on the network, besides I am connected by cable, on the computer and on the raspberry, (LAN). The video quality I transfer has a size of 1920 x 1080.

Using the NetGear library I transfer the video correctly, but at an average speed of 3 FPS.

For my purposes I need more video at a higher speed, but without reducing the quality of the video or resizing it.

server = NetGear (address = '192.168.x.xxx', port = '5454', protocol = 'tcp', pattern = 1, receive_mode = False, logging = True, ** options) #Define netgear client at Server IP address .
server2 = NetGear ...... different port

Try to implement a communication using 2 threads, where in each thread you send a video frame to the computer so that in this way while sending a video frame you can also make another sending at the same time of a subsequent frame. In theory this could help me increase the amount of SPF.

But what I have observed, is that internally even if you use threads, the video sending is only done one by one, even if you have two sending objects, one or the other will wait internally for the other to finish sending to send yours.

for example, implement 3 threads to send video at the same time but with consecutive frames, executing the code for 10 seconds. Each thread could only send 10 frames, where in total for the ten seconds were 30, 3 FPS.

Each thread is constantly checking on a corresponding video stack if there is a frame to send.

while True:
    if len (queue1)> 0:
        server.send (frame)

Acknowledgment

• [*] A brief but descriptive Title of your issue.
• [*] I have searched the issues for my issue and found nothing related or helpful.
• [*] I have read the FAQ.
• [*] I have read the Wiki.

Context

I mainly want to increase the amount of video frames I receive

In context, I transfer video from a camera on a raspberry pi 3, to a local computer. The local network is only used for this purpose so there is no congestion on the network, besides I am connected by cable, on the computer and on the raspberry, (LAN). The video quality I transfer has a size of 1920 x 1080.

Your Environment

• VidGear version: 0.1.5
• Branch:
• Python version: 3.5
• pip version: current version
• Operating System and version: Ubuntu 16.04

Optional

[abhiTronix]

Using the NetGear library I transfer the video correctly, but at an average speed of 3 FPS.

@julio16101 That's painfully low, what's the pattern value you are using in your code? I normally get 10-20 FPS on my Raspberry Pi 3 on a wifi connection, something is wrong at your end. Are you using multiple Raspberry Pi at the same time?

Try to implement a communication using 2 threads, where in each thread you send a video frame to the computer so that in this way while sending a video frame you can also make another sending at the same time of a subsequent frame.

Not possible, this Will lead to packet collision. Also, there's a problem with multithreading that if threads are using the same waypoint/instance such as the same port to send data, then it leads to slower performance as compared to not using threading. In addition to this, there's a major problem called race condition in multi-threading, Which can lead to the corrupted video at the output if the wrong packet gets received at the wrong time. Hence, There are many reasons not to use multithreading for this purpose.

We are already using multi-threading at Client's end to receive frame. Therefore NetGear performance is already optimized for handling the connection. The problem is somewhere lies in your code or machine or connection.

The video quality I transfer has a size of 1920 x 1080.

You should downsize before sending it. The 1920x1080 frame is too big for your Raspberry Pi to handle due to its limited memory as at the same time it is decoding another frame, I think this is the reason for your Slower speeds.

[julio16101]

Thanks for your prompt response, I am using pattern = 1, I have seen in other questions that you recommend using pattern = 2, where you mention that it is the best for real-time transmission. I tried it in the branch of evidence, but for some reason the image is frozen, watching the record, I see that I am receiving video, but the image is the same. If I put a time.sleep (0.1) the video recovers its movement but is just as slow as with pattern = 1.

On the collision of frames, place an algorithm to prevent the reading of incorrect video frames, so that although in theory the threads sent frames in different positions, upon receiving them they were in the correct order.

Reducing the size of the video in my case is not optional, since I use it for facial recognition processes, and the quality of the video is my priority.

You should downsize before sending it. The 1920x1080 frame is too big for your Raspberry Pi to handle due to its limited memory as at the same time it is decoding another frame, I think this is the reason for your Slower speeds.

I do not know if you mean this, but I have tested the reading speed of my raspberry pi 3B +, in 1920x1080, and the reading is done at a good speed, but it does not matter if in the end the speed of network delivery is slow .

It is worth mentioning the local network, I only use it for the purpose of video transmission where I only have one raspberry and the computer where I receive the video

[abhiTronix]

On the collision of frames, place an algorithm to prevent the reading of incorrect video frames, so that although in theory the threads sent frames in different positions, upon receiving them they were in the correct order.

@julio16101 We cannot prevent network collisions as they happen in a network beyond the scope of this library. Secondly, even if we added safeguards to prevent incorrect frames, it will add to latency if wrong shows up before the next frame because of time and memory consumed in handling those incorrect frames. Currently, You can expect the best performance out of a more powerful machine on both server and client-side, and in my tests, it has reached 30~45FPS at 1080i without any problem on a wired connection.

but for some reason the image is frozen,

Definitely, Something is wrong, check your code.

I do not know if you mean this, but I have tested the reading speed of my raspberry pi 3B +, in 1920x1080, and the reading is done at a good speed, but it does not matter if in the end the speed of network delivery is slow .

You're interpreting this wrong. Raspberry Pi has limited 1GB RAM which is shared with GPU(which can take upto 128mb) and on remaining RAM, when few threads of PiGear is using to decode frame from the camera(that works without any freezing) can take up to 70~90% of that RAM, therefore there is a very little RAM left for NetGear API to work properly on the same machine. You can replace raspberry Pi with a more powerful machine with more memory, you can expect a better result.

Reducing the size of the video in my case is not optional, since I use it for facial recognition processes, and the quality of the video is my priority.

Nope, you're interpreting this wrong too, I meant downsample/downsizing image before sending and upsampling/upscaling on clients-end. Also, It's not mostly the quality of video that helps in better recognition but instead is the quantity of features that are available at the time of recognition. So a 640x480 frame can outperform a 1980x1080 frame if more features are available in that frame. In addition to this Facial recognition can be improved by training a more robust NN for recognition with the following recommendations for training your model:

• Use an image with a face that is within the recommended range of angles. The pitch should be less than 30 degrees face down and less than 45 degrees face up. The yaw should be less than 45 degrees in either direction. There is no restriction on the roll.
• Use an image of a face with both eyes open and visible.
• Use an image of a face that is not obscured or tightly cropped. The image should contain the full head and shoulders of the person. It should not be cropped to the face bounding box.
• Avoid items that block the face, such as headbands and masks.
• Use an image of a face that occupies a large proportion of the image. Images, where the face occupies a larger portion of the image, are matched with greater accuracy.
• Use color images.
• Use images with flat lighting on the face, as opposed to varied lighting such as shadows.
• Use images that have sufficient contrast with the background. A high-contrast monochrome background works well.
• Use images of faces with neutral facial expressions with mouth closed and little to no smile for applications that require high precision.
• Use images that are bright and sharp. Avoid using images that may be blurry due to subject and camera motion as much as possible. DetectFaces can be used to determine the brightness and sharpness of a face.

Using a well-trained Neural network can achieve the same or more performance in recognition on a much smaller resolution frame, according to my experience in this field.

[julio16101]

I understand the point that the raspberry is not powerful enough for this frame size.

On the bad performance in pattern = 2, use the branch of tests and with the examples that you have placed I get the problems that I mentioned before. Something that I imagine is that at first I installed pyzmq from pip and not directly. Although later uninstall and install in the manner recommended.

No, you are also misinterpreting this, I was referring to an image of decrease / reduction in size before sending and increasing / increasing the scale in the final customer. In addition, it is not primarily the quality of the video that helps better recognition, but the amount of functions available at the time of recognition.

You mean apply a resize? as with PIL use Image.resize or in cv2.resize? I have used them but in this case I am not sure that so much quality loses the image.

I do not understand what you mean by "functions available at the time of recognition", if any, you have documentation that may be useful to understand that point. Also which options to increase and reduce the size are more optimal from your experience in these cases? I would greatly appreciate it

In your other advice I already apply the majority in my project. I am using dlib for facial detection and a knn for face prediction.

[abhiTronix]

You mean apply a resize? as with PIL use Image.resize or in cv2.resize? I have used them but in this case I am not sure that so much quality loses the image.

@julio16101 Yes, i meant cv2.resize, It resizes the image to a given size. and interpolation can be selected as cv2.INTER_LANCZOS4 to obtain best interpolation results. and also it do not reduce quality but instead shrinks the image so that it consumes less memory, hence faster transmission.

I do not understand what you mean by "functions available at the time of recognition"

See clearly, I wrote "features that are available at the time of recognition." its features not functions. In machine learning and pattern recognition, a FEATURE is an individual measurable property or characteristic of a phenomenon being observed. Choosing informative, discriminating and independent features is a crucial step for effective algorithms in pattern recognition, classification and regression. For example in dlib, Faces are encoded and represented as 128-dimension points in space (called face vectors) which can also be called as features of that face.

I am using dlib for facial detection and a knn for face prediction.

That works too on smaller images, you don't need full hd 1920x1080 images for detection and recognition. See this document.

Also, I'm closing this issue now and marking invalid as it is not related to vidgear.

[abhiTronix]

Thanks for your prompt response, I am using pattern = 1, I have seen in other questions that you recommend using pattern = 2, where you mention that it is the best for real-time transmission. I tried it in the branch of evidence, but for some reason the image is frozen, watching the record, I see that I am receiving video, but the image is the same.

@julio16101 I've discovered a bug within NetGear API which causes a slow response time with Subscriber/Publisher pattern and appeared as frozen frame at your end. I'll inform you here when that bug is addressed and fixed in the upcoming PR. You can expect improved performance after that PR even with Raspberry Pi. :smiley:

[julio16101]

I thought I was the only one who had that problem, yes, it happens that in that way the image is frozen, a solution I found was to add a delay between frame shipments. I will be watching.

[abhiTronix]

a solution I found was to add a delay between frame shipments. I will be watching.

That's can be a temporary solution but non-optimal for the long run as it adds more latency. I'm currently experimenting with different settings for getting the best results on Pi. So stay tuned.

[abhiTronix]

Hello @julio16101, Kindly install the related PR(#76) as follows:

git clone https://github.com/abhiTronix/vidgear.git
cd vidgear
git checkout development
sudo pip3 install .
cd

and test if it works as you expected or not? and then revert here. Good luck!

[abhiTronix]

@julio16101 I'm hoping you to test this PR as soon as possible as it will be going to be merged by tomorrow.

[julio16101]

I will try to try in the next few hours

[julio16101]

I have already tried the development branch. The Pub / Sub protocol is already working correctly and the frozen screen problem no longer remains. As I am working with a large image size, I do not see an increase in image transfer speed.

[abhiTronix]

@julio16101 can you set pattern=1 i.e. use REQ/REP pattern instead of PUB/SUB? Does that make any difference in performance with large images?

[julio16101]

Because my camera a logitech c920 stopped working due to the cable, I am occupying another camera with the same resolution but less MP, at first glance I can say that, yes, it shows a slight increase using as pattern = 1, no I could assure you until I run in my full development environment.

But yes, a slight reduction in transmission delay is visible to the naked eye.

[julio16101]

FPS in a resolution of 1920 * 1080 is around 3.35. this in pattern = 1 Previously I obtained a speed of 2.8-3.1 in FPS using the previous version of netgear. All this without applying image compression operations.

In the minimum example

[abhiTronix]

FPS in a resolution of 1920 * 1080 is around 3.35. this in pattern = 1 Previously I obtained a speed of 2.8-3.1 in FPS using the previous version of netgear. All this without applying image compression operations.

In the minimum example

@julio16101 Can you this test code using different machines, i.e. Server at Raspberry Pi and Client at your Laptop? This setup must get at least 10-15 fps at least with pattern=1 as I'm getting around 15-20 fps (with my pi and laptop at 720p) and 20-25 fps (with my pi and laptop at 480p) over WiFi.

[julio16101]

I'm testing with a resolution of 1920 * 1080 with 15 MP

[abhiTronix]

I'm testing with a resolution of 1920 * 1080 with 15 MP

@julio16101 What framerate(fps) are you getting?

[julio16101]

3.3 FPS

[abhiTronix]

3.3 FPS

@julio16101 using different machines, i.e. Server at Raspberry Pi and Client at your Laptop? Or you running both on the same machine but different terminals?

[julio16101]

I get the video from a Raspberry pi 3 B + in a wired local connection, the video I receive on a computer with ubuntu operating system

[abhiTronix]

@julio16101 How are you calculating FPS? Can you use the following test code at the Client's end(i.e Ubuntu computer)? (:warning: Remember to change IP address, port address and pattern value with yours)

Client's End Code (For Measuring FPS)

# import libraries
from vidgear.gears import NetGear
import cv2
import time

class FPS:
    """
    Class to calculate FPS based on time.time python module
    """
    def __init__(self):
        #initiating FPS class and its variable
        self._start = 0
        self._end = 0
        self._numFrames = 0

    def start(self):
        #start timer
        self._start = time.time()
        return self

    def stop(self):
        #stop timer
        self._end = time.time()

    def update(self):
        #calculate frames
        self._numFrames += 1

    def total_time_elapsed(self):
        #return total time elaspsed = start time - end time(in sec)
        if (self._end <= self._start):
            self._end = time.time()

        return (self._end - self._start)%60

    def fps(self):
        #return FPS
        return self._numFrames / self.total_time_elapsed()

options = {'flag' : 0, 'copy' : False, 'track' : False}

#change following IP address '192.168.x.xxx', port address and pattern value with yours
client = NetGear(address = '192.168.x.xxx', port = '5454', protocol = 'tcp',  pattern = 0, receive_mode = True, logging = True, **options) #Define netgear client at Server IP address.

fps = FPS().start()

# infinite loop
while True:
    # receive frames from network
    frame = client.recv()

    # check if frame is None
    if frame is None:
        #if True break the infinite loop
        break

    # do something with frame here
    fps.update()

    # Show output window
    cv2.imshow("Output Frame", frame)

    key = cv2.waitKey(1) & 0xFF
    # check for 'q' key-press
    if key == ord("q"):
        #if 'q' key-pressed break out
        break

# close output window
cv2.destroyAllWindows()
#get FPS
fps.stop()
#then print output
print("[LOG] total elasped time: {:.2f}".format(fps.total_time_elapsed()))
print("[LOG] approx. FPS: {:.2f}".format(fps.fps()))
# safely close client
client.close()

Then paste the complete terminal log here. Goodluck!

[julio16101]

I will try tomorrow early in the morning, at this moment I do not have access to my work area. I'll keep you informed

[julio16101]

[LOG]: ZMQ Security Mechanism is disabled for this connection! [LOG]: Threaded Queue Mode is enabled by default for NetGear. [LOG]: Successfully Binded to address: tcp://192.168.0.107:5454. [LOG]: Multi-threaded Receive Mode is enabled Successfully! [LOG]: Device Unique ID is 861A5. [LOG]: Receive Mode is activated successfully! [LOG] total elasped time: 38.30 [LOG] approx. FPS: 3.34

[LOG]: Terminating various Receive Mode Processes

[abhiTronix]

@julio16101 That's interesting, Is still wondering what is causing this slower performance at your end. Can you resize the frame to 1280x720 before sending, i.e using frame = imutils.resize(frame, width=1280) by importing imutils module in your code. What's the framerate you're getting? Also try, 640x480 size also as above.

[julio16101]

If I reduce the resolution the speed increases.

But in the current size that I use, it's the speed I get

[abhiTronix]

@julio16101 Thanks alot for this insight, I think the conclusion is that the Raspberry Pi's hardware is limiting the performance. If you get a chance with any better hardware machine with nice GPU, kindly redo this test and if still performance is low, then inform me here. Goodluck!

[abhiTronix]

I'll close this issue when the PR is merged. I'm marking this issue resolved for now.

[julio16101]

(1280) [LOG]: ZMQ Security Mechanism is disabled for this connection! [LOG]: Threaded Queue Mode is enabled by default for NetGear. [LOG]: Successfully Binded to address: tcp://192.168.0.107:5454. [LOG]: Multi-threaded Receive Mode is enabled Successfully! [LOG]: Device Unique ID is FCD73. [LOG]: Receive Mode is activated successfully! [LOG] total elasped time: 39.48 [LOG] approx. FPS: 12.21

[julio16101]

Using your code I have noticed that the longer the system is running, the fps increase, but the speed at a glance remains.

The formula I occupy is, FPS: 1 / (Tf - Ti)

[abhiTronix]

@julio16101

Using your code I have noticed that the longer the system is running, the fps increase, but the speed at a glance remains.

I don't know but I found that on the internet. Even imutils library uses similar.

The formula I occupy is, FPS: 1 / (Tf - Ti)

no, The actual formula is (fps = total frames/ total time spent)

[julio16101]

I have always calculated it that way and it matches the frame count that I make in a list in an amount of time.

[abhiTronix]

@julio16101 Wow, we both are true, check this out https://stackoverflow.com/a/43761675/10158117

[abhiTronix]

Successfully resolved and merged in commit: e918d13

[abhiTronix]

This issue will be resolved by re-implementing NetGear with powerful Asyncio asynchronous programming in the upcoming PR in order to achieve faster and lag-free video streaming over the network.

[abhiTronix]

Successfully resolved and merged in commit e421975