Closed PrashantSaikia closed 3 years ago
Thanks for opening this issue, a maintainer will get back to you shortly!
Say I want two of these feeds to be displayed side by side - one for the webcam stream, and one for streaming another video stream (like a pre-downloaded mp4 video, etc).
@PrashantSaikia Good question, I was hoping someone to ask this. Thanks to modular stucture of WebGear, the simplest trick here is to run one source generator directly through config["generator"]
and other source through different Streaming Response and Route. In this way you can easily create any number of source generators over different routes.
For example in below example, I will run first source "foo1.mp4" from my_frame_producer1
directly through web.config["generator"] = my_frame_producer1
as given in docs. But for second source "foo2.mp4", I'll create a new custom_video_response
and assign it to a completely different route i.e. /video2
, thereby you can easily access it on different URL(http://localhost:8000/video2
) after running this example:
# import necessary libs
import uvicorn, asyncio, cv2
from vidgear.gears.asyncio import WebGear
from vidgear.gears.asyncio.helper import reducer
from starlette.responses import StreamingResponse
from starlette.routing import Route
# initialize WebGear app without any source
web = WebGear(logging=True)
# create your own custom frame producer
async def my_frame_producer1():
# !!! define your own video source here !!!
# Open any video stream such as "foo1.mp4"
stream = cv2.VideoCapture("foo1.mp4")
# loop over frames
while True:
# read frame from provided source
(grabbed, frame) = stream.read()
# break if NoneType
if not grabbed:
break
# do something with your OpenCV frame here
# reducer frames size if you want more performance otherwise comment this line
frame = await reducer(frame, percentage=30) # reduce frame by 30%
# handle JPEG encoding
encodedImage = cv2.imencode(".jpg", frame)[1].tobytes()
# yield frame in byte format
yield (b"--frame\r\nContent-Type:video/jpeg2000\r\n\r\n" + encodedImage + b"\r\n")
await asyncio.sleep(0.00001)
# close stream
stream.release()
# create your own custom frame producer
async def my_frame_producer2():
# !!! define your own video source here !!!
# Open any video stream such as "foo2.mp4"
stream = cv2.VideoCapture("foo2.mp4")
# loop over frames
while True:
# read frame from provided source
(grabbed, frame) = stream.read()
# break if NoneType
if not grabbed:
break
# do something with your OpenCV frame here
# reducer frames size if you want more performance otherwise comment this line
frame = await reducer(frame, percentage=30) # reduce frame by 30%
# handle JPEG encoding
encodedImage = cv2.imencode(".jpg", frame)[1].tobytes()
# yield frame in byte format
yield (b"--frame\r\nContent-Type:video/jpeg2000\r\n\r\n" + encodedImage + b"\r\n")
await asyncio.sleep(0.00001)
# close stream
stream.release()
async def custom_video_response(scope):
"""
Return a async video streaming response for `my_frame_producer2` generator
"""
assert scope["type"] in ["http", "https"]
await asyncio.sleep(0.00001)
return StreamingResponse(
my_frame_producer2(),
media_type="multipart/x-mixed-replace; boundary=frame",
)
# add your custom frame producer to config
web.config["generator"] = my_frame_producer1
# append new route i.e. new custom route with custom response
web.routes.append(
Route("/video2", endpoint=custom_video_response)
)
# run this app on Uvicorn server at address http://localhost:8000/
uvicorn.run(web(), host="localhost", port=8000)
# close app safely
web.shutdown()
So after running this example you'll have two Video URL sources, one is default http://localhost:8000/video
and other one is http://localhost:8000/video2
, so you can change/modify your index.html
HTML file to point to these two Video URL sources:
Goodluck!!!
@PrashantSaikia Please test this example as sson as possible and let us know how it went.
Marked solved, as working as intended. Feel free to test it and provide valuable feedback when possible.
Hi there. I'm a little unclear on where do I need to put this index.html, or how to run it. Do I need to run the code you wrote above, and then separately run this index.html, or something like that? If so, its getting a bit complicated. Also, I actually have done this with websockets; however, after doing pose estimation on the video feeds, the output rendered is very choppy (even with the answer marked as the solution). So, the whole reason why I was trying out Webgear was in the hope that I might get better FPS with Webgear. Do you think that's going to be the case?
Hi there. I'm a little unclear on where do I need to put this index.html, or how to run it. Do I need to run the code you wrote above, and then separately run this index.html
@PrashantSaikia You need to look at docs on how WebGear works. WebGear dynamically checks for HTML, CSS, JS files at runtime with its Auto-Generation workflow. So you need to change HTML file generated at Default Location to your use case.
So, the whole reason why I was trying out Webgear was in the hope that I might get better FPS with Webgear. Do you think that's going to be the case?
@PrashantSaikia It depends upon how fast source is outputting the frames. If source itself is slow, then WebGear can't do much. See WebGear's Performance Enhancements. Also, you can try WebGear_RTC that utilizes much faster WebRTC technology in the similar way.
Ok, so I saved the index.html as follows:
{% extends "base.html" %}
{% block content %}
<h1 class="glow">WebGear Video Feed</h1>
<div class="rows">
<video id="myvideo" width="320" height="240" controls style="background:black">
<source class="active" src="http://localhost:8000/video" type="video/mp4" />
<source src="http://localhost:8000/video2" type="video/mp4" />
</video>
</div>
{% endblock %}
and used options
to point to the location where it is saved (which I've kept as ./
for convenience as of now). After now running your code, I can see the two html files in the folder, but it doesn't display the two videos side by side - it still shows only one, which I guess probably means they are overlapping, which I pointed out in my original post.
I can see the two html files in the folder, but it doesn't display the two videos side by side
@PrashantSaikia What two html files?
index.html and base.html
(which I've kept as ./ for convenience as of now)
@PrashantSaikia Kindly enable debugging(logging=True
) and Paste the terminal log here. In my opinion ./
is not working but still using old files located at .vidgear
at your home directory.
index.html and base.html
@PrashantSaikia That is wrong, it should not be two files but it must also contain 404.html
and 500.html
, otherwise WebGear triggers Auto-Generation process that will overwrite old files.
Oh ok, I copied the 404.html and 500.html files too into the folder. Still no difference in result. Could you please correct my index.html and tell me exactly what to do after that? I just want to see if there's any improvement in FPS or not.
Oh ok, I copied the 404.html and 500.html files too into the folder. Still no difference in result.
@PrashantSaikia Are you able to achieve two streams on your browser? You're not helping me with debug output, I can't help with your problem.
@PrashantSaikia Just paste whatever is in your terminal after running the python script.
No, I'm not able to achieve two streams in my browser. Here's what is in my terminal after running the code:
10:28:54 :: Helper Asyncio :: DEBUG :: Directory already exists at `/Users/username/Documents/OpenCV_Project/.vidgear/webgear`
10:28:54 :: Helper Asyncio :: DEBUG :: Directory already exists at `/Users/username/Documents/OpenCV_Project/.vidgear/webgear/static`
10:28:54 :: Helper Asyncio :: DEBUG :: Directory already exists at `/Users/username/Documents/OpenCV_Project/.vidgear/webgear/templates`
10:28:54 :: Helper Asyncio :: DEBUG :: Directory already exists at `/Users/username/Documents/OpenCV_Project/.vidgear/webgear/static/js`
10:28:54 :: Helper Asyncio :: DEBUG :: Directory already exists at `/Users/username/Documents/OpenCV_Project/.vidgear/webgear/static/css`
10:28:54 :: Helper Asyncio :: DEBUG :: Directory already exists at `/Users/username/Documents/OpenCV_Project/.vidgear/webgear/static/img`
10:28:54 :: Helper Asyncio :: DEBUG :: Found valid WebGear data-files successfully.
10:28:54 :: WebGear :: DEBUG :: `/Users/username/Documents/OpenCV_Project/.vidgear/webgear` is the default location for saving WebGear data-files.
10:28:54 :: WebGear :: DEBUG :: Setting params:: Size Reduction:20%, JPEG quality:90%, JPEG optimizations:False, JPEG progressive:False.
10:28:54 :: WebGear :: WARNING :: Given source is of NoneType!
10:28:54 :: WebGear :: DEBUG :: Initiating Video Streaming.
10:28:54 :: WebGear :: DEBUG :: Running Starlette application.
INFO: Started server process [67988]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://localhost:8000 (Press CTRL+C to quit)
INFO: ::1:51917 - "GET / HTTP/1.1" 200 OK
INFO: ::1:51918 - "GET /video HTTP/1.1" 200 OK
INFO: ::1:51917 - "GET /static/css/custom.css HTTP/1.1" 304 Not Modified
INFO: ::1:51919 - "GET /static/js/custom.js HTTP/1.1" 304 Not Modified
INFO: ::1:51917 - "GET /video HTTP/1.1" 200 OK
10:28:54 :: WebGear :: DEBUG ::
/Users/username/Documents/OpenCV_Project/.vidgear/webgear
is the default location for saving WebGear data-files.
@PrashantSaikia Ok, the default location have changed successfully.
INFO: Uvicorn running on http://localhost:8000 (Press CTRL+C to quit) INFO: ::1:51917 - "GET / HTTP/1.1" 200 OK INFO: ::1:51918 - "GET /video HTTP/1.1" 200 OK INFO: ::1:51917 - "GET /static/css/custom.css HTTP/1.1" 304 Not Modified INFO: ::1:51919 - "GET /static/js/custom.js HTTP/1.1" 304 Not Modified INFO: ::1:51917 - "GET /video HTTP/1.1" 200 OK
But Still no /video2
here. Did you changed index html? Also what happens when you visit http://localhost:8000/video and http://localhost:8000/video2 (open both in different tabs), did you see output simuntaneously on both?
Did you changed index html?
Yes, I copy-pasted my index.html
above. Pasting it again here:
{% extends "base.html" %}
{% block content %}
<h1 class="glow">WebGear Video Feed</h1>
<div class="rows">
<!-- <div id="imgcontainer" onclick="toggleFullScreen('imgcontainer')"><img src="/video" alt="Feed" title="click to go fullscreen"></div> -->
<video id="myvideo" width="320" height="240" controls style="background:black">
<source class="active" src="http://localhost:8000/video" type="video/mp4" />
<source class="active" src="http://localhost:8000/video2" type="video/mp4" />
</video>
</div>
{% endblock %}
Also what happens when you visit http://localhost:8000/video and http://localhost:8000/video2, did you see output?
Yes, I checked. When I give the video path as webcam for both the streams, it doesn't display anything on the browser. So I changed the video paths to two pre-downloaded mp4 videos. After that, when I run the script and visit http://localhost:8000/video
and http://localhost:8000/video2
, it downloads the two video files, but without any extension. So they get downloaded as Document files, and even though they are the same size as my mp4 files (15-20 MB), they appear as a static image upon opening them and can't be played as video.
{% extends "base.html" %} {% block content %} <h1 class="glow">WebGear Video Feed</h1> <div class="rows"> <!-- <div id="imgcontainer" onclick="toggleFullScreen('imgcontainer')"><img src="/video" alt="Feed" title="click to go fullscreen"></div> --> <video id="myvideo" width="320" height="240" controls style="background:black"> <source class="active" src="http://localhost:8000/video" type="video/mp4" /> <source class="active" src="http://localhost:8000/video2" type="video/mp4" /> </video> </div> {% endblock %}
@PrashantSaikia Also your html code is wrong. Source is of img
type not video
. So change it to following:
{% block content %}
<h1 class="glow">WebGear Video Feed</h1>
<div class="rows">
<img src="/video" alt="Feed1">
<img src="/video2" alt="Feed2">
</div>
{% endblock %}
Ok, done. Still see only one stream in the browser:
Yes, I checked. When I give the video path as webcam for both the streams, it doesn't display anything on the browser. So I changed the video paths to two pre-downloaded mp4 videos. After that, when I run the script and visit http://localhost:8000/video and http://localhost:8000/video2, it downloads the two video files, but without any extension. So they get downloaded as Document files, and even though they are the same size as my mp4 files (15-20 MB), they appear as a static image upon opening them and can't be played as video.
@PrashantSaikia That's normal behavior since WebGear uses motion-jpeg, that display images frame by frame and not a valid video container.
Ok, done. Still see only one stream in the browser:
@PrashantSaikia What source are you using? I'm getting correct out put here:
@PrashantSaikia What source are you using? I'm getting correct out put here:
I am using pre-downloaded videos, the same ones from here.
Also, like I mentioned before, if I give the video path as 0
(i.e., webcam), it doesn't display anything in the browser.
@PrashantSaikia Ok I'm uploading complete Zip file here with everything working, Please wait.
Also, like I mentioned before, if I give the video path as 0 (i.e., webcam),
@PrashantSaikia Are you sure? OpenCV works with camera out of the box. Try -1
integer value at input.
@PrashantSaikia Are you sure? OpenCV works with camera out of the box. Try
-1
integer value at input.
Ok, tried this as well just now. Now there's a "?" that appears in the div where the video is supposed to display. And the terminal output also says that the path is wrong:
OpenCV: out device of bound (0-0): -1
OpenCV: camera failed to properly initialize!
So I guess 0 is the correct path for webcam. When I give 0, there's nothing that appears in the video placeholder/div - no question mark, no video.
@PrashantSaikia See its working:
Just do python test_example.py
inside extracted folder to run it.
So I guess 0 is the correct path for webcam. When I give 0, there's nothing that appears in the video placeholder/div - no question mark, no video.
@PrashantSaikia Your code must be wrong, just use mine with 0
value. Goodluck!
And here is the working zip file:
Ok, now it shows the two videos one after the other. Upon refreshing the page though, the two videos play side by side as desired. And also, the videos are playing at a faster speed, looks like 2-3x from eyeballing.
Here, I did a screen recording:
And giving the webcam still produces blank output:
I guess this might be a Windows vs MacOS issue then, since you're getting the desired output but I'm not.
I guess this might be a Windows vs MacOS issue then, since you're getting the desired output but I'm not.
I'm also on windows. Try this example:
# import required libraries
from vidgear.gears import CamGear
import cv2
# Open suitable video stream, such as webcam on first index(i.e. 0)
stream = CamGear(source=0).start()
# loop over
while True:
# read frames from stream
frame = stream.read()
# check for frame if Nonetype
if frame is None:
break
# {do something with the frame here}
# Show output window
cv2.imshow("Output", frame)
# check for 'q' key if pressed
key = cv2.waitKey(1) & 0xFF
if key == ord("q"):
break
# close output window
cv2.destroyAllWindows()
# safely close video stream
stream.stop()
What do you see?
And also, the videos are playing at a faster speed, looks like 2-3x from eyeballing.
@PrashantSaikia You can play with WebGear's Performance Enhancements for further performance boost. Also, you can try WebGear_RTC that utilizes much faster WebRTC technology in the similar way.
I'm also on windows.
Yeah, that's why I said, because I'm on Mac.
What do you see?
I can see the webcam feed. When displaying a single video, its not a problem. In fact, I had already ported my pose recognition module into your example code before I made this post, and it works great. Its just that making two videos play side by side is proving difficult.
Its just that making two videos play side by side is proving difficult.
@PrashantSaikia This is making no sense. If you're having no problem with single video, then you must have no problem with any number of video. Can you update your vidgear to latest as follows:
# clone the repository and get inside
git clone https://github.com/abhiTronix/vidgear.git && cd vidgear
# checkout the latest testing branch
git checkout testing
# install with asyncio support
pip install .[asyncio]
and then try.
@PrashantSaikia Also, are you using 0
index this time? Double check it is 0
and not -1
. Also problem with OpenCV is that it sometimes doesn't display the actual problem.
@PrashantSaikia Also, are you using
0
index this time? Double check it is0
and not-1
.
Yes, I'm using 0. I never use -1, except now to test it.
Also problem with OpenCV is that it sometimes doesn't display the actual problem.
Yes, that's true unfortunately.
Who knew rendering the two videos on the browser instead of on two popup windows as OpenCV does would be such a time consuming and energy-intensive task. I just wish OpenCV had an inbuilt function to render the results on browser, sigh.
Hi, i tested the zip file you shared and it also works with two webcams.
Now, instead of using WebGear, i tried to use WebGear_RTC but I'm keep getting this error when i visit http://localhost:8000/video2.
TypeError: 'coroutine' object is not an iterator
This is the debug and full error
This is my code (Since WebGear and WebGear_RTC very similar, i didn't change a lot of stuff from your zip file )
#import libs
import uvicorn, asyncio, cv2
from starlette.routing import Route
from vidgear.gears.asyncio import WebGear_RTC
from vidgear.gears.asyncio.helper import reducer
from starlette.responses import StreamingResponse
from aiortc import VideoStreamTrack
from av import VideoFrame
options = {"custom_data_location": "./"}
#initialize WebGear app without any source
web = WebGear_RTC(logging=True, **options)
class Custom_RTCServer(VideoStreamTrack):
"""
Custom Media Server using OpenCV, an inherit-class
to aiortc's VideoStreamTrack.
"""
def __init__(self, source=None):
# don't forget this line!
super().__init__()
# initialize global params
self.stream = cv2.VideoCapture(0)
self.stream2 = cv2.VideoCapture(1)
async def recv(self):
"""
A coroutine function that yields `av.frame.Frame`.
"""
# don't forget this function!!!
# get next timestamp
pts, time_base = await self.next_timestamp()
# read video frame
(grabbed, frame) = self.stream.read()
# if NoneType
if not grabbed:
return None
# reducer frames size if you want more performance otherwise comment this line
frame = await reducer(frame, percentage=30) # reduce frame by 30%
# contruct `av.frame.Frame` from `numpy.nd.array`
av_frame = VideoFrame.from_ndarray(frame, format="bgr24")
av_frame.pts = pts
av_frame.time_base = time_base
# return `av.frame.Frame`
return av_frame
async def recv2(self):
"""
A coroutine function that yields `av.frame.Frame`.
"""
# don't forget this function!!!
# get next timestamp
pts, time_base = await self.next_timestamp()
# read video frame
(grabbed, frame) = self.stream2.read()
# if NoneType
if not grabbed:
return None
# reducer frames size if you want more performance otherwise comment this line
frame = await reducer(frame, percentage=30) # reduce frame by 30%
# contruct `av.frame.Frame` from `numpy.nd.array`
av_frame = VideoFrame.from_ndarray(frame, format="bgr24")
av_frame.pts = pts
av_frame.time_base = time_base
#return `av.frame.Frame`
return av_frame
async def custom_video_response(self, scope):
"""
Return a async video streaming response for `my_frame_producer2` generator
"""
assert scope["type"] in ["http", "https"]
await asyncio.sleep(0.00001)
return StreamingResponse(
self.recv2(),
media_type="multipart/x-mixed-replace; boundary=frame",
)
def terminate(self):
"""
Gracefully terminates VideoGear stream
"""
#don't forget this function!!!
#terminate
if not (self.stream is None):
self.stream.release()
self.stream = None
if not (self.stream2 is None):
self.stream2.release()
self.stream2 = None
Object = Custom_RTCServer()
web.config["server"] = Custom_RTCServer()
#append new route i.e. new custom route with custom response
web.routes.append(
Route("/video2", endpoint=Object.custom_video_response)
)
#run this app on Uvicorn server at address http://localhost:8000/
uvicorn.run(web(), host="localhost", port=8000)
#close app safely
web.shutdown()
What I'm trying to do is use WebGear_RTC to stream two cameras at the same time. Thanks
@ahmedosman2001 It will not work by simply changing python code alone in WebGear_RTC
since its backend aiortc
library is pretty complex and need both frontend javascript
and backend python
code to be altered simuntaneously and synchronized properly. Since javascripting is not forte, so I can't really help you, but you can look into this issue: https://github.com/aiortc/aiortc/issues/382
However, you can do one more thing, that you can concatenate two frames horizontally using numpy
and then send it to peer, as follows:
# import necessary libs
import uvicorn, asyncio, cv2
import numpy as np
from av import VideoFrame
from aiortc import VideoStreamTrack
from vidgear.gears.asyncio import WebGear_RTC
from vidgear.gears.asyncio.helper import reducer
# initialize WebGear_RTC app without any source
web = WebGear_RTC(logging=True)
def get_conc_frame(frame1, frame2):
h1, w1 = frame1.shape[:2]
h2, w2 = frame2.shape[:2]
# create empty matrix
vis = np.zeros((max(h1, h2), w1 + w2, 3), np.uint8)
# combine 2 frames
vis[:h1, :w1, :3] = frame1
vis[:h2, w1 : w1 + w2, :3] = frame2
return vis
# create your own Bare-Minimum Custom Media Server
class Custom_RTCServer(VideoStreamTrack):
"""
Custom Media Server using OpenCV, an inherit-class
to aiortc's VideoStreamTrack.
"""
def __init__(self, source1=None, source2=None):
# don't forget this line!
super().__init__()
# check is source are provided
if source1 is None or source2 is None:
raise ValueError("Provide both source")
# initialize global params
# define both source here
self.stream1 = cv2.VideoCapture(source1)
self.stream2 = cv2.VideoCapture(source2)
async def recv(self):
"""
A coroutine function that yields `av.frame.Frame`.
"""
# don't forget this function!!!
# get next timestamp
pts, time_base = await self.next_timestamp()
# read video frame
(grabbed1, frame1) = self.stream1.read()
(grabbed2, frame2) = self.stream2.read()
# if NoneType
if not grabbed1 or not grabbed2:
return None
# concatenate frame
frame = get_conc_frame(frame1, frame2)
# reducer frames size if you want more performance otherwise comment this line
frame = await reducer(frame, percentage=30) # reduce frame by 30%
# contruct `av.frame.Frame` from `numpy.nd.array`
av_frame = VideoFrame.from_ndarray(frame, format="bgr24")
av_frame.pts = pts
av_frame.time_base = time_base
# return `av.frame.Frame`
return av_frame
def terminate(self):
"""
Gracefully terminates VideoGear stream
"""
# don't forget this function!!!
# terminate
if not (self.stream1 is None):
self.stream1.release()
self.stream1 = None
if not (self.stream2 is None):
self.stream2.release()
self.stream2 = None
# assign your custom media server to config with both adequate sources (for e.g. foo1.mp4 and foo2.mp4)
web.config["server"] = Custom_RTCServer(source1="foo.mp4", source2="foo.mp4")
# run this app on Uvicorn server at address http://localhost:8000/
uvicorn.run(web(), host="localhost", port=8000)
# close app safely
web.shutdown()
@ahmedosman2001
Just do python test_example.py
inside extracted folder to run it.
I tested earlier and it works fine when streaming video file. However, i couldn't make it work for two webcams. For example, when the sources is changed from foo.mp4 to webcam (0 and 1), it just displays black screen. Thanks.
@ahmedosman2001 It's working fine on my end. And If it works with video files, webcam is no exception. Sort this out yourself.
@PrashantSaikia @ahmedosman2001 This might be error why two cameras are not working with OpenCV: https://stackoverflow.com/questions/29664399/capturing-video-from-two-cameras-in-opencv-at-once
This solution would be to use multithreaded CamGear/VideoGear instead of OpenCV in your code. goodluck!
Let's take the example code here that displays the webcam feed using OpenCV. Say I want two of these feeds to be displayed side by side - one for the webcam stream, and one for streaming another video stream (like a pre-downloaded mp4 video, etc). How would I do that? I'm guessing I'll need to have two async custom frame producers. But I tried it the following way:
and one overlaps the other. So, I'm guessing there should be a way to give the location coordinates to display the stream. That should solve the overlap issue. How do I do that?