search

opencv / opencv

Open Source Computer Vision Library
https://opencv.org
Apache License 2.0
75.95k stars 55.61k forks source link

Camera is very slow to open when using the MSMF VideoCapture backend #17687

Closed urbenlegend closed 2 years ago

urbenlegend commented 3 years ago
System information (version)
Detailed description

I am trying to open my Logitech HD Pro C910 USB webcam using OpenCV 4.3 in my Java app. If I use CAP_MSMF (or CAP_ANY), it takes several minutes for the camera to open up, delaying my application start time. CAP_DSHOW does not exhibit this behavior. The Python OpenCV bindings is also experiencing slow open times with CAP_MSMF.

I want to use MSMF for the GPU acceleration, but I can't use it with these startup times. Is there a way around this issue?

This issue does not show up when running my code on my Surface Pro 7 with its integrated webcam.

Steps to reproduce

Just call VideoCapture.open(0, CAP_MSMF) using certain webcams.

Issue submission checklist
alalek commented 3 years ago

Do you observe similar problem with other MSMF applications? If yes, then it is better to report problem to MSMF (w/o extra dependency on OpenCV).

it takes several minutes for the camera to open up

Can't reproduce with other internal and external USB cameras.

Try to profile your application for collecting details. For example, using VTune (it is free since 2018 Q1)

urbenlegend commented 3 years ago

Is the Windows 10 Camera app a MSMF application? Do you know of a good app I can test with? Every other camera app seems to work fine. I'll look into profiling, but I am not really doing anything in my app besides just calling VideoCapture.open(0).

mshabunin commented 3 years ago

Where did you get OpenCV? Did you try older version? Are your graphics drivers up-to-date?

urbenlegend commented 3 years ago

On the Java side, I tried two separate libraries, one offering the official OpenCV Java API and one that's a wrapper around the C++ version. Both exhibit the same issues.

The issue happens on the Python side as well. I installed via pip install opencv-python which is at version 4.2.0.34

Even just running a simple Python OpenCV demo app exhibits this behavior.

import cv2

cap = cv2.VideoCapture(0)
print("Test")

while(True):
    # Capture frame-by-frame
    ret, frame = cap.read()

    # Our operations on the frame come here
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

    # Display the resulting frame
    cv2.imshow('frame',gray)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

# When everything done, release the capture
cap.release()
cv2.destroyAllWindows()

"Test" takes forever to print out when I launch the script, indicating that it is stuck at the cv2.VideoCapture(0) line.

I have not tried an older version, as I was under the impression that MSMF support wasn't as fleshed out in the older versions. My graphics drivers are up to date. I have a AMD Vega 64 card running version 20.5.1.

mshabunin commented 3 years ago

MSMF worked in older versions. There was some refactoring in 4.3.0, so it would be interesting to check if the issue is reproduced in previous release. At least one fix is waiting to be released soon, it can be related to your issue. You can try to build latest version from master or 3.4 branch and check it too.

urbenlegend commented 3 years ago

I tested with opencv-python 3.4.9.33 and the issue still occurs. I took a look at that fix. Are you sure that's related? I am not loading from an MP4 file. I am opening up a webcam.

mshabunin commented 3 years ago

No, I'm not sure, but this code is shared between file and camera readers.

AugustinMohr commented 3 years ago

Hi, I have the same problem, and I was wondering if you found a solution ?

urbenlegend commented 3 years ago

I did not find a solution to it. I think it may be webcam specific. This is a 10 year old webcam after all. I ended up just adding an option in my application to allow the user to toggle between DSHOW and MSMF.

OnlineReserveren commented 3 years ago

I have a brand new logitech camera om windows 10 and am expiriencing the same problem. I also ran the app on my laptop with a build in camera. Startup time is also in the order of minutes. I didn't have a change yet to profile the issue since I need to get the app logic working.

cyberpsyche commented 3 years ago

I have the same problem here, either on a laptop computer or a desktop one, the issue remains. I use Window 10 , opencv-4.4.0, usb webcam with desktop computer and on-board webcam with laptop. The issue remains. How to speed up the start up process ???

mshabunin commented 3 years ago

Please try to run native MediaFoundation camera samples and check camera initialization delay, for example this one: https://github.com/microsoft/Windows-classic-samples/tree/master/Samples/Win7Samples/multimedia/mediafoundation/MFCaptureD3D (you might need to disable "Treat warnings as error" compiler option).

vanc commented 3 years ago

The same slow open issue happened to me as well. Tried the MFCaptureD3D sample and the camera opened instantly.

So the issue seems to be on the opencv side.

I am using Python 3.8.7 downloaded from the official Python web site. Installed OpenCV via pip. The version is 4.4.0.

vanc commented 3 years ago

Just as mentioned in the original report, with CAP_DSHOW, the camera opens instantly.

SilenSang commented 3 years ago

Use OpenCV 340, instead of 430.

gillerman commented 3 years ago

Has anyone tried this: OpenCV VideoCapture lag due to the capture buffer, OpenCV Solution​​ According to this source, you can set the buffersize of a cv::VideoCapture object. cv::VideoCapture cap; cap. set(CV_CAP_PROP_BUFFERSIZE, 3); // internal buffer will now store only 3 frames // rest of your code How to set cv2.VideoCapture() image size in Python Use cv2.CAP_PROP_FRAME_WIDTH and cv2.CAP_PROP_FRAME_HEIGHT in order to tell OpenCV which image size you would like. import cv2 video_capture = cv2.VideoCapture(0) # Check success if not video_capture.isOpened(): raise Exception("Could not open video device") # Set properties.

zhaixing commented 3 years ago

##############This might help############ change video_capture = cv2.VideoCapture(0) to video_capture = cv2.VideoCapture(0,cv2.CAP_DSHOW)

redwhatduck commented 3 years ago

D_SHOW is not good enough for me, it drops frame rate to just 10 fps, initialization time is like 5 seconds. If I use MSMF my frame rate is 30 fps, but the initial initialization time is about 20 seconds.

On Linux however, initialization time is almost instant, but I'm not able to achieve 30 fps with any backend. All of them max out at about 15 fps. Which is weird.

baizebing commented 3 years ago

Please try to run native MediaFoundation camera samples and check camera initialization delay, for example this one: https://github.com/microsoft/Windows-classic-samples/tree/master/Samples/Win7Samples/multimedia/mediafoundation/MFCaptureD3D (you might need to disable "Treat warnings as error" compiler option).

I have tried the sample code and it works fine and quick. I think there must be something wrong with the cap_msmf.cpp

mshabunin commented 3 years ago

There were some recent changes in the backend, so it is now possible to disable HW-acceleration during initialzation: https://github.com/opencv/opencv/blob/b19f8603843ac63274379c9e36331db6d4917001/modules/videoio/src/cap_msmf.cpp#L815-L826

Please try to check if the problem is reproduced with disabled HW-acceleration (this works only on the latest master branch):

VideoCapture cap(0, CAP_MSMF, {CAP_PROP_HW_ACCELERATION, VIDEO_ACCELERATION_NONE});
gillerman commented 3 years ago

Disabling HW acceleration during init sounds good. I will test and let you know if I run into problems.

baizebing commented 3 years ago

There were some recent changes in the backend, so it is now possible to disable HW-acceleration during initialzation: https://github.com/opencv/opencv/blob/b19f8603843ac63274379c9e36331db6d4917001/modules/videoio/src/cap_msmf.cpp#L815-L826

Please try to check if the problem is reproduced with disabled HW-acceleration (this works only on the latest master branch):

VideoCapture cap(0, CAP_MSMF, {CAP_PROP_HW_ACCELERATION, VIDEO_ACCELERATION_NONE});

Hi, disabling HW still can't work. Actually it cost a lot of time at

https://github.com/opencv/opencv/blob/b19f8603843ac63274379c9e36331db6d4917001/modules/videoio/src/cap_msmf.cpp#L688

and the output window of Visual Studio shows repeatedly:

'opencv_test_videoiod.exe' (Win32): Unloaded 'C:\Windows\System32\DriverStore\FileRepository\iigd_dch.inf_amd64_1fac62ebeda54cee\igd11dxva64.dll' 'opencv_test_videoiod.exe' (Win32): Unloaded 'C:\Windows\System32\ntasn1.dll' 'opencv_test_videoiod.exe' (Win32): Unloaded 'C:\Windows\System32\ncrypt.dll' 'opencv_test_videoiod.exe' (Win32): Unloaded 'C:\Windows\System32\DriverStore\FileRepository\iigd_dch.inf_amd64_1fac62ebeda54cee\igd10iumd64.dll' 'opencv_test_videoiod.exe' (Win32): Unloaded 'C:\Windows\System32\dxva2.dll' 'opencv_test_videoiod.exe' (Win32): Unloaded 'C:\Windows\System32\DriverStore\FileRepository\iigd_dch.inf_amd64_1fac62ebeda54cee\libmfxhw64.dll' 'opencv_test_videoiod.exe' (Win32): Loaded 'C:\Windows\System32\DriverStore\FileRepository\iigd_dch.inf_amd64_1fac62ebeda54cee\libmfxhw64.dll'. 'opencv_test_videoiod.exe' (Win32): Loaded 'C:\Windows\System32\dxva2.dll'.

it seems it spent lots of time searching the decoder for YUY2 to RGB32(MODE_HW)/RGB24(MODE_SW). Given that my webcam is logitech C920 which has more than 300 native media type. It can cost lots of time to do so.

It seems that we cannot avoid it if we use RGB Mat to receive the YUY2 input frames?

alalek commented 3 years ago

@baizebing Thank you for investigation!

to receive the YUY2 input frames

"Manual" format conversion on OpenCV side should be avoided. Especially for HW-accelerated decoding scenario. OpenCV VideoCapture API returns RGB (BGR), so this is why this format is requested from MSMF framework.

Some report about repeated Loaded-Unloaded MSMF issue is here: https://stackoverflow.com/questions/43740817/windows-media-foundation-imfsourcereadersetcurrentmediatype-execution-time-is

Thenorthernsea commented 3 years ago

I have the same problem with my laptop's build in camera and a usb cam, and the problem disappeared when I switched the backend to Direct Show API

Platform: Win10 professional 20H2

JasonZink commented 3 years ago

I had a similar issue, where the MSMF backend was searching for a good match of the requested formats and parameters. One thing that I found that minimizes the issue is if you set all of your capture properties (such as frame rate, width/height, etc...) before opening the capture object. This still results in some delay due to searching for the desired combination, but the searching only happens once at opening instead of once for each time a parameter is set.

ercarpio commented 3 years ago

Setting MF_READWRITE_ENABLE_HARDWARE_TRANSFORMS to false in getDefaultSourceConfig solved this issue for me. According to the documentation this should not interfere with HW acceleration and I haven't seen any decrease in performance.

mshabunin commented 3 years ago

@ercarpio , could you please propose a pull request with the fix?

tibaes commented 2 years ago

A workaround for this was merged on master branch.

Now you can set the environment flag "OPENCV_VIDEOIO_MSMF_ENABLE_HW_TRANSFORMS" to 0, which will trigger the @ercarpio suggestion and the camera will be loaded without the buggy delay.

On C++ it is as simple as:

#include <cstdlib>
putenv("OPENCV_VIDEOIO_MSMF_ENABLE_HW_TRANSFORMS=0");

Or on Python:

import os
os.environ["OPENCV_VIDEOIO_MSMF_ENABLE_HW_TRANSFORMS"] = "0"
jchrisli commented 2 years ago

Thanks for the fix @tibaes This saves me tons of time. I'd also like to add that setting the variable (OPENCV_VIDEOIO_MSMF_ENABLE_HW_TRANSFORMS) and then immediately initiating the camera in the same script would not work as the environment variable values are set when the Python process starts. Setting the variable in the Shell (PowerShell in my case) and then run the Python script that uses the capture source works.

chocolate-byte commented 2 years ago

save problem working with vs2015，usb cam, x86 compiled with vs

ketangangal commented 2 years ago

Having same problem even after import os os.environ["OPENCV_VIDEOIO_MSMF_ENABLE_HW_TRANSFORMS"] = "0"

janistrov commented 2 years ago

Having the same problem. os.environ["OPENCV_VIDEOIO_MSMF_ENABLE_HW_TRANSFORMS"] = "0" did not help.

Any new suggestions?

JustinLloyd commented 2 years ago

Exhibiting the same problem on latest OpenCV, Python 3.7, Windows 10 Pro. Built-in webcams on Surface Book 2 initialize quickly and as expected. Logitech BRIO webcams connected via USB do not. If I use gstreamer to inspect the USB webcams, or view the incoming video, the USB wecams initialize quickly and as expected. Disabling the HW_TRANSFORMS flag and then restarting the python enviroment does not fix the issue.

whiteHuai commented 2 years ago

Having the same problem. os.environ["OPENCV_VIDEOIO_MSMF_ENABLE_HW_TRANSFORMS"] = "0" did not help.

Any new suggestions?

You must put os.environ["OPENCV_VIDEOIO_MSMF_ENABLE_HW_TRANSFORMS"] = "0" before import cv2

pstricks-fans commented 2 years ago

Prefixing with

import os
os.environ["OPENCV_VIDEOIO_MSMF_ENABLE_HW_TRANSFORMS"] = "0"
import cv2

does solve the problem.

But doing this for every project does not seem to be fun. Will this problem be fixed but with a much better approach?

SMedd69 commented 2 years ago

Si je peux me permettre = Votre problème vient de votre ordinateur et spécialement votre caméra qui est un peu obsolète, ou votre ordinateur ! Essayez de faire votre programme sur un autre ordinateur.. Bon courage à vous. Je suis débutant je me permet de repondre car j'avais eu le même soucis de lenteur. J'ai changé d'ordinateur et cela vas beaucoup mieux ! Bisou

cwfatm commented 1 year ago

You can try use open CAP_DSHOW and set(CAP_PROP_FRAME_WIDTH, width+1); eg: set(CAP_PROP_FRAME_WIDTH, 641); I sucess.

ricksuggs commented 1 year ago

Found this thread while searching for solutions to slow initialization of Logitech webcam. I set the environment variable, OPENCV_VIDEOIO_MSMF_ENABLE_HW_TRANSFORMS=0, and it solved the issue. Is there any place where all of the different environment variables for opencv are documented? Happy to contribute a PR if this setting is not documented anywhere. Thank you.

OaimIsGnay commented 1 year ago

How to fix this issue with Opencvsharp

OaimIsGnay commented 1 year ago

此问题的解决方法已合并到主分支上。

现在，您可以将环境标志“OPENCV_VIDEOIO_MSMF_ENABLE_HW_TRANSFORMS”设置为 0，这将触发建议，并且摄像机将在没有错误延迟的情况下加载。

在C++上，它就像：

#include <cstdlib>
putenv("OPENCV_VIDEOIO_MSMF_ENABLE_HW_TRANSFORMS=0");

或者在 Python 上：

import os
os.environ["OPENCV_VIDEOIO_MSMF_ENABLE_HW_TRANSFORMS"] = "0"

How to fix this issue with Opencvsharp

OaimIsGnay commented 1 year ago

Prefixing with

import os
os.environ["OPENCV_VIDEOIO_MSMF_ENABLE_HW_TRANSFORMS"] = "0"
import cv2

does solve the problem.

But doing this for every project does not seem to be fun. Will this problem be fixed but with a much better approach?

How to fix this issue with Opencvsharp

INNOSYS commented 1 year ago

I tried followings

    import os
    os.environ["OPENCV_VIDEOIO_MSMF_ENABLE_HW_TRANSFORMS"] = "0"
    import cv2

But, my logitech webcam open after several minutes. So, I found this way, try following,

      import os
      import cv2
      from time import time

      st_time = time()
      cap = cv2.VideoCapture(1,cv2.CAP_DSHOW,(cv2.CAP_PROP_HW_ACCELERATION, cv2.VIDEO_ACCELERATION_ANY))
      # capture = cv2.VideoCapture(1,cv2.CAP_MSMF)
      ed_time = time()
      cap.set(cv2.CAP_PROP_FPS, 30.0)
      cap.set(cv2.CAP_PROP_FOURCC, cv2.VideoWriter.fourcc('m','j','p','g'))
      cap.set(cv2.CAP_PROP_FOURCC, cv2.VideoWriter.fourcc('M','J','P','G'))
      cap.set(cv2.CAP_PROP_FRAME_WIDTH, 1920)
      cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 1080)
      print(f"elapsed time: {ed_time - st_time}")

      while(True):
          ret, frame = cap.read()
          cv2.imshow('title',frame)
          if cv2.waitKey(1) & 0xFF == ord('q'):
              break

      cap.release()
      cv2.destroyAllWindows()
aditrhamid commented 2 weeks ago

Prefixing with

import os
os.environ["OPENCV_VIDEOIO_MSMF_ENABLE_HW_TRANSFORMS"] = "0"
import cv2

does solve the problem.

But doing this for every project does not seem to be fun. Will this problem be fixed but with a much better approach?

In the Environment Variables window, under "System variables," click on "New." Enter OPENCV_VIDEOIO_MSMF_ENABLE_HW_TRANSFORMS for the variable name and 0 for the variable value.