Open PuchatekwSzortach opened 8 years ago
can't reproduce it on win
I can't reproduce it with Opencv 2.4.9 on Ubuntu either.
Reproduced it with latest OpenCV master on Linux. But instead of crashing, processes hung. Works well if one of cvtColor
calls is commented.
I move all opencv code into the subprocess function, and it works.
So it sames when you call cv2 functions (except imread), it initializes something which can not be passed to subprocess.
so the idea is to always run opencv code in subprocess.
this is the solution for you.
import multiprocessing
import glob
import numpy
def job2(path):
import cv2
image = cv2.imread(path)
image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
return path
def job1(path):
import cv2
main_image = cv2.imread(path)
main_image = cv2.cvtColor( main_image, cv2.COLOR_BGR2GRAY)
if __name__ == "__main__":
p = Process(target=job1, args=("./image.png"))
p.start()
p.join()
paths = glob.glob("./data/*")
pool = multiprocessing.Pool()
result = pool.map(job2, paths)
print 'Finished'
for value in result:
print value
I just came across this bug myself. In my case I'm doing a lot of work inside of RQ subprocs so moving the conversion to the main "thread" isn't an option. I've got two calls into cvtColor
. The first one is a conversion from a PIL image and the second one is converting a CV image into grayscale. I can work around the first problem with this
cv_image = np.array(image)
cv_image = cv_image[:, :, ::-1].copy()
but I don't have a work around for the second one. Is there another way to convert an image to grayscale without cvtColor
?
I'd suggest to use concurrent.futures.ThreadPoolExecutor from python3.
I ended up switching celery to use eventlet to work around the problem.
When I run this on El Capitan with Python 3.5 and OpenCV3 the script hangs - when I comment out the main_image lines, it finishes.
I have been testing the code of PuchatekwSzortach on two different ubuntu os (14.04), with 8GB and 64GB of ram respectively. I have the same problem as others when using 8 GB. However when using the 64GB it works.
This is however not sustainable with larger code. When running a code with opencv functions calls embedded in an homebrewed function parallelized in a "multiprocessing" pool, the code eventually ends up with idle processors after a number of call to the pool that can fluctuate from run to run.
Seeing the same issue where the script hangs if main_image = cv2.cvtColor(...) is uncommented. OSX 10.11.1, Python 3.5. Came here because I've seen the bug in other places in my codebase as well, not just with cvtColor.
Setting the start method to 'spawn' avoids the hang:
import cv2
import multiprocessing
import glob
import numpy
def job(path):
image = cv2.imread(path)
image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
return path
if __name__ == "__main__":
main_image = cv2.imread("./image.png")
main_image = cv2.cvtColor( main_image, cv2.COLOR_BGR2GRAY)
multiprocessing.set_start_method('spawn')
paths = glob.glob("./data/*")
pool = multiprocessing.Pool()
result = pool.map(job, paths)
print('Finished')
for value in result:
print(value)
thanks ekolve. Do you know if there is an equivalent for python 2.7?
I tried the code with OpenCV 3.1 on OSX 10.11.2 and Python 3.4 and it worked just fine.
I'm having similar issues on OSX 10.11.2 (15C50), Python 3.5.1, OpenCV 2.4.12_2 installed via homebrew.
Have the same problem after upgrading my python 2.7 code from opencv 2.4.11 to opencv 3.1. For me its is a cvtColor call which converts from BGR to HSV. Does anybody have any hints where to look in the c++ code to start fixing this annoying problem?
I experienced the same problem (code hangs) now in another call: mySurfDetector.detectAndCompute
@georgwaechter , there' a similar observation (c++ / multithreading/ cvtColor) here
I'm currently testing whether this maybe originates from the thread pool used in functions like cvtColor and detectAndCompute. In my understanding forking (without an "exec" call afterwords as done in python under linux) could be a problem, but thats only a guess. I've read that forking only clones the current thread. In the case the thread pool may assume wrongly that it has more threads that it has ...
@berak thanks for the link.
Ok, I think i nailed down the problem to the implementation of the pthreads thread pool in https://github.com/Itseez/opencv/blob/master/modules/core/src/parallel_pthreads.cpp ..
I have recompiled OpenCV "-D BUILD_TBB=ON -D WITH_TBB=ON" and suddenly it works! Using this cmake option i use the intel threading library instead of the standard pthreads. The intel library states it has "better fork support".
@berak The problem described there is something else as this problem with python only relates to multiple processes, not multi-threading per se.
Idea to solve the problem within the c++ code: Register a fork handler via http://pubs.opengroup.org/onlinepubs/009695399/functions/pthread_atfork.html and reset the state of the thread pool on forking the process. This has the effect that the next call to cvtColor (or any other method using the thread pool) initializes the thread pool. thus starts the threads.
At the moment it seems that the code is waiting infinitley for a condition that is never signaled because the number of threads is not as expected after forking: https://github.com/Itseez/opencv/blob/master/modules/core/src/parallel_pthreads.cpp#L440 The "notify_complete" method only signals the condition if all of the threads completed its work ..
This also explains why setting the start method to "spawn" solves the problem as described by @ekolve. Spawing resets the memory after forking, therefore forcing a re-inizialization of all data structures (as the thread pool) within OpenCV.
@mshabunin Can you change the category of this issue? This is not related to the python bindings, but to the core.
Here is another library with the same problem: https://github.com/flame/blis/issues/30
I'll try to prepare a pull request for this.
@georgwaechter Thanks for problem digging!
Fork is very dangerous operation. In general, after fork any mutex/semaphore/other resources should be reset to initial state. This problem is actual not for "pthread threadpool" only, but for other many states (during fork call OpenCL/CUDA can run/schedule/poll/wait for background tasks, we can capture images from camera or interact with GUI) - all these things may go into broken state with unpredictable behavior. Also the first message of this issue comes with problem on Mac OSX where "pthreads threadpool" is not used by default - GCD is used, and it is failed.
I don't see here an easy fix.
I believe more reliable way is to detect fork()
call, write error message with problem and workarounds, and die.
Unfortunately the possibility to fork opencv processes is very important for the python-world, where multi-processing is more common than multi-threading. The problem is the Global Interpreter Lock (GIL) within the python runtime which prevents python-code from running in parallel in multiple threads.
So, producing an error in case of fork() would prevent many use-cases. In my example we are processing camera frames in near-realtime and do further costly analysis for only some selected frames in a second (and third) process.
You're right, there are many situations where fork could bring the system into an inconsistent state. But most programs fork at startup where the state can be cleaned up easily as in this case.
I'm currently testing whether resetting the thread pool works as expected. Most importantly there should be no worker thread running, so forking is only safe when no task is running at this moment. However as multithreading is not used in Python, this can not occur. In c++ this is possible but can also be detected and handled using the child-part of the fork-handler. On forking, we could automatically wait until the task is complete, so that it is safe to reset the pool state.
Other things like the background thread of the camera are more complicated - i guess. Those could throw an error on fork, but i have not investigated any time in understanding the related code.
Sadly, on OSX this seems to be impossible. There is no fork handler i know of. Apple seems to warn about this situation and recommends to execute exec() after fork() .. https://developer.apple.com/library/ios/documentation/Performance/Reference/GCD_libdispatch_Ref/
As a workaround one can disable multithreading before forking and reenable it in each child process:
import cv2
import multiprocessing
import glob
import numpy
def job(path):
# enable multithreading in OpenCV for child thread
cv2.setNumThreads(-1)
image = cv2.imread(path)
image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
return path
if __name__ == "__main__":
# disable multithreading in OpenCV for main thread to avoid problems after fork
cv2.setNumThreads(0)
main_image = cv2.imread("opencv/samples/data/lena.jpg")
main_image = cv2.cvtColor( main_image, cv2.COLOR_BGR2GRAY)
paths = glob.glob("opencv/samples/data/*.png")
pool = multiprocessing.Pool()
result = pool.map(job, paths)
print 'Finished'
for value in result:
print value
It works in Ubuntu (pthreads backend), but I didn't tested it on Mac (with GCD backend).
On OSX El Capitan, Python 3.5.1 this workaround works as long as threads are not re-enabled in the child process. If I place the line:
cv2.setNumThreads(-1)
in the child process then the children hang.
I'm actually getting a similar bug using the resize function in a subprocess. I'm using python 2.7 and opencv 2.4.11
Thread 0 Crashed:: Dispatch queue: com.apple.root.default-qos
0 libdispatch.dylib 0x00007fff8b5d013f dispatch_apply_f + 769
1 libopencv_core.2.4.dylib 0x0000000102ea58b3 cv::parallel_for_(cv::Range const&, cv::ParallelLoopBody const&, double) + 115
2 libopencv_imgproc.2.4.dylib 0x0000000102b4b0ae void cv::resizeGeneric_<cv::HResizeLinear<unsigned char, int, short, 2048, cv::HResizeNoVec>, cv::VResizeLinear<unsigned char, int, short, cv::FixedPtCast<int, unsigned char, 22>, cv::VResizeLinearVec_32s8u> >(cv::Mat const&, cv::Mat&, int const*, void const*, int const*, void const*, int, int, int) + 590
3 libopencv_imgproc.2.4.dylib 0x0000000102b4a49f cv::resize(cv::_InputArray const&, cv::_OutputArray const&, cv::Size_<int>, double, double, int) + 8463
But it looks like the problem is a bit different because I can run resize function inside a subprocess with importing cv2 globally. The following code is just resizing an image and works fine.
import multiprocessing
import cv2
def resize1(path):
img = cv2.imread(path, 0)
print "Start resizing"
cv2.resize(img, (400, 400))
print "Done resizing"
def main():
path = "image1.jpg"
p1 = multiprocessing.Process(target=resize1, args=(path,))
p1.start()
p1.join()
if __name__ == "__main__":
main()
However if I change my main function to also resize an image, my subprocess crash.
def main():
path = "image1.jpg"
img = cv2.imread(path)
cv2.resize(img, (400, 400))
p1 = multiprocessing.Process(target=resize1, args=(path,))
p1.start()
p1.join()
The only solution which works so far it's too create a subprocess every time i need to use opencv which is pretty annoying. I also tried the cv2.setNumThreads solutions and it doesn't work.
@mshabunin solution works on mac 10.10.5 , opencv2.4 , python 2.7 using celery to resolve the issues at image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY) .
Was this ever resolved? I'm encountering the same issue using OpenCV 3.2.0-dev with Python 2.7.6 on Ubuntu 14.04.
EDIT: After following the suggestion by @georgwaechter to rebuild OpenCV with TBB, I no longer encounter this issue.
I have encountered the same issue using OpenCV 3.2.0-dev and python3.6 on Ubuntu14.04 .
Compiling OpenCV with TBB solved the problem I had in my use case. I dont know if this is a decent workaround.
any update? same here with mac os 10.12.3 with both python2 and python3, had tried build opencv with TBB, but did not solve the problem.
also tried in ubuntu(virtual machine) 16.04+python2 +opencv2.x, do not work again.
thanks a lot, I ran into the same problem, took some time to find a solution.
multiprocessing.set_start_method('spawn')
works for python 3.5, opencv 3.2 on os x
encountered the same issue, cvtColor causes process to hang.
multiprocessing.set_start_method('spawn')
fixes the issue for python3.5, opencv 3.2 built from source on ubuntu 16.04
We designed loky to mitigate this issue on any versions from python 2.7+ and 3.3+.
It's based on the concurrent.futures.ProcessPoolExecutor
API but uses the spawn
start method by default and fixes other possible deadlocks with non-picklable objects.
This sounds related to the problem I have in hand. In python2.7 on MacOS Sierra I need to access the cv2.VideoCapture(0)
web cam stream in a different multiprocessing.Process()
, but I can't make it work without an abrupt exit, or if I have imported pygame, Fatal Python error: (pygame parachute) Segmentation Fault
. I still don't know why pygame complains since I commented out all other pygame code but it is pygame that throws the Segmentation Fault...
I tried @georgwaechter 's TDD compilation, @mshabunin 's setting single thread, @corentin87 's in-proc import without global import, but none worked. @ekolve 's multiprocessing.set_start_method('spawn')
in a different python3 environment gave me raise RuntimeError('context has already been set
, and if I switch on force=True
, there is a dark warning:
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
Any idea?
You need to launch your program under the if __name__ == '__main__'
block as explained in the error message.
also got hit by this (in https://github.com/pytorch/pytorch/issues/1838's scenario)
I encounter similar error https://stackoverflow.com/questions/46782277/opencv-sigsegv-when-applying-backgroundsubtractor-on-a-frame-that-read-from-anot
I am having the same issue on windows 7, OpenCV 3.3.1
, Python 3.6.4
, from Anaconda.
cv2.setNumThreads(0)
and multiprocessing.set_start_method('spawn', force=True)
do not fix the issue of processes hanging. I am reading a video file in the main thread, sending frames to a Queue
where those frames are then processed by multiprocessing.Process
instances.
Any ideas?
I am using multiprocessing to speed up ARUCO target detection on a raspberry pi. (Keep in mind I have gotten this working using no multiprocessing.)
Using this code: `` workers = [] for i in range(num_workers): workers.append(Worker_Bee(tasks,results,i))
for worker in workers: worker.start()
for i in range(20): tasks.put(Image_Processor(vs.read())) time.sleep(.5)
`` The Image_Processor looks at the frame vs.read() which is an image. The image processor tries to find targets in the image. The worker process Image_Processor hangs on the following line: corners, ids, rejectedImgPoints = aruco.detectMarkers(gray, self.aruco_dict, parameters=self.parameters) where gray is the image vs.read() I've check the arguments to aruco.detectMarkers and they seem to be correct (and as I've said before, I've ran this before without multiprocessing fine). There is no error message, and looking at top, I see the python process spawning but then quickly going away. The parent process stays.
I've seen similar problems with openCV hanging on cvtColor or resize, but all of their problems are fixed for me (compiling with TBB options and such).
Any advice on how to get this working?
I switched to eventlet to get this working.
@tkoch96 Any chance you made any progress on the issues you were seeing? I'm running into nearly exactly the same problem you are.
No, it seems the library is designed with this inherent flaw and the developers here have insinuated it probably won't be fixed. I'm not too savvy with understanding how this low-level stuff works so I did not try to fix it myself.
If I were you I'd look for alternative solutions to speeding up processing.
@tkoch96 Are you're referring to the aruco library or OpenCV?
I believe I posted this as an OpenCV issue (separate to this post) and they referred me to this post and closed my issue. I have not posted on any aruco specific forum.
I had the same issue on OSX.
Thanks to @mshabunin concurrent.futures.ThreadPoolExecutor
works fine for me.
I have encountered the same issue on Python 2.7.10, Opencv 3.2 and Ubuntu 16.04
cv2.cvtColor(sheet_image.image, cv2.COLOR_GRAY2RGB)
caused the problem, both parent process and child process hang forever, and the child process still exits after parent process gets killed.
I'm trying to use OpenCV with Python's multiprocessing module, but it breaks on me even with very simple code. Here is an example:
If I remove
main_image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
script works, but with it in there it doesn't, even though that line shouldn't affect jobs processed by the pool. All image paths lead to simple images, about ~20 of them in total. Funny enough code works fine if I create images in memory withnumpy
instead of reading them withimread()
. I'm guessing OpenCV uses some shared variables behind the scene that aren't protected from race conditions.My environment: Mac OS X 10.10, OpenCV 3.0.0, Python 2.7. The last few lines of stack trace are:
BTW - I got other OpenCV functions to crash when used with Python multiprocessing too, above is just the smallest example I could produce that reflects the problem.
Also, I got above algorithm (and much more complicated ones) to work in multithreaded C++ programs, using same OpenCV build on same machine, so I guess the issue lies on Python bindings side.