Background image thread for Windows

[GoogleCodeExporter]

I do a lot of interactive experimentation on Windows and early on
implemented my own poor man's background window implementation so I didn't
have to keep calling cvWaitKey all the time, since OpenCV has no native
cvStartWindowThread implementation on Windows.

I decided to try packaging it up more neatly, and it's not too bad.  I've
attached two files.  The first (highgui_win32) is a module implementing the
highgui window-related functions that can defer their operation to a
background thread, if it's been created with the standard
cvStartWindowThread function.  Otherwise they just pass control to the
normal highgui wrappers.

The second is a patch to __init__ to load this module on Windows, in which
case its functions will overlay the originals in highgui, so on Windows it
transparently takes over and gives users a working cvStartWindowThread.

This form of the wrapper is a little new (as opposed to my grown-over-time
original version), so may have some rough edges, but it still works with my
stuff, and it only really becomes active if you create the thread. 
Existing demos run fine without doing so.

-- David

Original issue reported on code.google.com by db3l.em...@gmail.com on 14 Jan 2009 at 2:23

Attachments:

• highgui_win32.py
• win32_thread.diff

[GoogleCodeExporter]

I was missing an import for the traceback module in the first attached
highgui_win32.py.  Attached is an updated version that includes it.

Original comment by db3l.em...@gmail.com on 14 Jan 2009 at 3:07

Attachments:

• highgui_win32.py

[GoogleCodeExporter]

This is great. I've been looking for such a thread-based solution for highgui
functions. My experiments' timing have always been interfered by small delays
occurred by cvWaitKey(). This solves my problem nicely. Patched. Thanks.

Original comment by pmtri80@gmail.com on 14 Jan 2009 at 3:56

• Changed state: Fixed

[GoogleCodeExporter]

I have attempted to on cvStartWindowThread() in a few demo files. They are 
working
well. However, I've noticed that when threading is used, the drawings 
occasionally
blink on top of the background images. This effect may be because the underlying
repainting process of OpenCV is being run in a different thread from the main 
thread.
I'm fine with the blinking effect though. But I think I will not turn on
cvStartWindowThread() in the demos, to avoid confusion to the user.

Original comment by pmtri80@gmail.com on 14 Jan 2009 at 4:09

[GoogleCodeExporter]

I definitely don't think you need to use it by default in the demos - it's not 
really
very beneficial for code already structured in a WaitKey loop.

I'll have to experiment to see if I can see the blinking you're referring to, 
as I'm
not quite sure of the drawings vs. background image issue (presumably any 
drawing is
being done into a different image which is then shown)?  Do you have a 
particular
demo I can try with?  

Note that I do try to mimic foreground behavior with cvNamedWindow by bringing 
it to
the foreground (which is what happens when its called from the main thread), so 
maybe
there's an interaction there.  It can be disabled with an extra (to the 
highgui_win32
implementation) keyword argument of "foreground=False").

Original comment by db3l.em...@gmail.com on 14 Jan 2009 at 4:19

[GoogleCodeExporter]

The most obvious one at my end is the lkdemo.py demo. At every iteration, it 
needs to
draw tracked points on top of the current frame. What I did was just putting a 
call
"cvStartWindowThread()" without any argument at line 51. I haven't tried with 
the
"foreground" keyword though.

Original comment by pmtri80@gmail.com on 14 Jan 2009 at 7:25

[GoogleCodeExporter]

I can't create it on my end, but I think I know what's going on and will have an
update to highgui_win32 completed shortly.

Original comment by db3l.em...@gmail.com on 14 Jan 2009 at 7:28

[GoogleCodeExporter]

I have tried the lkdemo.py demo with both foreground=True and foreground=False 
in
calling cvNamedWindow(). The blinking is appearing in both cases. However, if I 
copy
the image to be displayed to another temporary image, and display this temporary
image, then the blinking is gone. I hope these facts will help you narrow down 
the cause.

Original comment by pmtri80@gmail.com on 14 Jan 2009 at 7:44

[GoogleCodeExporter]

Yes, that agrees with my theory, though I haven't seen it yet myself (lkdemo is 
one
of those that requires a video file input that I don't have lying around yet).

The problem is that the current implement enqueues requests for the window 
thread to
execute by immediately returns to the foreground thread.  So if the foreground 
thread
continues to use a parameter (such as an image to ShowImage), you get thread
contention to the same object.  Most of my own use already cloned images I was
showing due to restrictions on my older thread implementations so I didn't 
notice this.

I thought of always cloning the image being displayed so the copy in the queue 
was
independent of the caller's, but that could be very space inefficient and it 
still
lets the foreground thread get ahead of the background.

So instead the attached patch changes things so the foreground cv* calls are, by
default, synchronized with the background thread, blocking until they finish
executing.  I think this is probably closer in spirit to cvCreateWindowThread 
on the
other platforms where it is primarily designed for the message loop rather than
deferring operations.

This does somewhat limit performance (about 65fps on my P4) since you have to 
context
switch back and forth for each call.  So I've added an optional - and
platform-specific - "synchronized" parameter to the cvStartWindowThread call 
for code
that understands the impact - e.g., doesn't manipulate an image it gives to
ShowImage, but clones if needed.

I think having things synchronized by default makes the most sense though.

-- David

PS: While working on this I also discovered that the OpenCV WaitKey() method on
Windows is very CPU inefficient if there aren't any windows, so I only call 
that once
I see a window created - avoids heavier CPU usage once the thread is created but
prior to any windows.

Original comment by db3l.em...@gmail.com on 14 Jan 2009 at 7:50

Attachments:

• highgui.diff

[GoogleCodeExporter]

BTW, I found an avi to try lkdemo with (though I had to modify lkdemo to 
continously
loop through the same file since it was short) and either my machine is fast 
enough
or slow enough that I'm not seeing the blinking.  So it'll be useful to see if 
the
changes resolve it for you.

Original comment by db3l.em...@gmail.com on 14 Jan 2009 at 8:05

[GoogleCodeExporter]

I'd like to confirm that the latest patch has fixed the blinking issue. There's 
one
thing interesting: even when cvStartWindowThread is called with keyword
'synchronized' set to False, there is no blinking.

Original comment by pmtri80@gmail.com on 14 Jan 2009 at 8:11

[GoogleCodeExporter]

One other tiny update - shift so flag for having windows is triggered both by
NamedWindow as well as the convenience "show" method.  This diff is intended to 
be
applied on top of the last one.

Original comment by db3l.em...@gmail.com on 14 Jan 2009 at 8:12

Attachments:

• highgui-1.diff

[GoogleCodeExporter]

"even when cvStartWindowThread is called with keyword 'synchronized' set to 
False,
there is no blinking."

Hmm, that's interesting.  My guess is that the changes I made to reduce the 
latency
in the window thread means it gets to the image fast enough that you don't get 
the
overlapped usage.  The older implementation had a 100ms WaitKey, so it could
essentially take up 100ms to execute the ShowImage.  With the new 
implementation, it
shouldn't be any longer than 5-10ms, so as long as the foreground thread yields 
at
least briefly (such as the WaitKey in lkdemo) it may be safe.  Certainly has 
more
wiggle room.

Original comment by db3l.em...@gmail.com on 14 Jan 2009 at 8:16

[GoogleCodeExporter]

Patched. Thanks.

Original comment by pmtri80@gmail.com on 14 Jan 2009 at 8:18

[GoogleCodeExporter]

Yeah. I also think that is the reason. In any case, I am eager to use the new 
highgui
(with synchronized=False) in my experiments. I'm hoping to see some improved 
results...

Original comment by pmtri80@gmail.com on 14 Jan 2009 at 8:25

[GoogleCodeExporter]

While working on other code today using a similar thread/queue system, it made 
me
realize that when the window thread is unsynchronized, the queue of function 
results
is still building up.  Unlikely to burn a ton of memory, but this patch 
suppresses
the result queue when it isn't needed.

Original comment by db3l.em...@gmail.com on 20 Jan 2009 at 2:10

Attachments:

• hgwin32_unsync.diff

[GoogleCodeExporter]

Oh, sorry. I've forgotten about this patch. Patched. Thanks.

Original comment by pmtri80@gmail.com on 31 Jan 2009 at 3:06