GLPlot.jl and OpenCV.jl interface possibilties?

[maxruby]

@SimonDanisch.

I figure it might be interesting to exchange some ideas on how OpenGL textures could be processed in OpenCV, and discuss possible interfaces for advanced image processing and graphics. In my first version of OpenCV.jl, I have used OSX Cocoa first to display the images but now I also managed to get Qt 5.3 + OpenGL working as well. It seems that there are multiple options available.

I had a look at your GLPlot package and the supporting packages (GLWindow, etc). My first general impression is that it would be good to have the option to extract the minimum necessary support from these packages to interface with OpenCV rather than calling everything at once. I am not too keen on slowing down the whole thing, especially as this is a serious caveat with Julia right now. What is it that you want to get out of the OpenCV framework? On my side, I am particularly interested in using image display and processing tools for high-speed real-time tracking and object recognition.

[SimonDanisch]

Well, my mean goal would be, to have Computer Vision running completely on the GPU, allowing for great real time possibilities. But for that quite a bit of work is required, also for the OpenCL package (cc @vchuravy) In the end it would be nice, to have one abstract array type, wrapping OpenCL, OpenGL and OpenCV memory, which you can run a lot of different algorithms on... We can start with the trivial approach, which is that I add the support to visualize the cv::Mat, which than enables imshow capabilities inside GLPlot. This makes sense, if you also want to visualize 3D tracking data for example, so that you can do this in one framework. GLPlot is also very good at visualizing depth data and switching between different visualization types, so it would be a good addition I think. I'm actually working on a scripting environment building up on GLPlot, which supports slider, general editing of values and multiple windows and such ;)

[maxruby]

Sounds great! I assume that combining all these so that they can be accessed through a single abstract type will be a slightly longer term goal. . . By the sounds of it you will develop your own independent access to OpenCV directly? It is trivial to add the support for OpenGL textures in OpenCV - this is documented. The bigger challenge is to pass the Textures to a UMat or GpuMat (CUDA) -- I have found this to be difficult in OSX because of the complications with compiling OpenCV with CUDA (also depending hardware, etc), so I am also looking into the OpenCL alternative.

My main priority now is improving real-time object detection and tracking through OpenCV - rather than the visualization and plotting, which as you say is better suited for OpenGL and others. Currently, I am working on a tracking method that uses principal skeleton extraction (focused on zebrafish larvae). The main effort is in how to speed up new algorithms for the tracking itself in C++/Julia, and clearly having great visualization and plotting will be an important complement.

[SimonDanisch]

Yes, I'd just add a glplot(::cvMat) function, which just takes the memory and displays it like a normal image for now... I think the OpenGL texture support is basically just a very simple wrapper for OpenGL inside of OpenCV for the QT/whatnot OpenGL context. I just posted to the OpenCV mailing list... hope we can get some more information out of that! http://opencv-users.1802565.n2.nabble.com/OpenGL-OpenCL-Open-td7585246.html

[maxruby]

From reading the OpenCV Q&A forum, I think the first 2 points in your proposal are likely not to be considered OpenCVissues per se, i,e., not officially supported by the OpenCV API . . . For example, in principle, there is no official OpenGL support for Mac OSX according to the official documentation. However, this is strictly not true.

I will focus on customizing OpenCV.jl for real-time tracking and object detection using OpenCL and CUDA if possible. Perhaps once this is working well we can see how our efforts can be combined/complemented?

[SimonDanisch]

Oh, I guess I didn't phrase that right... This is definitely not supported by OpenCV and doesn't have to, as it is supported by me ;) I will completely ignore OpenCV's OpenGL support and context creation, as this is the Job of the GL-libraries. Sounds like a good plan to me!

[maxruby]

Seeing this thread it does appear that there are ongoing discussions about how to get smooth conversion/handling of UMat between CPU/GPU, especially when this needs to be dynamic . . .

[SimonDanisch]

I think this is somehow doable... And it seems that for OpenGL interop with CUDA they are already implementing the basics: https://github.com/Itseez/opencv/blob/aadab03b7bf99cbeb604da451e10a6e7decf39bc/modules/core/src/opengl.cpp Which makes me a little sad ;) But it seems, that you can initialize OpenCV with an OpenGL interop context:

Alternative way to initialize OpenCL computation context.
C++: void ocl::initializeContext(void* pClPlatform, void* pClContext, void* pClDevice)
Parameters: 
pClPlatform – selected platform_id (via pointer, parameter type is cl_platform_id*)
pClContext – selected cl_context (via pointer, parameter type is cl_context*)
pClDevice – selected cl_device_id (via pointer, parameter type is cl_device_id*)
This function can be used for context initialization with D3D/OpenGL interoperability.

I still don't know how to initialize a umat with an opencl buffer, but from reading the source I infer, that its somewhat possible, if you know OpenCV well enough!

[maxruby]

Thanks for the update. I have already wrapped OpenCL support for OpenCV.jl and OpenGL can be used without much extra effort. Its easy to initialize a UMat in OpenCV and use it with most functions. There is no even a need to convert the UMat (GPU) to CPU before displaying in an OpenGL context - this can be done using imshow with the flag WINDOW_OPENGL. I have even multithreading working through boost and OpenCV.jl so that its possible to run concurrent threads for CPU and use GPU whenever you like in Julia via C++. The challenge actually is how to build a decent GUI interface to drive the applications using OpenCV.jl in Julia. This is not trivial because it seems that Qt is by far the best option (other alternatives such as Gtk/Glade or FLTK would make things more complicated). I am looking into whether to wrap the core of Qt for Julia so we can do something more realistic here. . .

[SimonDanisch]

I actually hope to build GUIs with my GL packages at some point ;) That's why it'd be nice to display the umat with my packages....

2015-01-07 0:51 GMT+01:00 Maximiliano Suster notifications@github.com:

Thanks for the update. I have already wrapped OpenCL support for OpenCV.jl and OpenGL can be used without much extra effort. Its easy to initialize a UMat in OpenCV and use it with most functions. There is no even a need to convert the UMat (GPU) to CPU before displaying in an OpenGL context - this can be done using imshow with the flag WINDOW_OPENGL. I have even multithreading working through boost and OpenCV.jl so that its possible to run concurrent threads for CPU and use GPU whenever you like in Julia via C++. The challenge actually is how to build a decent GUI interface to drive the applications using OpenCV.jl in Julia. This is not trivial because it seems that Qt is by far the best option (other alternatives such as Gtk/Glade or FLTK would make things more complicated). I am looking into whether to wrap the core of Qt for Julia so we can do something more realistic here. . .

— Reply to this email directly or view it on GitHub https://github.com/SimonDanisch/GLPlot.jl/issues/18#issuecomment-68956392 .

[maxruby]

Yes, but its not only the GUI windows (rapid drawing) that needs support, also the entire "signal/slots" event handling, multithreading (concurrent) GUIs and image processing that must be supported as smoothly as possible (without involving a lot of dependencies), I have investigated very closely many of the existing GUI interfaces in C++ and how they are typically used, and it seems to me that every time you need to re-invent the wheel. This appears very wasteful and it would be a lot wiser to either have something like Qt or a native Julia solution like Qt that supports most applications . . .

[SimonDanisch]

You're right, but I'd really like to have a GUI library written in a sane language, while being very fast, so that its easy to extend. I don't know how far I'll get, but I at least have all the event handling and things like sliders natively implemented in OpenGL+Julia ;)

2015-01-07 2:15 GMT+01:00 Maximiliano Suster notifications@github.com:

Yes, but its not only the GUI windows (rapid drawing) that needs support, also the entire "signal/slots" event handling, multithreading (concurrent) GUIs and image processing that must be supported as smoothly as possible (without involving a lot of dependencies), I have investigated very closely many of the existing GUI interfaces in C++ and how they are typically used, and it seems to me that every time you need to re-invent the wheel. This appears very wasteful and it would be a lot wiser to either have something like Qt or a native Julia solution like Qt that supports most applications . . .

— Reply to this email directly or view it on GitHub https://github.com/SimonDanisch/GLPlot.jl/issues/18#issuecomment-68964186 .

[maxruby]

Good luck! I will work on wrapping Qt. Together with multithreading support from C++ boost, we should be able to run high-performance image acquistion/display through Qt. What I am really looking forward to with the Julia interface is that the algorithms and data analysis of events can then be done with the high peformance algorithms in Julia (machine learning, K-clustering) in an independent thread (which does not have to be real-time) and hopefully sometime in the future perhaps we can also use cloud computing as these guys have done with Thunder with Python/Amazon cloud services.

[SimonDanisch]

Thanks =) Who knows, maybe I'll end up using your QT wrapper... If I am given a moment of sanity ;)

Thunder looks awesome!

2015-01-07 2:38 GMT+01:00 Maximiliano Suster notifications@github.com:

Good luck! I will work on wrapping Qt. Together with multithreading support from C++ boost, we should be able to run high-performance image acquistion/display through Qt. What I am really looking forward to with the Julia interface is that the algorithms and data analysis of events can then be done with the high peformance algorithms in Julia (machine learning, K-clustering) in an independent thread (which does not have to be real-time) and hopefully sometime in the future perhaps we can also use cloud computing as these guys have done with Thunder https://github.com/freeman-lab/thunder/ with Python/Amazon cloud services.

— Reply to this email directly or view it on GitHub https://github.com/SimonDanisch/GLPlot.jl/issues/18#issuecomment-68966378 .

[SimonDanisch]

Hey I was just thinking, wouldn't it be smart, if we at least try to use a similar designed pipeline? Common types and such? Than we can reuse code and probably simply exchange the front-end, if one comes out better than the other... I'm pretty open to any design choice, I "just" want to have something easy to extend, fast and flexible in the end ;) I guess this mainly means using the same Vector math / Color library and a common abstraction for the event system...

[maxruby]

Sure, I have been open and eager to have more interaction in the development of GUI and Computer Vision support for Julia. What do you have in mind for the vector math/color library on your end? I am not as familiar with the integration of sliders and callbacks in OpenGL. Are you using GLUT for handling callbacks? If so, isnt´t the support for callbacks pretty limited, especially when compared to Qt? Can you explain the overview of your proposed approach for capture/processing and viewing and integration with event handling (i.e., flexible callbacks) with OpenGL? I think we need to start by discussion and clarifying the "design" principle - once we have that I believe that we could collaborate productively. . .

[SimonDanisch]

Types: I'm working on a FixedSizeArray implementation with the following features (besides being very fast & with C memory layout):

#D3 => generic vector with 3 dimensions
immutable RGB{T} <: FixedSizeWrapper{D3{T}}
    data::D3{T}
end
# Define accessors to the vector
# Red = inspired by SIUnits.jl
# abstract Dimension{T} <: Number
# Red{T <: FloatingPoint} <: Dimension{T}
@type_accessors RGB (Red => 1, Green => 2, Blue => 3)

RGB{T}(a::T,b::T,c::T) = RGB(D3{T}(a,b,c))
RGB{T}(i::T) = RGB(D3{T}(i,i,i))

abstract GeometricVector{T} <: FixedSizeWrapper{T}
@type_accessors GeometricVector (X => 1, Y => 2, Z => 3, W => 4)

immutable Point{T <: FixedSizeVector} <: GeometricVector{T}
    data::T
end

immutable Normal{T <: FixedSizeVector} <: GeometricVector{T}
    data::T
end

immutable Vertex{T <: FixedSizeVector} <: GeometricVector{T}
    data::T
end
const white = RGB(1f0)
const img = [white for i=1:100, j=1:100]
redchannel = img[Red] # Array{Red{Float32},2}
redchannel = img[1:10, 1:10, Red:Green] # Array{D2{Red{Float32}, Green{Float32}},2}
#Conversion to other spaces, without any overhead:
in_geometric_colorspace = convert(Point, img) # Array{Point{D3{Float32}},2}
in_geometric_colorspace[X] # Array{X{Float32}, 2}
#Set single color value in the matrix
img[1, 1, Red] = 0.77f0
const p = Point(v) 
p[X] # X{Float32}(1.0f0)
p[1] # 1.0f0
p+p; normalize(p); p/1f0; #Basic arithmetic operations are defined

Biggest advantage of a system like this is, that I always know how to visualize the values. Matrix{Float32} could mean a myriad of things, but with Matrix{Z} or Matrix{Red} it's relatively clear what you want to see. Similar there will be a matrix type like

immutable TranslationMatrix{T <: FixedSizeMatrix}  <: FixedSizeWrapper{T}
  data::T
end
immutable RotationMatrix{T <: FixedSizeMatrix}  <: FixedSizeWrapper{T}
  data::T
end
immutable ViewMatrix{T <: FixedSizeMatrix}  <: FixedSizeWrapper{T}
  data::T
end

I'm not sure if I'll go through with this though... Probably I'll just have one matrix type. Advantage: We would have more information to detect common mistakes like transforming vectors in the wrong spaces or in the wrong order. Disadvantage: Possibly really complicated to implement.

So far I'm using GLFW.jl together with Reactive.jl for the event handling. GLFW.jl is a very nice library, Reactive is sort of nice, but slow and with just a small feature set. Biggest drawbacks so far: It works only with immutables, or at least doesn't use mutability. This means if you want to do something like this (Which I do want to have at some point):

image = lift(fps(30.0)) do fpsdiff # sample video source at 30 frames
 VideoIO.read(camera)
end
edges = lift(canny, image)
gaussed = lift(gaussian, image)
whatnot = lift( image) do img
 #fancy stuff
end
image_tiles = lift(vcat, image, edges, gaussed, whatnot) # turn into array of images
lift(visualize, image_tiles) # have a two by two grid view of the images

Reactive will create a lot of copies with every new value, which is quite unacceptable... Also its impossible to rewire the lifts interactively, which I find especially bad, for designing a generic API. You basically need to know from the beginning, how your pipeline should look in the end.

You can look at https://github.com/SimonDanisch/Romeo.jl/blob/master/test/runtests.jl for an example of how I build GUIs at the moment. This (at the moment not working) code yields this screen: Where all textfields are editable and all numbers and colors (but the ones in the textfields) can be dragged to change it.

[maxruby]

First, thanks for sharing your progress on this! You have been definitely putting in some effort into your approach. I like very much the idea of doing something with the "native" julia packages. Do you need to redefine RGB{T}(a::T,b::T,c::T) = RGB(D3{T}(a,b,c))? I thought this is part of Color and you just tap into it. I need to look carefully through your code so I understand fully what you are doing, but my first impressions are these:

• Dependencies: I see that you are relying on a lot of packages. Not sure that this really such a great asset right now, at least until all are "stable". I worry about having too many dependencies, and actually this is also a problem with big packages like Qt. I don`t like having to rely on so many internals that can go wrong for reasons beyond your control.
• Image type and declaration: The good thing here is that with your bottom-up approach, it should be easy in principle to create new types and extend the support far more easily than modifying an external C++ library. I would keep a single Matrix type, just as you said, this can complicate things eventually quite a lot. Even OpenCV is using Mat only (though also more specialized SparseMat or UMat) and the rest is done by applying transformations on Mat.
• Event handling and coupling display/processing: I think this is the weakest part of the approach you propose right now. I have tried Reactive.jl, and I like the concept but it is currently too slow for real-time applications and has other challenges. As you said, it creates copies of the images (which is not good for us), and lifting, while nice, appears to me a bit limited in scope.
• GUI builder: looks interesting. I am very curious to understand how you designed it and whether we could easily make menus and add buttons and more controls/callbacks so that it is a full-fledged GUI.

The other big issue I take into consideration right at the beginning is how will we be able to parallelize processes (display vs processing), because there is no way you can aim for high speed real-time (>200 fps) computer vision without running things in parallel. This should be considered in the design of the application just as Qt has QtThread and concurrency support.

[maxruby]

@SimonDanisch.

I ran your Romeo.jl test and it did work on my Mac OSX 10.9.5 :) However, I got the following WARNING constantly running while displaying the graphics:

WARNING: (convert{T})(p::Type{Ptr{T}},a::Array) is deprecated, use convert(p,pointer(a)) instead.
 in depwarn at ./deprecated.jl:40

My first impression (not a criticism:) is that it is very slow and I have no idea yet what to suggest to speed it up. I will have a look at the code and see whether I understand it well enough to make useful suggestions. . . I do suspect there is something about the function visualize that might be improved because that alone took a while to produce the window.

[SimonDanisch]

What part is slow? The loading is incredibly slow, but the rest should be fast. I've noticed, that sometimes, also the camera etc was very slow on the first run, but okay on the second run. But MacOS might be a whole different topic....

2015-01-08 23:47 GMT+01:00 Maximiliano Suster notifications@github.com:

@SimonDanisch https://github.com/SimonDanisch.

I ran your Romeo.jl test and it did work on my Mac OSX 10.9.5 :) However, I got the following WARNING constantly running while displaying the graphics:

WARNING: (convert{T})(p::Type{Ptr{T}},a::Array) is deprecated, use convert(p,pointer(a)) instead. in depwarn at ./deprecated.jl:40

My first impression (not a criticism:) is that it is very slow and I have no idea yet what to suggest to speed it up. I will have a look at the code and see whether I understand it well enough to make useful suggestions. . . I do suspect there is something about the function visualize that might be improved because that alone took a while to produce the window.

— Reply to this email directly or view it on GitHub https://github.com/SimonDanisch/GLPlot.jl/issues/18#issuecomment-69264325 .

[SimonDanisch]

• Colors: Yes sadly, I need to redefine this. I hope this will be merged into the color package in the end. Why? Because than we can use all the functionality from my FixedSizeArray library like img[Red] -> Red channel. Also, things like color arithmetic, or sin(ColorValue) could be implemented just one time. It's a big plus if we don't have to duplicate the code for geometric vectors and color vectors. Also it makes life a lot easier for OpenCL and OpenGL, as I can program against the abstract interface of FixedSizeVector, which has all the information I need. Internally, I'm already doing something like this for OpenGL buffers and textures, but the code is rather ugly without a proper FixedSizeArray library. I hope that @timholy will support this and we can get a tad closer to JuliaLang/julia#7941 . Downside of my current implementation: It needs 0.4 and works best with triangular dispatch (JuliaLang/julia#6984) which is not even implemented yet.
• Dependencies: I'm not sure if I agree. I don't have that many "hard" dependencies, and I broke a lot of things into packages to have a more modular design. If modules are also used inside other packages, chances are higher that bugs get found and fixed. Actually, I'd love to have a largely function based package manager, with every function having its own tests and they can be used completely independent from a module, to enable far better testing and re usability. But that's a whole different topic.
• Image type and declaration: The FixedSizeMatrix is not intended for images. Its only for small transformation matrices and the like. I'd simply use the Julia array type, with e.g. the color FixedSizeVector as elements for images. I'd only use other matrix types for matrices residing on the GPU, but they should implement the same interface as the Julia arrays.

Event handling and coupling display/processing: I think this is the weakest part of the approach you propose right now I must admit that this is correct, its basically non-existent :( In my ideal world with a lot of time, I would re-implement Reactive to do some fancy stuff, but I haven't thought everything through yet. Core elements of the design I've in mind: With lift you basically build a tree like this:

A
|                       \
B = transform(A)        C = transform(A)
|              \
D = transform(B)    E = transform(B)
|
F = transform(D)

Now, with this tree and Julias JIT you can do some crazy things, like actually compiling the whole tree with inlined functions without callbacks, making the event tree really fast. Also we could use the tree to find out, which functions have to be re-executed, if you change the code of a function, enabling us to update the running program correctly. We could probably use meta information, which are also needed for this issue: JuliaLang/julia#265

Also you can identify branches which can be executed in parallel, which could be used for some automated multi-threading. Finally, one can detect if a branch consists only of GPU datatypes and compile parts of the tree entirely to the GPU, getting further speed-ups. This would be quite the dream event system, but I'm not really sure who would have the required time and skill to implement this. I'd work on this, but I've more pressuring things at hand right now :(

Excuse me if things don't make sense, I'm a little bit in a hurry right now. Thanks for any criticisms/ideas and contributions! :)

[maxruby]

As I said, your visualize is slow on my MacOSX. I will check through the entire code to get an idea how it works.

If modules are also used inside other packages, chances are higher that bugs get found and fixed. Actually, I'd love to have a largely function based package manager, with every function having its own tests and they can be used completely independent from a module, to enable far better testing and re usability. But that's a whole different topic.

I can not agree fully with this. I have seen how complicated things can get in VideoIO.jl with many submodules and it becomes very difficult to find and fix bugs without knowing how the packages are organized (especially when containing cross-platform libraries with multiple different versions). However, I agree this can not be generalized. I really like the idea of a package-independent function tester - this would make it much easier to debug in your own package during development.

Q: With the Julias JIT lift tree approach, can you change the lift tree instructions and GUI at runtime?

My rambling here: I envision (in an ideal word) a lisp-like Lush approach to computer vision, where fast command-line instructions can rapidly change a GUI but still the GUI interface can be made as fancy as you like (debug-style graphics to fancy OpenGL) depending on what you are building (instead of the verbose and demanding Qt interface to build GUIs with signals and callbacks, plus the lack of native fast graphics in all systems). Obviously, Lush is not an ideal option since it is not well documented or supported any longer (and its all C and lisp and Linux mainly) - but I think it would be worthwhile examining their design principles.

[SimonDanisch]

Short reply: I intend to have visualize and edit like tostring in java or show in julia, meaning it can be easily used together with a debugger, or by people who dont know anything about the Api. All visualize\edit functions can be specialized with the optional field style=Style{:Default}. So if you want your own visualize/edit function, you can simply implement it like visualize(data, ::Style{:MyStyle}) and now it will spit out a customized visualization. Together with the possibility to define the global default style, this will enable you to rapidly change the complete user experience of your slider/debugger/GUI/IDE.

[maxruby]

Sounds good, will be interesting to see how ::Style{:MyStyle} works out. There is a lot to think about!

[SimonDanisch]

Definitely! By the, I also use style to select a preset of keyword arguments, like you can see in visualize/edit_interface

On Fri, 9 Jan 2015 01:22 Maximiliano Suster notifications@github.com wrote:

Sounds good, will be interesting to see how ::Style{:MyStyle} works out. There is a lot to think about!

— Reply to this email directly or view it on GitHub https://github.com/SimonDanisch/GLPlot.jl/issues/18#issuecomment-69274740 .

[SimonDanisch]

I just saw, that Romeo prints a lot of warnings while running... Could this be why it was slow for you?

[maxruby]

Yes, see comment above.

[SimonDanisch]

Okay, this should be fixed then. I thought you meant that this was a one off warning, and not firing the whole time (which was the case on ubuntu so far).