V4D - A High performance visualization module with GUI

[kallaballa]

Describe the feature and motivation

I am writing a high performance visualization module. It should support OpenGL, 2D vector graphics (through NanoVG) a GUI system (based on ImGui) and hardware acceleration whenever it can (e.g.: OpenGL, CL-GL interop and CL-VA interop). Also drawing on top of streaming videos should be supported. It must support WebAssembly. I would also like to support Vulkan, DirectX and Metal - through ANGLE.

I have released an alpha version of the module and the documentation.

At the moment it depends on pull requests that haven't been merged yet but could work without, just slower: #22780 and #22704

What do you think?

[kallaballa]

https://user-images.githubusercontent.com/287266/208235640-82e67a28-91df-41c3-ac21-4d3a47555802.mp4

A screen capture of the full example as it looks now.

Video credits in the README

[kallaballa]

A WebAssembly/WebGL application based on nanogui

[kallaballa]

I redesigned it into a simple object model.

https://viel-zu.org/opencv/apidoc/

Notes:

• A lot of things still not use OpenCV types and ways
• To retain NanoVG functionality I consider creating a OpenCV-wrapper for it in its own namespace. Similar to cv::ogl
• I still have to write a demo without CL-interop
• The member functions makeVAcapture(), makeVAwriter(), captureVA() and writeVA() are only required till #22780 is resolved.

Is see large potential to go beyond the highgui API but I guess one of the next things to do is to implement my version of namedWindow and imshow.

[kallaballa]

After reading some more I think this fits much better into cv::viz.

[kallaballa]

I have a first version of a wrapper for NanoVG. There are still some things to do (like using cv::String everywhere and cv::Mat for the transform functions) but it basically works. all the functions inside namespace kb::nvg should only be used inside kb::Viz2D::nanovg(). Still a lot of testing needed.

[mshabunin]

I'm wondering why is it necessary to wrap nanovg? Isn't it easier to use this library directly and maybe provide small compatibility header from OpenCV side?

highgui module purpose is to provide cross-platform interface for very basic GUI applications, probably because there were no appropriate existing libraries at the time of its creation.

viz module has been moved to the _opencvcontrib (#13069) because it was considered that this module is not useful for majority of users and sometimes adds build problems due to its dependency on VTK.

Also there is ovis module wrapping Ogre3d library (cross-platform, HW-accelerated).

[kallaballa]

The wrapping of nanovg might be questionable but I just found it natural. I am trying to get some guidance on the whole idea. :) You don't think it is worth it?

[kallaballa]

The wrapping of nanovg might be questionable but I just found it natural. I am trying to get some guidance on the whole idea. :) You don't think it is worth it?

Anyway, they are a few good reasons to wrap NanoVG. NanoVG requires OpenGL which involves a complex setup (especially cross-platform). By wrapping it that initialization could be required (e.g. by exception) or even performed on use. My motivation behind wrapping NanoVG is to bring hardware accelerated 2D Vector graphics to the opencv user in an easy manner. That's also why the wrapper (mostly) provides an interface using OpenCV types and behaviour (e.g. colorspace conversions and the definition of hue).

At the moment the use of NanoVG is bound to a kb::viz::Viz2object which ensures everything is setup right and CL/GL interop can be used in conjunction with it seamlessly. Btw. what was the motivation behind the cv::ogl wrapper?

[kallaballa]

https://user-images.githubusercontent.com/287266/208972149-c6615ae0-470a-4d6c-886a-9c6d43f9174a.mp4

New features: Multiple light- and heavy-weight windows, zooming, feeding multiples windows from one frame buffer and a couple of little things.

https://github.com/kallaballa/GCV/blob/main/src/optflow/optflow-demo.cpp

[kallaballa]

The wrapping of nanovg might be questionable but I just found it natural. I am trying to get some guidance on the whole idea. :) You don't think it is worth it?

Anyway, they are a few good reasons to wrap NanoVG. NanoVG requires OpenGL which involves a complex setup (especially cross-platform). By wrapping it that initialization could be required (e.g. by exception) or even performed on use. My motivation behind wrapping NanoVG is to bring hardware accelerated 2D Vector graphics to the opencv user in an easy manner. That's also why the wrapper (mostly) provides an interface using OpenCV types and behaviour (e.g. colorspace conversions and the definition of hue).

At the moment the use of NanoVG is bound to a kb::viz::Viz2object which ensures everything is setup right and CL/GL interop can be used in conjunction with it seamlessly. Btw. what was the motivation behind the cv::ogl wrapper?

Another important point is resilience against API changes in NanoVG

[kallaballa]

updated the skeleton api docs

[kallaballa]

Anyway there is still much to do but I am hesitant about which direction this project should go. Pushing it to contrib at start and re-evaluating based on user acceptance later would be fine for me. I believe this may become very useful to the user and then it doesn't matter if it's in contrib for starters. also if it would stay for various reasons (like dependencies) in contrib that's also an acceptable outcome to me. As long as the user has a decent way of getting hands on that code. I actually thought about documenting it by adapting existing OpenCV tutorials to it one by one. And there is of course the challenge of the minimal source commenting at the moment. :)

If OpenCV decides it doesn't want the module at all I would still continue developing it but the project would probably take quite another direction.

Would be nice to gain more clarity on that.

[kallaballa]

I ported the Viz2D library and one of the simpler demos (no input video required) to WebAssembly

https://user-images.githubusercontent.com/287266/209198199-2a4ee58c-8513-4fa6-bb44-5cdd55cc04ae.mp4

[kallaballa]

A more complete WASM build of the font-demo.

• High-DPI support works even in the browser.
• Coding as well as using the GUI feels good.
• No more html5 frontends! (except for special cases :))

Next I'll see how much more performance I can squeeze out of it, but I don't think It'll be much.

https://user-images.githubusercontent.com/287266/209398854-bf0827c8-a854-4aba-bbfa-925aeae47053.mp4

[kallaballa]

I am working right now on a wasm port of the optflow demo. I think because of no decoding and encoding overhead (I get raw images from html5 and I won't write a video for starters), it might run decently at maybe even 720p15+? Maybe more if I tweak visual settings.

[kallaballa]

An early version of the wasm port of the optflow-demo: https://viel-zu.org/opencv/optflow/index.php I use Firefox 107.0.1 for testing. You have to press the "start capture" button to activate camera input.

[kallaballa]

Now the wasm build should work for most modern browsers.

[kallaballa]

Improved the font-demo and put it online: https://viel-zu.org/opencv/font/

[mshabunin]

Pushing it to contrib at start and re-evaluating based on user acceptance later would be fine for me.

Yes, I think it would be fine.

[kallaballa]

alright. will take some time.,. in the meantime do you have an example for an excellent contrib-module to use as a template?

[kallaballa]

anyway, i finished two additional demos:

Shader Demo

https://user-images.githubusercontent.com/287266/222944648-05774cc6-24ed-4d3d-ac19-53d60fe21db2.mp4

Beauty Demo

https://user-images.githubusercontent.com/287266/222971754-f1fc141f-453e-408e-ae35-addeeba625e6.mp4

I ported both to WASM

https://viel-zu.org/opencv/shader/ https://viel-zu.org/opencv/beauty/

[kallaballa]

Ported the beautification demo to WASM and added a GUI: https://viel-zu.org/opencv/beauty/

[kallaballa]

I finished the last demo: Pedestrian detection.

https://user-images.githubusercontent.com/287266/222980241-d631f7e5-e7a3-446e-937e-bce34e194bd1.mp4

[kallaballa]

Renamed the project to Viz2D and included instructions for Ubuntu 22.04.2 LTS in the README.

[mshabunin]

do you have an example for an excellent contrib-module to use as a template?

Perhaps ovis is similar to your module. General guidance is here: https://github.com/opencv/opencv/wiki/Coding_Style_Guide and documentation writing guide is here: https://docs.opencv.org/4.x/d4/db1/tutorial_documentation.html#tutorial_documentation_quick_start

[kallaballa]

thx!

[kallaballa]

Progress:

• Transparent hardware acceleration: CL-GL sharing and VAAPI are automatically detected and therefor are now optional
• The only hardware capability required is either OpenGL 3.2 compat or OpenGL ES 3.0 core
• Improved WebAssembly support

[kallaballa]

https://github.com/opencv/opencv/wiki/Coding_Style_Guide

This guide mentions C++11. Is this still a hard requirement? I read the C++11 will be dropped soon. Also, aren't there modules that use C++20? I am asking because at the moment Viz2D is using C++20.

[kallaballa]

From now on my efforts to port Viz2D to a OpenCV-module reside in a seperate branch: https://github.com/kallaballa/Viz2D/tree/opencv-module

[kallaballa]

From now on my efforts to port Viz2D to a OpenCV-module reside in a seperate branch: https://github.com/kallaballa/Viz2D/tree/opencv-module

In that branch i already changed the namespace to cv::viz and changed the license to Apache2. I also wrote source documentation for all public functions (detail namespace excluded): https://viel-zu.org/opencv/apidoc/namespaces.html

[kallaballa]

I finished converting the project into an OpenCV module, so i merged back to main and updated the build instructions in the README. The build requires C++20 at the moment. That said, there is still a lot to do.

[mshabunin]

This guide mentions C++11. Is this still a hard requirement?

Yes, we will stick to C++11 in 4.x branch at least. For 5.x min standard version might be raised to C++14 or C++17, not sure regarding C++20.

Perhaps you can check C++ version in your module cmake and disable itself if compiler does not support C++20. Similar check has been implemented in the cvv module: https://github.com/opencv/opencv_contrib/blob/6ebb38db6d0504682d72d168c4bba13a6b70895c/modules/cvv/CMakeLists.txt#L1-L4

[kallaballa]

Thx! I'll check how much effort it would be to port to C++11.

[kallaballa]

I heavily extended documentation (for the moment) in the README. I intend to write tutorials based on the samples

[kallaballa]

Thx! I'll check how much effort it would be to port to C++11.

NanoGUI needs C++20 so for the moment i will stick to it and check for the C++ version as suggested.

[kallaballa]

I converted the documentation to an OpenCV module-documentation and heavily extended it including tutorials, samples, images and videos. https://viel-zu.org/opencv/doxygen/html/dc/d7b/viz2d.html

Looks good?

[kallaballa]

There is another thing. Since Viz2D supports OpenGL the name doesn't fit. How about renaming it to cv::viz::Vizual?

[kallaballa]

Ported from OpenGL 4.6 Compatibility to OpenGL 3.2 Compatibility. The compatibility profile is only required for some of the samples that use legacy OpenGL. Viz2D itself could work with OpenGL 3.0 core.

[kallaballa]

Implemented (transparent) parallel capturing/writing for a considerable performance boost.

[kallaballa]

This guide mentions C++11. Is this still a hard requirement?

Yes, we will stick to C++11 in 4.x branch at least. For 5.x min standard version might be raised to C++14 or C++17, not sure regarding C++20.

Perhaps you can check C++ version in your module cmake and disable itself if compiler does not support C++20. Similar check has been implemented in the cvv module: https://github.com/opencv/opencv_contrib/blob/6ebb38db6d0504682d72d168c4bba13a6b70895c/modules/cvv/CMakeLists.txt#L1-L4

I think i found a good solution that only turns on C++20 for Viz2D while other code compiles as C++11.

First i check for C++20 support and disable the module if it isn't found:

get_property(known_features GLOBAL PROPERTY CMAKE_CXX_KNOWN_FEATURES)
list (FIND known_features "cxx_std_20" idx)
if (${idx} LESS 0)
  message(STATUS "Module opencv_viz2d disabled because it requires C++20")
  ocv_module_disable(viz2d)
endif()

If C++20 is available I enable it using target_compile_features which turns on all required flags for a feature.

target_compile_features(opencv_viz2d PRIVATE cxx_std_20)

No need to build all of OpenCV as C++20.

[kallaballa]

I'm not sure that's the way to do things but for now I included nanogui (which includes nanovg) as a submodule and adjusted the build instructions

[kallaballa]

Viz2D only depends on core modules while the samples depend on additional contrib modules. How and when should I enable building of samples and introduce the dependencies?

[kallaballa]

Also at which point should i request a review? It's not perfect but it is a working OpenCV module. Could use some guidance.

[kallaballa]

I integrated https://github.com/bloomen/cxxpool which is a MIT header-only implementation. Is that alright?

[kallaballa]

Added a new tutorial about custom video sources and sinks: https://viel-zu.org/opencv/doxygen/html/de/d82/v4d_custom_source_and_sink.html

[kallaballa]

I renamed the project to V4D and created a PR draft: https://github.com/opencv/opencv_contrib/pull/3471

[kallaballa]

What kind of tests should i write? performance?

[mshabunin]

Viz2D only depends on core modules while the samples depend on additional contrib modules. How and when should I enable building of samples and introduce the dependencies?

Something like this: https://github.com/opencv/opencv_contrib/blob/298fc7be958c10be8a8e807cf9a0dde512c13af5/modules/ovis/CMakeLists.txt#L27

I integrated https://github.com/bloomen/cxxpool which is a MIT header-only implementation. Is that alright?

OpenCV already have it's own thread pool (cv::parallel_for), it is better to reuse it if possible. Adding more threads can result in conflicts, oversubscription and performance degradation.

What kind of tests should i write? performance?

Some basic accuracy tests verifying module functionality. E.g. creating windows, showing images, etc.. Performance tests can be added in case if you expect some functions to have optimized implementation or for comparing between various HW/SW configurations.

[kallaballa]

Viz2D only depends on core modules while the samples depend on additional contrib modules. How and when should I enable building of samples and introduce the dependencies?

Something like this: https://github.com/opencv/opencv_contrib/blob/298fc7be958c10be8a8e807cf9a0dde512c13af5/modules/ovis/CMakeLists.txt#L27

I integrated https://github.com/bloomen/cxxpool which is a MIT header-only implementation. Is that alright?

OpenCV already have it's own thread pool (cv::parallel_for), it is better to reuse it if possible. Adding more threads can result in conflicts, oversubscription and performance degradation.

What kind of tests should i write? performance?

Some basic accuracy tests verifying module functionality. E.g. creating windows, showing images, etc.. Performance tests can be added in case if you expect some functions to have optimized implementation or for comparing between various HW/SW configurations.

Thx!

[kallaballa]

I wrote a tutorial on how to build WebAssembly/WebGL2 applications using V4D: https://viel-zu.org/opencv/doxygen/html/d5/d3c/v4d_webassembly_support.html

When BUILD_EXAMPLES is activated in a WASM build all the V4D-examples and demos are build. I will make the interactive ones part of the tutorial.