Suggestions for future GSoCs

[StevenPuttemans]

Just want to point out that it might be interesting to get the opinion of the community into new subjects for GSoC 2017 and collect them somewhere central. We could use this issue for a growing list of suggestions.

Let me start by adding one

• Fixing the terribly broken down video backend (both writer and reader). There is simply no guarantee that stuff works on different systems for now, codecs are poorly supported and the huge amount of issues on the Q&A clearly show we need some fixing in this part of the library.

[PkLab]

:+1: @StevenPuttemans (did you see the new doc ?)

• Another common question on the Q&A is about GUI. In addiction most of Windows user is moving to From Win32 MFC to .NET/WPF or Managed code. More in general the HighGui module is just to play and some wrapper is always needed to get a minimal usable interface, even if Qt, wxWin,...

  Sure GUI is not the core task for OpenCV but GUI is terribly useful or needed. In addiction I think that a comfortable way to use OpenCV with GUI from C#, VB, Delphi,...should exists.

  A crude idea on the fly: instead to open OpenCV to others, encapsulate it and user calling code code into any standard process provided with common interface.

  Thinking at user application = (opencv + user vision code) + user GUI code , we could have:

  • a context (interface, .a/.dll ,...) where user can encapsulate his/her pure C++ code that calls/uses OpenCV
  • any easy/standard way (pipe, shared mem, memfile, socket,...) to exchange informations as params or images with user GUI.

[kinchungwong]

To second @PkLab, I have used multiple processes and shared memory to workaround webcam video issues. I start up one child process which grabs frame from webcam, copy that into shared memory, and use Win32 event to communicate to the parent process that a frame is ready for processing.

With some modest logic, it should be able to implement double buffering, page flipping, and swap chain, etc.

However, I doubt that beginning users will be able to implement or use such mechanisms on their own. There is a minimum level of required knowledge just to use it. In order to make a dedicated helper process, there needs to be a separate executable file (exe) which needs to be bundled with OpenCV and/or the executable that the OpenCV user is developing. This may be too complicated or not acceptable to some users though.

Finally, whenever using blocking synchronization, remember to implement it in a way that abandonment (e.g. the helper process was killed by user or crashed on its own) results in an exception thrown at the waiter, so that it will not become a zombie thread.

It is possible to make a step-by-step tutorial to implement such a webcam grabber helper process.

[kinchungwong]

I just noticed that there is no "official" skeletonization algorithm implementations in OpenCV. There are many competing mathematical definitions for skeletonization, therefore it may be useful if a cv::Algorithm approach can be adopted for all skeletonization algorithms.

[StevenPuttemans]

I just noticed that there is no "official" skeletonization algorithm implementations in OpenCV.

Actually that is not true. About a month or two ago, I added the following thinning algorithm that can be used for skeletization.

[ghost]

While probabilistic graphical model, e.g., bayesian network or MRF have many important vision applications, I did not see such implementations in OpenCV. Will be looking forward to see such a module especially tuned for vision application or fast performance.

[PkLab]

About GUI Sure needed model behind a proper GUI usage with OpenCV is complex and vary as is the IPC subject. From (local)sockets, shared mem, mapped files is not easy to define a standard way. In addiction client platform/language, performance, priority and/or real-time needs, application architecture would require different choices.

Last but not least, technologies change and interfaces should kept up to date. This would require additional effort that is out of scope for OpenCV. Some dedicated library might be a way. For example YAMI4 or ZeroQM look interesting for features and performance even if oversized to the scope.

A simpler approach could be to prepare different tutorials e.g how to show video from OpenCV using .NET, ...using Delphi, ...using...

at the end I quote @mshabunin RFC label

[chrizandr]

Just looking for some clarification here. When you talk about GUI, does you mean the highgui module that is used for displaying images, videos, etc? Or does you mean a new user interface for users to play around, such as that in Matlab, where a lot of the tasks are done through the GUI?

[PkLab]

My idea is

I think that a comfortable way to use OpenCV with GUI from C#, VB, Delphi,...should exists.

It could be a tutorial, or interface, or module to offer

any easy/standard way (pipe, shared mem, memfile, socket,...) to exchange information such as params or images with user GUI.

I mean... between OpenCV + vision user code (C++) and user application GUI (maybe C#, Delphi, ...)

[chrizandr]

@PkLab : Would it be better to use maybe Qt with C++? As far as I know C# and VB use mainly WPF for GUI and there is little support for that in linux or other operating systems. Would it be possible for me to also get some guidance as to how I must approach this problem and maybe start working on it as soon as possible?

[PkLab]

@chrizandr It would be better but C# and WPF are widely used and it would be nice to have a short way to use OpenCV with them.

[chrizandr]

What about python + tkinter? Most of the opencv functionality has python wrappers around them. It shouldn't be a problem to create a good GUI without compromising on functionality right? Correct me if I'm mistaken, I feel that using WPF would constrict the user base to only people who use Windows.

[StevenPuttemans]

I feel that using WPF would constrict the user base to only people who use Windows

I feel your pain, I suggest to use a universal framework also!

[PkLab]

I'm thinking to something different. Computer vision code would be completely separated from the GUI by the way of a well defined communication interface.

In other words I'm talking about the interface: "vision core" from/to any graphical framework or languages.

Maybe the "vision core" (cpp) and GUI (others lang) should runs different processes, (and why not on different machines e.g. like plc vs supervisor).

The interface should transmit/receive data like structs and images. Some tutorial should explain hot to use it with common frameworks like WPF, Gtk, WxWin...

[chrizandr]

I see. Do you have something else that you want this interface to do? Exchanging information would be good, but handling/invoking processes would still largely be dependent on the GUI right?

[HagegeR]

As suggestion goes...

• could stereoCalibrate return rvecs and tvecs of checkerboard as well as per image average pixel error?

• a function like matlab's showExtrinsics could be good too (but might be too difficult with current GUI capabilities)...

I'd like a Stereo Calibration process that gives me similar results to the matlab one: CheckerBoard orientation and distances of the second camera are better guessed in Matlab than in OpenCV, with less user/developer side tinkering.

[sakshamgupta006]

I think it is high time that a state of art monocular SLAM is integrated with OpenCV. I know it is difficult to finish it within GSoC period, probably some prior development will be needed.

[StevenPuttemans]

I know it is difficult to finish it within GSoC period, probably some prior development will be needed.

Actually adding SLAM is on the TODO list for GSoC for a while now, but there has not been a student with enough background knowledge and experience yet to make this succesful within GSoC time... so indeed, if someone could lay out the base before, it might be an achievable goal.

[sakshamgupta006]

@StevenPuttemans I think you should assign someone prior to GSoC, to develop supporting functionalities for SLAM and then continue the remaining project in GSoC. Another option will be to develop SLAM in regular time not involving it in GSoC at all. Will love to work on it be it GSoC or no GSoC.

[StevenPuttemans]

Will love to work on it be it GSoC or no GSoC.

How about using the time from now to next year to start implementing the backend, then apply for GSoC next year to add improvements? This way you will be 1 step ahead of other applicants!

[sakshamgupta006]

@StevenPuttemans That's a great idea. will try to follow through. But to start work I need to know which SLAM to implement? So that I can finalize an architecture and start building supporting functionalities accordingly. Earlier you said that there are no mentors for SLAM, so If you can assign someone now so that we can work alongside and finalize the whole SLAM pipeline.

[StevenPuttemans]

@sakshamgupta006 most mentors are busy now guiding their students. So I am afraid finding you one now will be quite cumbersome.

Also, about the implementation, take a look here about suggestions: https://groups.google.com/forum/#!searchin/opencv-gsoc-2017/slam|sort:relevance/opencv-gsoc-2017/uRRghFW0UBQ/R8TXcuMqBQAJ

[sakshamgupta006]

@StevenPuttemans I have a good working knowledge of most of the monocular SLAM's. Will just need to know which SLAM OpenCV requires. No problem if you cannot assign a mentor till GSoC period as I am pretty swamped with my own GSoC project till then. We can initiate the work once GSoC-2017 ends.

[StevenPuttemans]

Will just need to know which SLAM OpenCV requires.

Actually I am not a core dev of OpenCV, so I might guess the others could clarify that for you. But from what I read in that topic, it seems whatever stable implementation can be provided is better than none.

[sakshamgupta006]

@StevenPuttemans Thanks for the details. Will try to finalize the pipeline for state of art Monocular SLAM and then discuss with you guys about further requirements. It would be really helpful if you can connect me the concerned person from OpenCV.

[StevenPuttemans]

@garybradski this might interest you, @sakshamgupta006 is interested in adding a SLAM implementation and wants some guidance on it.

[sturkmen72]

OpenCV documentation needs some efforts to be better. i think it will be a good idea if a GSOC student take this job next year. i know @Cartucho made some good stuff but there is still much work to be done

[StevenPuttemans]

@sturkmen72, I guess we agree both on this. I am hoping we can get documentation oriented students each year, to improve it step by step. In the meanwhile we can only ask people of the community to further help in improving it :)

[PkLab]

Reading Steven it seems that GSOC can't be considered as "work power" find a student, find a mentor... and maybe is not so attractive task to work on the doc for a GSOC student.

I think this is an effort that should be done by the community and the OpenCV team. This is OT here, maybe this discussion is a better place: https://github.com/LorenaGdL/opencv/issues/1

\cc @LorenaGdL

[StevenPuttemans]

Reading Steven it seems that GSOC can't be considered as "work power" find a student, find a mentor... and maybe is not so attractive task to work on the doc for a GSOC student.

Oh you would be amazed how many documentation applications we had, only they lacked in quality. But there were about 10 students interested in working on this.

I think this is an effort that should be done by the community and the OpenCV team.

I agree, but there will always be the task for bug fixing, updating tutorials, making new tutorials, ... which perfectly suits the purpose of GSoC ;)

[sturkmen72]

@pklab i want to mention @abidrahmank ( Python Tutorials done by Abid Rahmank in 2013). I think he made a great contribution to documentation. why not we expect every year such a good contribitions

[StevenPuttemans]

I think he made a great contribution to documentation. why not we expect every year such a good contribitions

Agreed but then again, the framework backend for multi language tutorials will help us more in the long run. And keep in mind that GSoC has only limited slots, so we need to get the best students in first. If they do not select documentation as topic, then we should respect that. Thats how the spirit of open-source works.

[sturkmen72]

If they do not select documentation as topic, then we should respect that

We ( OpenCV community ) should think of ways to convince them :)

[StevenPuttemans]

Ah well, many of those students focus on GSoC as some sort of project for adding to their curriculum. Then stating you actually contributed functionality, instead of telling, I made code samples and documentation, is prolly a bit better :D

[PkLab]

I agree

project for adding to their curriculum

...so the match are tutorials. We might organize a list of tutorials&mentor to be done. Maybe students can easily select them.

Some of this tutorial can be basic or really advanced opencv how to, or any implementation form papers, or any solution to a real machine/computer vision issue,

[IAmUnbounded]

@sakshamgupta006 I would also like to work on SLAM. Hope we can get in touch to bring SLAM to OpenCV.

[sakshamgupta006]

@sukhad-app I would be delighted to work together on SLAM module, but to kick-start our work we first need to lock down some requirements from OpenCV's side or we can decide those amongst ourselves and initiate the work. What do u suggest?

[IAmUnbounded]

@sakshamgupta006 , I think we can add the functionality to the SFM module OpenCV. I think we should first work on monocular SLAM and try implementing ORB SLAM.

I would like to know what you and OpenCV developers think about it. I would also like to know what dependencies can we use for the development of the project, as it would require libraries like G2o, PBA etc. @StevenPuttemans , @garybradski what do you suggest?

[StevenPuttemans]

I am not the one who can decide on depencies and way to move forward.

Maybe @mshabunin @vpisarev or @alalek can tell us what OpenCV would prefer as SLAM implementation?

[sakshamgupta006]

@sukhad-app I think we should ask @edgarriba for some advice on SLAM as he implemented the SFM and Scene reconstruction modules. @edgarriba can you please help us with some issue's related to SLAM?

[catree]

@HagegeR Maybe a Python sample like this can be used to mimic the Matlab showExtrinsics function?

[terfendail]

There is a request for fingertips detection

[IAmUnbounded]

@terfendail I would be happy to build the requested module. How can we take this forward ?

[terfendail]

@sukhad-app You could create pull request to opencv_contrib with your implementation. This PR could initially contain small piece of the code you going to implement(e.g. detector class interface and/or partial implementation of functionality or particular algorithm implemented as a standalone app) that could be iteratively improved later. Please note that if you're going to participate in GSoC you have to perform some formal actions requested by GSoC Program Rules (participating in GSoC isn't necessary to contribute, but since this thread related to GSoC I think it would be right to mention GSoC rules here) It would be great if final version of PR will contain well documented implementation of detection algorithm(or algorithms), some tests that ensure correctness of the implementation and a sample application that shows basic usage scheme. You could refer to already implemented Face Landmark detector

[kallaballa]

Since GUI often is a topic in this issue check this out: https://github.com/opencv/opencv/issues/22923 A new approach to GUI and (high performance) visualization: docs