EduFdez commented 6 years ago

Initial description

Range and visual sensors are being increasingly used alongside one another in mobile robots. With their increasing use, calibration techniques that can accurately estimate the 6-DoF transformation between them are becoming increasingly important. In this regard, an end-to-end application with an easy-to-use graphical user interface for the extrinsic calibration of different types of sensors is proposed. The app will be able to calibrate the extrinsics of 3D LiDARs, RGB-D cameras, RGB cameras, and any combination between them. Automatic and target-less calibration algorithms based on line matching, plane matching, and trajectory matching will be implemented and integrated into the app. The user will be able to directly visualize the calibration results inside the app and also compare different algorithms wherever possible, and significantly reduce calibration efforts.

See the GSoC project proposal

Progress: All comments below reflect the interactions between the GSoC student and the mentors.

EduFdez commented 6 years ago

Student: @karnikram Main mentor: @EduFdez Backup mentors: @jlblancoc , @jolting

karnikram commented 6 years ago

Hi @EduFdez, @jlblancoc, and @jolting! Thank you so much for this opportunity, I'm really looking forward to this project and working with you guys.

During this bonding period I want to work on the GUI skeleton and get it ready. I'll also be spending time to get more familiar with the mrpt libraries and the coding style guides. Is there anything else that you guys want me to get started on? I also wanted to know about when and how the existing calibration codebases will be made available, so that I can start reading them.

Thank you again!

EduFdez commented 6 years ago

Hi @karnikram , Welcome to MRPT and to GSoC, and congratulations for being selected for this project! I hope it will be very motivating for all of us! Together with @jlblancoc and @jolting we will help you succeed in this project. I agree with you about starting with the coding style guide and the GUI skeleton.

I have been looking at my previous code that I wrote for the experiments that are presented in some of the papers related to this project. This code is divided in different projects, some in Matlab and some in C++, and in some projects it's not very well structured. So I will save you some pain by putting the main resources that you will need into the project. I forked the project from your git, I will add some structure and files and I'll do my pull request to your fork, giving you some indications on where to start, and where to download first datasets to work with. I will come to you soon :-)

Best regards,

karnikram commented 6 years ago

Thanks @EduFdez, looking forward to it! Do let me know if I can help with that in anyway.

EduFdez commented 6 years ago

Hi @karnikram ! I did some updates in my fork of autocalib-sensor-extrinsics to set the project structure, which should help you to organize the code in the future. I also added some "empty" files to start separating different classes. The directory tree is explained in its commit msg: _the core classes for the calibration (calib_core), the GUI squeleton (autocalib-gui), testing individual examples (examples), program unit tests (test), build the doc with doxygen, and search for dependencies with cmake scripts (cmakemodules). I did a pull request to your repository, and feel free to change this structure as you like the most.

You can see that I added some dependencies:

MRPT is a dependency because we'll set the autocalib-sensor-extrinsics as an external application by now
OpenCV will be usefull to process images and extract lines.
PCL can be used to extract planes from depth images or point clouds. This could be changed in the future to reduce the number of dependencies, but we should use it by now to obtain results more quickly.
Boost can be used for programming the unit tests, among other things.

You can download and compile these libraries to start using them in this project.

Then, you can have a look at my unfinished MRPT application for calibrating 2D LiDARs, if look down to the main function you can have an idea of the algorithm. I'll try to split this long file into the classes of autocalib-sensor-extrinsics so that we get soon a basic example to build on.

karnikram commented 6 years ago

Looks neat! I shall get started with it.

karnikram commented 6 years ago

Hi @EduFdez,

I've built and installed the latest mrpt libraries (v-1.9.9) from source but I'm not able to build our project since it includes mrpt-base which I believe is not included in mrpt 1.9.9. Shall I go back to the maintenance branch (v-1.5) or shall I just not include mrpt-base?

I've also been looking into your MRPT application for calibrating 2D LiDARs, it looks interesting, but calibrating 2D LiDARs wasn't actually included in my proposal! We'd only included calibrating 3D LiDARs, RGB-D cameras, RGB cameras, and any combination between them. And out of them I'd wanted to implement Extrinsic calibration of a set of range cameras in 5 seconds without pattern first. So what do you suggest we do here? :)

And also just wanted to update that I haven't been able to access my proposal or any of the other students' proposals on the issue tickets. I guess the links are broken now.

jlblancoc commented 6 years ago

Hi @karnikram !

I've built and installed the latest mrpt libraries (v-1.9.9) from source but I'm not able to build our project since it includes mrpt-base which I believe is not included in mrpt 1.9.9. Shall I go back to the maintenance branch (v-1.5) or shall I just not include mrpt-base?

It's better to move forward to the new mrpt master branch (1.9.9). Please, remove the reference to mrpt-base to the list of libraries that the code really needs. Typically you can find it out by checking the namespaces used by your code. See the full list of libs here: http://mrpt.ual.es/reference/master/

Make sure of reading the guide on porting old code to mrpt-2: http://mrpt.ual.es/reference/master/porting_mrpt2.html

since you'll have to apply it. Let us know if you find further incidences.

karnikram commented 6 years ago

Hi @jlblancoc!

I shall move forward with the latest branch. Until now there has been only one reference to mrpt-base, I shall read the documents you linked for any future conflicts.

One more minor issue, earlier when I ran cmake while building mrpt I got the following warning even though I have vtk6, libvtk6-dev, and python-vtk6 properly installed. Is this expected?

The imported target "vtkRenderingPythonTkWidgets" references the file
   "/usr/lib/x86_64-linux-gnu/libvtkRenderingPythonTkWidgets.so"
but this file does not exist.  Possible reasons include:
* The file was deleted, renamed, or moved to another location.
* An install or uninstall procedure did not complete successfully.
* The installation package was faulty and contained
   "/usr/lib/cmake/vtk-6.2/VTKTargets.cmake"
but not all the files it references.

jlblancoc commented 6 years ago

Yes, that vtk error is "well known"... it can be fixed installing additional packages, but it's harmless anyway so you can safely ignore it.

EduFdez commented 6 years ago

Hi @karnikram !

I've also been looking into your MRPT application for calibrating 2D LiDARs, it looks interesting, but calibrating 2D LiDARs wasn't actually included in my proposal! We'd only included calibrating 3D LiDARs, RGB-D cameras, RGB cameras, and any combination between them. And out of them I'd wanted to implement Extrinsic calibration of a set of range cameras in 5 seconds without pattern first. So what do you suggest we do here? :)

I forgot that 2D LiDAR calibration was finally not included in the proposal. Maybe it could be addressed at the end of this GSoC if there is time for it.

I agree about starting with the calibration of 3D LiDAR / range images. You can have a look at the method PbMapMaker::detectPlanesCloud which contains a wrapper of PCL to extract planes by region growing. Have a look also to the class CFeatureLines that I was using to segment lines from images. I have just done a pull request to include this class in MRPT.

I just set you through "WeTransfer" a rawlog of an RGB-D video from 2 sensors to start working on 3D range calibration (Extrinsic calibration of a set of range cameras in 5 seconds without pattern) and RGB to Depth calibration. The first thing that you could try to do is to extract the 3D planes in this dataset, and then perform data association and display it.

And also just wanted to update that I haven't been able to access my proposal or any of the other students' proposals on the issue tickets. I guess the links are broken now.

The links to the student proposals work for me, can you check again and tell me if there is any broken link? Please.

karnikram commented 6 years ago

Hi @EduFdez!

Thanks for the code files and the dataset, they are interesting.

The first thing that you could try to do is to extract the 3D planes in this dataset, and then perform data association and display it.

Sounds good!

The links to the student proposals work for me, can you check again and tell me if there is any broken link? Please.

I'm still not able to access any of the proposals linked here. I think only the mentors have access to them!

EduFdez commented 6 years ago

I'm still not able to access any of the proposals linked here. I think only the mentors have access to them!

You were right. I have updated the link of your proposal, and we'll update those of other students too.

karnikram commented 6 years ago

@EduFdez, @jlblancoc I've been getting comfortable with the mrpt-libs and the coding style, and have started to code some changes. I shall start pushing my changes in a couple of days.

But I'm having difficulty with one part. After loading the input .rawlog file, I believe the user should be able to inspect the observations and visualize the images, point clouds within the app just like how RawLogViewer does it. Could you give me some pointers about how to go about bringing RawLogViewer's functionality (tree-view of observations, visual preview) into our app? Or is this even the right thing to do?

jlblancoc commented 6 years ago

What I'm doing in all my custom apps (outside of MRPT, using MRPT libs) is to create a CDisplayWindow3D with a COpenGLScene in which all objects (observations, lines, etc.) are rendered using mrpt::opengl objects.

Using overlay text messages (see the methods in CDisplayWindow3D) to show key-based menus, the current state of the app, results, etc. it's a very convenient way of designing a "GUI" pretty fast and without having to work with wxWidgets or Qt.

Alternatively, depending on how much time you want to spend on the GUI, you can create a Qt app (see robot-maps-gui https://github.com/MRPT/mrpt/tree/master/apps/robot-map-gui as a pretty good example), but it might eat your time that should probably better spent in the algorithms themselves...

Cheers,

karnikram commented 6 years ago

Hi @jlblancoc, I've been reading the robot-maps-gui, it's very well designed! Since I already have a start with Qt, I'll continue with it with this as an example. I'm looking to complete the observation tree and the viewer before the coding period starts.

jlblancoc commented 6 years ago

Hi @jlblancoc, I've been reading the robot-maps-gui, it's very well designed! Since I already have a start with Qt, I'll continue with it with this as an example. I'm looking to complete the observation tree and the viewer before the coding period starts.

Great! Recall to commit often, explaining what's your progress. Relatively small commits are normally preferred to huge, bulk one-app-out-of-the-blue commits ;-)

In which repo are you working right now?

karnikram commented 6 years ago

Sure :) I'm working on my fork of autocalib-sensor-extrinsics.

karnikram commented 6 years ago

Hi,

I'm having difficulty in de-serializing the observation objects from the rawlog file.

When I run,

CFileGZInputStream rawlog(file_name);
CSerializable::Ptr obj;

archiveFrom(rawlog) >> obj;

I get the following exception

 =============== MRPT EXCEPTION =============
typename T::Ptr mrpt::serialization::CArchive::ReadObject() [with T = mrpt::serialization::CSerializable; typename T::Ptr = std::shared_ptr<mrpt::serialization::CSerializable>], line 191:
Stored object has class 'CObservation3DRangeScan' which is not registered!:

Aren't all the mrpt classes already registered automatically? Even when I run

mrpt::rtti::getAllRegisteredClasses()

I get no output.

Am I missing something?

EduFdez commented 6 years ago

Hi, I think you need to use a CObservation instead of a CSerializable. You may have a look at the method loadFrame in DifOdometry_Datasets.cpp to see an example of loading observations from a rawlog. There are many more examples that load rawlog within mrpt.

karnikram commented 6 years ago

Hi @EduFdez!

In the example that you've mentioned the rawlog is being loaded in one step. But for our use case I'm actually trying to load the observations one by one and insert them into a tree structure, so that they can be presented easily as a tree view.

I was following this example for loading but it doesn't seem to work..

jlblancoc commented 6 years ago

Your code to deserialize seems correct.

Make sure of adding to cmake's FIND_PACKAGE(MRPT ...), the list of all libraries whose objects you are deserializing, e.g. slam, obs, maps,...

Also: are you compiling mrpt from sources, right? Make sure of not building as static libraries

karnikram commented 6 years ago

Hi @jlblancoc!

I've included most of the modules in my cmake script find_package(MRPT COMPONENTS hwdrivers obs serialization rtti slam maps graphslam pbmap QUIET)

I have also built mrpt from source. I have pushed the code to my fork

jlblancoc commented 6 years ago

Hi,

I've built your code. First thing first: it builds cleanly, which is a really good thing to start!

The list of mrpt libs is correctly set via cmake and it makes its way to the linker:

VERBOSE=1 make 
...
/usr/bin/c++    -std=c++1y  -std=c++17  -O3 -DNDEBUG    CMakeFiles/autocalib_sensor_extrinsics.dir/main.cpp.o CMakeFiles/autocalib_sensor_extrinsics.dir/CMainWindow.cpp.o CMakeFiles/autocalib_sensor_extrinsics.dir/autocalib_sensor_extrinsics_automoc.cpp.o  -o autocalib_sensor_extrinsics  -L/home/jlblanco/code/mrpt/build/lib -rdynamic /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5.5.1 -lmrpt-hwdrivers -lmrpt-comms -lmrpt-maps -lmrpt-gui -lmrpt-vision -lmrpt-obs -lmrpt-opengl -lmrpt-poses -lmrpt-db -lmrpt-tfest -lmrpt-serialization -lmrpt-rtti -lmrpt-core -lmrpt-slam -lmrpt-graphs -lmrpt-graphslam -lmrpt-pbmap -lmrpt-io -lmrpt-img -lmrpt-bayes -lmrpt-system -lmrpt-math -lmrpt-config -lmrpt-containers -lmrpt-random -lmrpt-expr /usr/lib/x86_64-linux-gnu/libQt5Gui.so.5.5.1 /usr/lib/x86_64-linux-gnu/libQt5Core.so.5.5.1 -Wl,-rpath,/home/jlblanco/code/mrpt/build/lib

But the executable does NOT actually depend (so it neither load) any of the "important" mrpt libs, the ones in which the deserialized objects live:

ldd autocalib-gui/autocalib_sensor_extrinsics | grep mrpt
    libmrpt-serialization.so.1.9 => /home/jlblanco/code/mrpt/build/lib/libmrpt-serialization.so.1.9 (0x00007fa7b4c67000)
    libmrpt-io.so.1.9 => /home/jlblanco/code/mrpt/build/lib/libmrpt-io.so.1.9 (0x00007fa7b4a4b000)
    libmrpt-rtti.so.1.9 => /home/jlblanco/code/mrpt/build/lib/libmrpt-rtti.so.1.9 (0x00007fa7b26ef000)
    libmrpt-core.so.1.9 => /home/jlblanco/code/mrpt/build/lib/libmrpt-core.so.1.9 (0x00007fa7b24ec000)
    libmrpt-system.so.1.9 => /home/jlblanco/code/mrpt/build/lib/libmrpt-system.so.1.9 (0x00007fa7b225d000)
    libmrpt-containers.so.1.9 => /home/jlblanco/code/mrpt/build/lib/libmrpt-containers.so.1.9 (0x00007fa7b0248000)

jlblancoc commented 6 years ago

I can't find an obvious reason for this, apart of the usual optimizations of the linker.

For now, just adding one variable like this:

mrpt::obs::CObservation3DRangeScan o;

Anywhere (e.g. at the global scope of the GUI app) just fixes the problem. Give it a try, run ldd again (it worked for me and should for you as well) and try the serialization again. I think it will work now.

karnikram commented 6 years ago

Hi @jlblancoc,

Thank you so much, it works now!

karnikram commented 6 years ago

Update: I'm more or less done with all the GUI elements. The app can now load a rawlog file and display all its entries as a tree. I've added two pcl visualizers for each of the two input sensors and one for the result.

status

Hope it's okay. The changes are in my fork.

I'm now re-reading your paper on fast extrinsic calibration of rgbd sensors using planes, and PbMap to implement plane extraction and association.

EduFdez commented 6 years ago

Hi, I'm glad that you solved the compilation problems and the GUI is in a good track. I am taking some holidays now and I cannot get internet all the time, but I am online at least once every 2 days. I will have a look at your code soon to give you some feedback.

karnikram commented 6 years ago

Hi @EduFdez @jlblancoc I've been trying to implement plane segmentation using pcl::OrganizedMultiPlaneSegmentation like in PbMap. But this method requires an organized pointcloud which I'm not able to get after converting CObservation3DRangeScan using mrpt::maps::CPointsMap::getPCLPointCloud.

Shall I do this conversion by myself or is there any other existing method that I can use?

jlblancoc commented 6 years ago

Hi, You can set MAKE_DENSE to true in the T3DPointsProjectionParams argument to CObservation3DRangeScan::project3DPointsFromDepthImageInto to obtain an organized pointcloud.

karnikram commented 6 years ago

Hi @jlblancoc, @EduFdez

It's still not working for me. I tried with MAKE_DENSE equal to both true and false. I've pushed the changes to my fork, would be glad if you could have a look at it.

The error causing line is this. The conversion is done in this line.

Update : I've also setup a blog post summarizing all the work that I've done so far here. I plan to update it every week.

Thanks!

jlblancoc commented 6 years ago

I've being revising the related code, and perhaps the MAKE_DENSE flag is not properly "exported" to the PCL object.

Please, consider experimenting yourself to find a solution given these pointers:

doc on organized point clouds in PCL.
The actual implementation of the project3D...() function is in the template do_project_3d_pointcloud() in the file CObservation3DRangeScan_project3D_impl.h. The pca variable is a template whose specialization for this case is in PCL_adapters.h, a header you know since you include it.
Please, think if it might be needed to add a new "virtual" (not actually virtual since it's all based on templates, but you get my point) method to PointCloudAdapter<> to enable/disable the "dense" mode so we have a chance to propagate it to the underlying PCL structure.

I think that should be the single most important missing piece, but double check it... probably would also need to add a resize(W,H) as an alternative to the current resize(W*H).

EduFdez commented 6 years ago

Hi, In order to check that the PCL point cloud is actually organized you can check its parameters: is_dense, width and height. Please, make sure that these are correct. If this error is not solved, you may try to save the point cloud in .pcd format and experiment with the PCL example _pcl_pcd_organized_multi_planesegmentation Let us know what you get, please.

PS. Sorry for not being able to reply more quickly, I am taking some holidays abroad and I cannot get connected as often as I would like.

karnikram commented 6 years ago

@jlblancoc @EduFdez I've been trying to fix the problem inside do_project_3d_pointcloud(). But meanwhile I've run into another build issue.

I want to pass a pointer to the viewer ui to the core plane matching algorithm so that it can display the status periodically inside the viewer. For this I need to include ui_CViewerContainer.h inside core/CPlaneMatching.h as #include<gui/ui_CViewerContainer.h> since it is located in the gui subdirectory of the project build directory.

But this becomes a problem when I include CPlaneMatching.h inside any source file in the gui directory since it only knows ui_CViewerContainer.h and not gui/ui_CViewerContainer.h. How do I solve this 'cyclic' inclusion problem? Can I control the location of the generated ui_*.h files to some common directory?

And when I start with a clean build sometimes ui_CViewerContainer.h has not even been created yet so #include<gui/ui_CViewerContainer.h> inside core/CPlaneMatching.h throws an error. How do I control the order of compilation?

Would be grateful for some pointers to solve these in CMake.

EduFdez commented 6 years ago

The cyclic inclusion issues arise often when working with interfaces. You can solve this managing the communication in the main class which has access to both your GUI and your PlaneMatching object. So I would create an interface class to address the communication outside your GUI folder. In this way you can avoid also the problem of your error which is thrown.

Tell me if I'm not clear on my answer, I am writing with my phone without seeing your code.

On Sat, Jun 9, 2018, 6:06 AM Karnik Ram notifications@github.com wrote:

@jlblancoc https://github.com/jlblancoc @EduFdez https://github.com/EduFdez I've been trying to fix the problem inside do_project_3d_pointcloud(). But meanwhile I've run into another build issue.

I want to pass a pointer to the viewer ui to the core plane matching algorithm so that it can display the status periodically inside the viewer. For this I need to include ui_CViewerContainer.h inside core/CPlaneMatching.h as #include<gui/ui_CViewerContainer.h> since it is located in the gui subdirectory of the project build directory.

But this becomes a problem when I include CPlaneMatching.h inside any source file in the gui directory since it only knows ui_CViewerContainer.h and not gui/uiCViewerContainer.h. How do I solve this 'cyclic' inclusion problem? Can I control the location of the generated ui*.h files to some common directory?

And when I start with a clean build sometimes ui_CViewerContainer.h has not even been created yet so #include<gui/ui_CViewerContainer.h> inside core/CPlaneMatching.h throws an error. How do I control the order of compilation?

Would be grateful for some pointers to solve these in CMake.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/MRPT/GSoC2018-discussions/issues/2#issuecomment-395960547, or mute the thread https://github.com/notifications/unsubscribe-auth/AEOs4Ri2MLqvg-o9B9-3TR_cfklhsnvEks5t66wpgaJpZM4TgWps .

karnikram commented 6 years ago

Thanks for the reply @EduFdez.

I don't fully understand. Should I declare an interface with pure virtual functions for all the communications (like updating the text box) inside a header file outside the gui folder? Then I should include this header file in all the classes that need this communication and override the virtual functions. But won't the problems still remain?

Is there any example I could refer, please?

Alternatively, I could simply use forward declaration to solve the inclusion issue but that would still depend on all the ui_*.h files to be built first.

jlblancoc commented 6 years ago

For this I need to include ui_CViewerContainer.h inside core/CPlaneMatching.h

Not really. When this problem arises, it normally means that there're other better ways to refactor the related code, but as the fastest solution, you can avoid adding the #include by using forward declaration in the .h and moving the real #include to the .cpp where it cannot cause harm.

See: https://stackoverflow.com/a/4757718/1631514

karnikram commented 6 years ago

Thanks @jlblancoc. But this is still a problem when I start with a clean build where all the ui_*.h files haven't even been built yet. So #include <ui_CViewerContainer.h> inside the .cpp would still throw an error when the cpp gets compiled before the ui files. Is there a way I can ensure that the ui files are compiled first before the other files get compiled?

jlblancoc commented 6 years ago

I don't fully understand where's exactly your need to use the class defined in the ui_xxx.h, please post a link to the exact line/function for me to get an exact idea of the problem.

Anyway: if you only want to use it to "notify about progress" (if I understood it right), you could probably refactor things to accept a std::function<> object, which you can call to notify on changes. Then, the ui_xxx.cpp could create the std::function<> using binding, etc. to have a reference to the ui_xxx.cpp object implicitly.

karnikram commented 6 years ago

This is the line. I just need to be able to notify about the calibration progress inside the viewer text box.

std::function<>seems like a better option, I shall look into it.

jlblancoc commented 6 years ago

This is the line.

Yes, please, refactor it so the user can plug in any std::function to get notifications... in this way the algorithm knows nothing about the UI, as it should be ;-)

karnikram commented 6 years ago

I solved the inclusion problem using std::function, and I've almost solved the conversion to organized point cloud problem too by making changes to PCL_adapters.h. But before I can test it I have a new problem :)

I fetched the latest changes from the mrpt repo and built it again, and since then I haven't been able to build my project. I get usr/local/include/mrpt/serialization/include/mrpt/serialization/serialization_frwds.:11: Parse error at "mrpt"

I believe it is not able to parse the new C++17 nested namespace. I've upgraded cmake and set CMAKE_CXX_STANDARD to 17 and I've also updated my gcc to 7.3.0 using this link. Is there any other change I need to make or is the error because of something else?

jlblancoc commented 6 years ago

usr/local/

Please, refer to these comments I made elsewhere about the recommendation for NOT doing "make install": https://stackoverflow.com/a/45806080/1631514

karnikram commented 6 years ago

Thanks @jlblancoc,

I removed all the mrpt files from /usr/local and did a clean build. I set MRPT_DIR to my build folder but I get the same error home/karnik/Tools/mrpt/libs/serialization/include/mrpt/serialization/serialization_frwds.:11: Parse error at "mrpt"

Not sure what's wrong..

jlblancoc commented 6 years ago

Hmmm... please, attach your MRPT_BUILD_DIR/CMakeCache.txt file...

karnikram commented 6 years ago

/home/karnik/Tools/mrpt/build/CMakeCache.txt

jlblancoc commented 6 years ago

ok, I see that you did the update-alternative thing such that a plain gcc --version shows it's gcc-7, right?

When does the error happen, building mrpt, or your app?

karnikram commented 6 years ago

gcc --version gives 7.3.0 This error happens when I build my app..

jlblancoc commented 6 years ago

Remove these lines (we don't support old versions of the compiler anyway...) and put on the top of the file, for example, after the PROJECT(...), this:

        set(CMAKE_CXX_STANDARD 17)

karnikram commented 6 years ago

Thanks @jlblancoc

But as mentioned in my previous comments I had already tried that.. Sorry for not being more clear. This is my updated project CMakeLists.txt

The other CMakeLists.txt scripts in my project subdirectories are the same.

MRPT / GSoC2018-discussions

GUI App for Automatic Extrinsic Calibration of Range and Visual Sensors (Karnik Ram R) #2

Initial description