MRPT / GSoC2017-discussions

See README
1 stars 0 forks source link

Algorithm improvements to KD-Tree module: applications to ICP (Pranjal Kumar Rai) #1

Closed jlblancoc closed 6 years ago

jlblancoc commented 7 years ago

Initial description

Nanoflann is a C++ library provided by MRPT for building KD-Trees. The library is a fork of the widely-used FLANN C++ library and is mostly optimized for lower dimensional point clouds. Most ICP algorithms for robot SLAM and localization require finding nearest neighbor in 2D/3D point clouds. Support for dynamic point clouds and nearest neighbour query in non flat spaces are two paramount features which are missing in the current nanoflann library.

See attached PDF.

5589797390254080_1491229396_MRPT_GSOC_Proposal.pdf

Final results:

jlblancoc commented 7 years ago

Student: @pranjal-rai

Main mentor: @jlblancoc Backup mentors: @feroze

pranjal-rai commented 7 years ago

Hi @jlblancoc and @feroze, Thank you for giving me the opportunity to work on this project. I am planning to start reading the existing code of nanoflann from the next week. Please share any resource/pointer which might be helpful in understanding the library.

jlblancoc commented 7 years ago

👍 !!

Since this project will be all about efficiency, I would start preparing, in your repo/fork, one subdirectory with a suite of benchmarks: executable programs that repeat certain operations and measure the time ellapsed, in microseconds or milliseconds; typically, one operation is repeated N times, then the time divided by N to have a better estimate. In our case, tasks to be benchmarked are, at least, building a KD-tree, and querying it for some random points. A couple pointcloud datasets could be added from some public repos as test cases.

You can take a look at the sources of mrpt-performance for "inspiration", but that is probably too complex for what we need here, which should be kept as simple as possible ;-)

pranjal-rai commented 7 years ago

While playing with the code I found a possible bug, I have created a pull request.

jlblancoc commented 7 years ago

Great! I've just answered you there.

pranjal-rai commented 7 years ago

@jlblancoc

jlblancoc commented 7 years ago

I think that the current KD-Tree index class should remain more or less as is, and tailored to static datasets. A new class should be created for the dynamic algorithm, since, if I recall it right, it requires having an array/vector of kd-tree indices, so this later class would use "composition" with the former one.

On the two approaches: I would prefer having this new class being unique. Then having some enum of possible alternatives. I think this way will be the best in terms of code reuse. We would change this decision later on if it seems impractical, but in principle, I always prefer having one single code whose behavior can be modified simply via a parameter/variable, without code changes.

We need executables to measure the time and validate correctness of the dynamic insertion approaches because the current tests do not handle dynamic clouds.

Yes, sure!

jlblancoc commented 7 years ago

Hi @pranjal-rai !

How are you doing with the preliminary work on the datasets and/or performance tests? Do you need any help?

PS: Please, keep your fork up-to-date to the latest accepted pull-requests in the main nanoflann repo.

pranjal-rai commented 7 years ago

Hi @jlblancoc

I forked flann and experimented with the code a bit. I noticed that there are two codes of our interest at this time.

I quickly looked through the tree building procedure of both the codes. I think they are using different approaches for building and searching the tree. I coded the rebuild threshold approach for nanoflann but I found some cases where the proposed algorithm fails for nanoflann but passes for the general flann code. I am not sure if rebuild threshold approach can be applied to nanoflann. Right now I am stuck with this part.

I also wrote a rough code for logarithmic approach, the performance of this method is much better than the one in the general flann code for large number of dynamic insertions.

I have updated my forked repository.

pranjal-rai commented 7 years ago

Hi @jlblancoc

I have implemented logarithmic approach for dynamic support. I have created a new branch with proposed changes. I have tested it using random point clouds. For now I have added just one example file, I will add more once you review it.

pranjal-rai commented 7 years ago

Hi @jlblancoc

I compared the index building time requirement for nanoflann and flann for inserting points dynamically. Flann is based on the rebuild threshold approach whereas nanoflann uses logarithmic approach.

plot

pranjal-rai commented 7 years ago

Hi @jlblancoc

Is there any comment on the logarithmic approach implementation? I have experimented with the following four libraries to compare their performance with nanoflann in later phase.

Out of these only FLANN supports dynamic insertions/deletions. So I plan to illustrate the dynamic support comparison only between FLANN and nanoflann. I will illustrate other comparison results for all the four libraries along with nanoflann.

jlblancoc commented 7 years ago

Hi! Sorry for the gap... two weeks ago my second baby daughter was born so these have been really busy days! ;-)

You made good progress, congrats! If you go on at this pace, I'm sure we'll achieve the project goals.

Some quick answers:

I compared the index building time requirement for nanoflann and flann for inserting points dynamically. Flann is based on the rebuild threshold approach whereas nanoflann uses logarithmic approach. (+ the figure)

Good results! If you are using octave or matlab for plotting, I recommend using something like: plot(x,y,'r-'); hold on; plot (x,y,'r.'); so we can see "dots" on each data point and clearly see how many samples do those graphs contain.

Another issue is: inserting points one by one is a good test case in general, but in mobile robotics it's far more common to insert them in batches of 100s to 1000s (for 2D or 3D LiDAR scanners, respectively). Probably the general trend of the graphs won't change but we will be reporting more realistic total time costs (many seconds seem too much, probably that's only because of the 1-by-1 approach). Give it a try to see how much results are affected...

Is there any comment on the logarithmic approach implementation?

I haven't compiled and tested it, just visually inspected the code for now.

Firstly, the approach of defining the new classes seems reasonable. :+1: However, I can't easily tell how much of the code in the new classes is identical to the static index version. How much would you say it is? I'm asking because code duplication is evil and a source of maintenance nightmares and bugs! What do you think about refacting the shared parts of static & dynamic in one common shared base class/struct? Perhaps it's not possible for some reason and you discarded it, let me know.

In any case, common code could be refactored anyway via the CRTP. Let me know if you want to learn more on this method.

One more thing: The vector here is expected to grow as new points are added, right? When we move to the range of millions of points, resizing a std::vector has an important cost due to reallocations. Consider using std::deque instead. Try running a benchmark using both options to see if it makes any significant difference but, unless std::deque ends up being worse in experimental measurements, we should prefer it over std::vector.

Also: in connection with the robotics applications, please consider adding an alternative addPoints() if that helps efficiency vs. calling N times addPoint() since, normally, maps are updated with batches of many points at once.

Out of these only FLANN supports dynamic insertions/deletions. So I plan to illustrate the dynamic support comparison only between FLANN and nanoflann. I will illustrate other comparison results for all the four libraries along with nanoflann.

:+1:

pranjal-rai commented 7 years ago

@jlblancoc thanks for the pointers. I will start working on all the pointers and get them done ASAP.

pranjal-rai commented 7 years ago

Hi @jlblancoc

Another issue is: inserting points one by one is a good test case in general, but in mobile robotics it's far more common to insert them in batches of 100s to 1000s (for 2D or 3D LiDAR scanners, respectively). plot

The results change significantly when points are added in chunks of size 100, our time is nearly the same because as per theory the points in logarithmic approach are still added one by one but in flann time improves because now it requires much lesser number of rebuilds.

Consider using std::deque instead. Try running a benchmark using both options to see if it makes any significant difference but, unless std::deque ends up being worse in experimental measurements, we should prefer it over std::vector.

I compared the time taken for std::deque and std::vector. The latter performs slightly better. I am not sure about the reason for this.

Also: in connection with the robotics applications, please consider adding an alternative addPoints() if that helps efficiency vs. calling N times addPoint() since, normally, maps are updated with batches of many points at once.

I have removed addPoint and added a new function addPoints to insert multiple elements at a time. I have committed the changes in my branch.

However, I can't easily tell how much of the code in the new classes is identical to the static index version. How much would you say it is?

Most of the functions are common between the two classes. I duplicated the functions as I had to modify some of the existing functions to include deletion support because this feature was missing in static index adaptor. I am still working on this part and I will try to create one common shared base class for the functions which are exactly same.

Rebuild Threshold Approach

Regarding this method, is there any paper/resource which validates its correctness. I coded this part but it fails for some cases. The failure itself seems genuine. The thing is when a new point is added dynamically without rebuilding the index it is not assured that it will be at the same location in the tree where it would have been if the index was rebuilt from scratch. So when we perform the nearest neighbour query on the tree this particular branch which contains the dynamically inserted point might get ignored even though it was closest to the query point. I came across many such examples.

jlblancoc commented 7 years ago

Good! Lots of results in a short time.

The results change significantly when points are added in chunks of size 100, our time is nearly the same because as per theory the points in logarithmic approach are still added one by one but in flann time

What method of flann are you comparing to? The "A"NN stands for approximate, so that may be also the answer to:

Rebuild Threshold Approach

Regarding this method, is there any paper/resource which validates its correctness. I coded this part but it fails for some cases. The failure itself seems genuine.

I don't have right now any reference at hand, but thresholding may be valid only to generate approximate trees. In that case, we can't compare an "exact vs approximate" algorithms, at least, not without making clear that they are different.

I will try to create one common shared base class for the functions which are exactly same.

👍 Remember trying to use CRTP to avoid virtual functions, which may (depends on compiler and many details) come with a runtime cost.

pranjal-rai commented 7 years ago

Hi @jlblancoc

I don't have right now any reference at hand, but thresholding may be valid only to generate approximate trees.

Even I think that rebuild threshold method can only be applied to approximate nearest neighbour searches.

pranjal-rai commented 7 years ago

Hi @jlblancoc

jlblancoc commented 7 years ago

Hi @pranjal-rai !

Apart of a comment I made regarding the documentation of classes, I'm happy with your approach! No virtual methods means efficient code... 👍

I have coded up both the approaches

Sorry, I missed something... what two methods? Logarithm method and threshold?

I still think that rebuild threshold approach works for approximate nearest neighbour searches. What should I do about this?

Since we aim at exact NN, just remove the method that is an approximation.

Also: how are you doing with the bencharking application from real pointcloud datasets? I couldn't find it in your branch... I saw your new example, and it's good.

Note that since we don't want large files in Git, if we want to use a dataset it might be better to either: (a) edit the dataset so only a small file is added to the repo, or (b) use CMake to download the dataset files from Internet.

If you hasn't found datasets, let me know and I'll give you some.

Cheers.

jlblancoc commented 7 years ago

This is a kind reminder to all GSoC students and mentors

According to GSoC FAQ, students are supposed to devote 30+ hours per week to their assigned project. This amounts to 6h/day (Mon-Fri) or 4.3h/day (Mon-Sun), or any equivalent distribution given the flexibility to work on your own schedule. Clearly failing to such a commitment may be a reason to fail in the monthly evaluations, since that information was clear at the time of applying to GSoC.

It's difficult to convert hours into GitHub commits, but as a rule of dumb, there should at least a substantial daily commit during each week working day, that is, roughly 5 substantial commits per week. This is a general rule, and mentors may be flexible depending on the difficulty and nature of each project.

The first evaluation will start June 26-30.

pranjal-rai commented 7 years ago

Hi @jlblancoc

I have added the documentation for all the classes.

Sorry, I missed something... what two methods? Logarithm method and threshold?

I proposed two different implementations for rebuild threshold method. I have implemented both but it seems that the rebuild threshold approach is valid only for approximate nearest neighbour searches. So this will not work for our purpose.

Note that since we don't want large files in Git, if we want to use a dataset it might be better to either: (a) edit the dataset so only a small file is added to the repo, or (b) use CMake to download the dataset files from Internet.

I think it will be better to use CMake to download the dataset files from Internet because we need to run tests on large files and editing the dataset won't be a feasible solution. I will update you with the real pointcloud datasets ASAP. If you have any suggestion for datasets, please share it with me.

jlblancoc commented 7 years ago

Sure! Some datasets:

pranjal-rai commented 7 years ago

Hi @jlblancoc

We now have the support for dynamic point clouds. I have tested the implementation and it looks good to me. We can verify it again once you are done with executables for testing.

Also: As the rebuilding threshold approach does not work for us, so should I push these changes to my repo in a new branch or just leave it?

Also: how are you doing with the bencharking application from real pointcloud datasets?

I am now planning to create a GUI based application to perform all the benchmarking tests. This can either be done in MATLAB or Python with Tkinter (preferably latter, as I don't have much experience with Tkinter and I want to learn more about it). I will first design the tool with all the basic functionalities and then start with the coding part after the first evaluation. We can also add an option for running the codes directly for users who wish to perform intensive testing with their own parameters. Do you have any other suggestion or should I proceed with the proposed plan?

Also, do I need to make a report for the first evaluation?

jlblancoc commented 7 years ago

Also: As the rebuilding threshold approach does not work for us, so should I push these changes to my repo in a new branch or just leave it?

You could push it to a separate branch so we can take a look at it just for evaluation purposes for this first evaluation period; then remove it after some more weeks if will be definitively useless.

Ok, go on with the GUI for benchmarking. Please, let me know the list of exact benchmarks you plan to support in it. I think we've discussed this in the past, but it's good to revisit it again ;-)

Also, do I need to make a report for the first evaluation?

Nope, we'll just review based on public discussion (this thread!) and your commit. Might come back to ask you for some doubts if we need it.

pranjal-rai commented 7 years ago

Hi @jlblancoc

I have added both approaches for rebuild threshold method as separate branches [1], [2]. Right now I have added the support for dynamic insertions by modifying the existing class itself without duplicating the class. I will structure the code properly if we plan to use it in the future.

List of benchmarks:

jlblancoc commented 7 years ago

All good.

I'm not sure about your plans to go on working with branches, but if the Logarithm method seems stable, you can merge it now to your fork's master... do you agree? Or do you have other plans? I mention it because in that way the next step (starting with benchmarking) could be carried out in master...

pranjal-rai commented 7 years ago

Hi @jlblancoc

Sure. I will go ahead and merge the Logarithm Method branch into master. I will then create another branch from master for benchmarking which can be merged later.

pranjal-rai commented 7 years ago

Hi @jlblancoc

I have merged Logarithmic Method into master and created another branch Benchmarking from master in which I will start working on the benchmarking tool.

I had some doubts regarding the work now. As we planned to compare our result with 4 other kd-tree libraries, each of these libraries is needed to be first set up on the user's system before running the benchmarking tool. For example if the user asks to generate plots for time performance of flann and nanoflann, then the user has to make sure that flann is already set up on the system. I was planning to write a script which will first check the availability of the required libraries and will go on to download the libraries which are not installed beforehand. We can do the same thing to check the availability of real point cloud datasets too. This script will be executed every time the tool is started. Is this solution reasonable or do you have any other suggestion?

jlblancoc commented 7 years ago

Hi,

That's a good idea. Only, make sure of using cmake to download and build the 3rd party libraries. Read on this command:

https://cmake.org/cmake/help/v3.0/module/ExternalProject.html

jlblancoc commented 7 years ago

Also, for each library, check if there exists a Debian/Ubuntu package, then make the cmake script to FIRST try to find and use the library in the system (installed via apt), and if that fails THEN use the cmake external projects.

pranjal-rai commented 7 years ago

Hi @jlblancoc

We need ICP for benchmarking real datasets, should I write my own version or use some existing library like MRPT or maybe some other light library which is just meant for ICP?

jlblancoc commented 7 years ago

Hi,

You could reuse mrpt code but.... I also think it's enough to benchmark one iteration of finding the nearest neighbour for all points in one point cloud to another, since that's the most costly part of icp, and the one we are interested in... ok?

pranjal-rai commented 7 years ago

okay. So instead of using any library I will just write a simple code which takes two point clouds and for every point in the first cloud finds closest point in the second one. We do not need any thing else from ICP. Right?

jlblancoc commented 7 years ago

Correct :-) That's my view and think it's perfectly plausible even if the benchmark is to be incorporated into a scientific paper or alike.

pranjal-rai commented 7 years ago

Hi @jlblancoc

Thanks for the reviews. Clarifications: I was working on the features in the following branches.

I was working on the last two features locally and added them to git just before the evaluations. I have merged the first branch with master now and made another branch Benchmark from master which I am working on now.

jlblancoc commented 7 years ago

Great! All is clear now.

pranjal-rai commented 7 years ago

Hi @jlblancoc

The kd-tree libraries we discussed earlier have the following ubuntu packages.

Can we list these libraries to be the dependencies of nanoflann library as they will be required to run the benchmarking tests. If the user has already installed these ubuntu packages (using apt-get) then we do not need to clone these libraries and include their path in our CMakeLists file?

I am new to cmake so I was having some problem using it. Earlier we talked about adding external projects using cmake. I wrote a sample CMakeLists file to clone flann from github to the path nanoflann/3rdparty/. I then linked this file to the main CMakeLists file. Now I need to include flann library in my code. For this I added the path of the cloned library to my CMakeLists. Now the problem is when I am running this code using this configuration I get the following error :

CMakeFiles/flann_test.dir/flann_test.cpp.o: In function flann::serialization::SaveArchive::initBlock()': flann_test.cpp:(.text._ZN5flann13serialization11SaveArchive9initBlockEv[_ZN5flann13serialization11SaveArchive9initBlockEv]+0x47): undefined reference to LZ4_resetStreamHC' CMakeFiles/flann_test.dir/flann_test.cpp.o: In function flann::serialization::SaveArchive::flushBlock()': flann_test.cpp:(.text._ZN5flann13serialization11SaveArchive10flushBlockEv[_ZN5flann13serialization11SaveArchive10flushBlockEv]+0x4b): undefined reference to LZ4_compress_HC_continue' flann_test.cpp:(.text._ZN5flann13serialization11SaveArchive10flushBlockEv[_ZN5flann13serialization11SaveArchive10flushBlockEv]+0x115): undefined reference to LZ4_compress_HC_continue'

The same code works perfectly when it is compiled using either the installed flann ubuntu package or using the CMakeLists file in the flann library. What is the correct way of including the 3rdparty library in my code?

jlblancoc commented 7 years ago

Hi!

If the user has already installed these ubuntu packages (using apt-get) then we do not need to clone these libraries and include their path in our CMakeLists file?

Correct, that's the idea. Downloading via CMake is actually most useful for poor Windows users, who can't enjoy apt-get ;-)

Your link error seems to me to be caused by building flann as a STATIC library (*.a) instead of a DYNAMIC library (*.so). Why? The missing functions seem to belong to another dependency of FLANN called LZ4. In a DYNAMIC library, all required functions are already linked, so the user of the lib doesn't need to know about them. In static libraries, the list of dependencies at link-time must be propagated aaaall the way down to the final user who is building an executable (or a dynamic lib).

In short: try to automatically instruct CMake to build FLANN as a dynamic library, not a static lib...

Probably this is the problem, let me know if it's not.

Cheers.

pranjal-rai commented 7 years ago

Hi @jlblancoc

There are a lot of files in the 4 k-d tree libraries which we do not need at all like the python/ruby/matlab wrappers. Further we do not need to build the tests and examples in each of these libraries. I was thinking of adding the minimalistic code from these libraries to nanoflann permanently. I checked and the code we require is less than 2 MB in size. We can directly add the path of these libraries to our CMakeLists file. Is this solution feasible?

jlblancoc commented 7 years ago

Hi!

I haven't check them all, but typically all libraries come with CMake variables like BUILD_TESTS, BUILD_EXAMPLES, etc. When automating its build via CMake external projects, we can instruct CMake to set them to "OFF" to build only what we need.

If there are not such CMake variables, there is even the option to apply a custom patch to disable the parts we don't want. But adding 2MB of 3rd party sources is something I don't like, because from my experience it ends up being a headache if the codebase must be maintained during years... imagine each new version of GCC giving new errors for each library... it's not our work to fix them! Instead, the best thing to do is to delegate to each maintainer, and download the last stable version at each time.

What do you think?

pranjal-rai commented 7 years ago

the best thing to do is to delegate to each maintainer, and download the last stable version at each time

I think that is a valid point. So now I have a CMakeLists file for 3rd part library. I have set the options such that when we build nanoflann it just clones flann and does not build it. Does this file look fine to you?

In short: try to automatically instruct CMake to build FLANN as a dynamic library, not a static lib...

How do I do this? I tried something like - ADD_LIBRARY(flann_test SHARED flann_test.cpp) TARGET_LINK_LIBRARIES(flann_test nanoflann) This creates a libflann_test.so file . I do not get an executable to run flann_test.cpp. I am stuck with this part.

jlblancoc commented 7 years ago

The lines you removed in this commit were going in the right direction:

CMAKE_ARGS -DBuildShared=OFF -DBuildExamples=OFF -DCMAKE_INSTALL_PREFIX=${EXECUTABLE_OUTPUT_PATH}/flann

So please add it again, but with some fixes: You'll have to go through the CMakeLists of each 3rd party lib to find the exact names of each CMake variable (since they are not standardized). For example, for flann, most of these ones should be set to OFF.

Let me know if it works like this... I think it will! ;-)

pranjal-rai commented 7 years ago

The lines you removed in this commit were going in the right direction

I fixed it in this commit. I switched BUILD_EXAMPLES to ON to build the examples in flann for now. The flann examples are built without any problem and the executables are saved in nanoflann/build/bin/flann/bin/. But the problem is there is no such variable like BuildShared in flann using which I can build my own flann file. When I build this flann file as a flann example by moving it to nanoflann/3rdparty/flann/example/flann_example.cpp, it works without any problem. But when I am building it outside flann it still gives the same problem as before probably because FLANN is not being built as a shared library. How do I instruct flann to build this file using the same way it builds its own examples?

jlblancoc commented 7 years ago

Try setting this variable for the flann project.

Also, let me know if you have everything pushed to your fork so I can check the latest version that gives you the build error.

pranjal-rai commented 7 years ago

I tried setting this variable, it still gives the same build error. This branch is up to date and this is the file which gives build error. Please let me know if you have any doubt.

pranjal-rai commented 7 years ago

I have attached the build error. build_log.txt

jlblancoc commented 7 years ago

It will be better to offer a separate CMake option for examples and for the benchmark, near this line.

jlblancoc commented 7 years ago

(I'll add comments on things I see as I inspect the code and test it under MS Visual Studio 2017):

#include <string> is required in files using the function getline(), please add it.

jlblancoc commented 7 years ago

You are missing a:

TARGET_LINK_LIBRARIES(flann_test flann)

in its CMakeLists file. By adding it, you ensure that flann is added as a dependency so it gets built by Makefile (or any other build system), and also you probably fix the linking issues...

pranjal-rai commented 7 years ago

It will be better to offer a separate CMake option for examples and for the benchmark, near this line.

Fixed it here.

include < string > is required in files using the function getline(), please add it.

Updated in both flann_test.cpp and nanoflann_test.cpp.

You are missing a: TARGET_LINK_LIBRARIES(flann_test flann)

I first had to set flann_LIBRARIES to path of libflann.so and then linked it to flann_test otherwise it was not able to locate libflann.so and gave this error. /usr/bin/ld: cannot find -lflann I updated this here. Now the build issue is fixed. Thank you very much for your help.

I had another doubt. Should a git commit message include information about all the changes I made or just the major changes?