map save/load

[dannyway03]

Hi Raul!

first of all thank you for sharing such a great piece of software! I was thinking it would be nice to have some map save/load functionality. This could enable some interesting functionalities, like the reconstruction of large scale environments in an incremental fashion (i.e., create and save and initial map, which is loaded and extended iteratively).

Do you plan to deliver such functionality? (i saw there are some commented functions in system.h) If not, do you have any advice to enable it?

thanks, danny

[marcelinomalmeidan]

Hey, check the Pull Request "Add getter for map in System. #14". Seems like its exactly what you are asking about!

[dannyway03]

Hi, that is quite not the same thing. The map "getter" of #14 does not allow storing/reloading the map, and re-use it. What I meant is to have a way to store the WHOLE map/system configuration. In this way it would be possible to load a pre-calculated environment map (and all the related information), and carry out e.g., only the camera localization (or keep on the tracking and extend the map)

[poine]

@dannyway03: I am working on load/save. I don't know if/when i will have it finished.

[poine]

Request for comments: I have been working on map saving/loading for a while and am starting to get the first results. I am able to run ORBSLAM2 on kitti0 monocular and save the resulting map. Subsequently, I am able to restart ORBSLAM2, load the map and run in localization only mode. A typical resulting map for kitti0 (100.000 MapPoints and 1700 KeyFrames) weights about 200mB, takes 2.5s to store and 10s to load.

In order to achieve this result, I have had to butcher ORBSLAM2's code quite a bit and before releasing a patch, I would like to gain a better understanding and refactor quite a bit too, hence that first request for comments (for now i shoved most everything in Map.cc and added accessors in other classes where needed).

In my understanding, at the moment, the camera calibration is parsed by Tracking. The calibration, and computed by-products, are then stored in Frames and KeyFrames, partly as class variables and partly as instance variables. Tracking has a ChangeCalibration methods which triggers the update of the Frames class variables but leaves existing instances as is. My question is about the use case of ChangeCalibration. How is this functionality intended? I would like to avoid having to store a calibration along each KeyFrame. I could store the different calibrations and have each Keyframe linked to its own one but that also sounds a bit complicated. I guess my question is: do we want to have the possibility of having several calibrations coexisting in one map? or was ChangeCalibration intended to be used in conjunction with a map reset and we can assume that all the Frames in a map use the same camera and the same calibration?

Oh... and once again, thank you so much Raul for your awesome work and your code. I learned so much by reading it, you are my hero!

[ghost]

@poine thats great news and great work, can you give an estimate on when you can release a patch?

[poine]

Depends on what kind of patch you want. As I said, I butchered orbslam2 quite a bit. For example, I made camera-related stuff common to all frames, which allowed me to not store anything camera-related in the map. Is that what we want? My first patch, related to the dictionary, did not get accepted (nor rejected) so I am not sure what to do nor where to start from. As I said before, I am a noob at C++, vision and git. I need guidance. On Mar 2, 2016 2:06 PM, "khedd" notifications@github.com wrote:

@poine https://github.com/poine thats great news and great work, can you give an estimate on when you can release a patch?

— Reply to this email directly or view it on GitHub https://github.com/raulmur/ORB_SLAM2/issues/19#issuecomment-191231718.

[ghost]

Any kind of patch will work for me.

However I do not know why your first patch did not get accepted, I see that you have you own repository. Was you first patch about speeding up the loading time? If so I can see that in https://github.com/poine/ORB_SLAM2.

2016-03-02 23:38 GMT+02:00 poine notifications@github.com:

Depends on what kind of patch you want. As I said, I butchered orbslam2 quite a bit. For example, I made camera-related stuff common to all frames, which allowed me to not store anything camera-related in the map. Is that what we want? My first patch, related to the dictionary, did not get accepted (nor rejected) so I am not sure what to do nor where to start from. As I said before, I am a noob at C++, vision and git. I need guidance. On Mar 2, 2016 2:06 PM, "khedd" notifications@github.com wrote:

@poine https://github.com/poine thats great news and great work, can you give an estimate on when you can release a patch?

— Reply to this email directly or view it on GitHub https://github.com/raulmur/ORB_SLAM2/issues/19#issuecomment-191231718.

— Reply to this email directly or view it on GitHub https://github.com/raulmur/ORB_SLAM2/issues/19#issuecomment-191444156.

[MathewDenny]

@poine i tried serializing and deserializing the ORB map and i have a couple of questions to ask you. I am storing the list of mappoints and keyframes in the Map class and I just store the ids of the observations ,the parentKFs , the spanning tree etc for each item. During loading, after loading the saved lists, im searching and the list of ids in the original lists of mappoints and keyframes and then referencing them.

In this way , I found there are non corresponding KF and Mappoint ids during crossreferencing. I can circumvent by checking for nonexistant KFs / Map points at multiple locations during run time.I am able to run in localization or mapping mode with the loaded map but fails during the instance of a loop closure. Could you advise if i would need to serialize other elements like the Local Graph or just identify and save the missing elements also. Have you faced this situation?

[poine]

@denny I am on vacations away from my computer. I'll answer in a few days.

On Fri, Mar 25, 2016 at 5:04 AM, Denny notifications@github.com wrote:

@poine https://github.com/poine i tried serializing and deserializing the ORB map and i have a couple of questions to ask you. I am storing the list of mappoints and keyframes in the Map class and I just store the ids of the observations ,the parentKFs , the spanning tree etc for each item. During loading, after loading the saved lists, im searching and the list of ids in the original lists of mappoints and keyframes and then referencing them.

In this way , I found there are non corresponding KF and Mappoint ids during crossreferencing. I can circumvent by checking for nonexistant KFs / Map points at multiple locations during run time.I am able to run in localization or mapping mode with the loaded map but fails during the instance of a loop closure. Could you advise if i would need to serialize other elements like the Local Graph or just identify and save the missing elements also. Have you faced this situation?

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/raulmur/ORB_SLAM2/issues/19#issuecomment-201127180

[shomin]

@poine To answer your request for comments: I don't see the function ChangeCalibration actually getting called anywhere. I think it's a very safe to assume a constant calibration throughout a map. If you're willing to commit what you have to your fork, I'd be happy to take a look at it.

[poine]

@Michael: i am not comfortable with git and a bit short in time at the moment to remedy it. I've uploaded an archive of my current tree here: http://recherche.enac.fr/~drouin/slam/orbslam2/poine_orbslam2_04_07_16.tgz

The code for loading/saving maps is in Map.cc .It's kind of ugly at the moment, but it's good enough for my needs. I have assumed a common camera calibration for all frames and stored it with the map.

If you (or someone else) is willing to use that code, I am willing to take the time to polish it and publish it on github.

hth

Poine

On Wed, Apr 6, 2016 at 10:56 PM, Michael Shomin notifications@github.com wrote:

@poine https://github.com/poine To answer your request for comments: I don't see the function ChangeCalibration actually getting called anywhere. I think it's a very safe to assume a constant calibration throughout a map. If you're willing to commit what you have to your fork, I'd be happy to take a look at it.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/raulmur/ORB_SLAM2/issues/19#issuecomment-206564043

[shomin]

@poine I'll look over the code and give it a try. Your link appears to be broken though.

[poine]

@Michael: sorry for the broken link. Should be fixed now

On Thu, Apr 7, 2016 at 7:29 PM, Michael Shomin notifications@github.com wrote:

@poine https://github.com/poine I'll look over the code and give it a try. Your link appears to be broken though.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/raulmur/ORB_SLAM2/issues/19#issuecomment-207014446

[riematrix]

@poine Looking forward to see this feature been published, but maybe the link is still broken.

[bhack]

@poine I hope that you can push your code on github soon. Would be very helpful.

[zsy372901]

@poine I'm glad your code released, but the link may broken...anyway, thanks for your efforts

[poine]

@zsy372901 The link seems to be working from my side (it was broken last night). Can you check and let me know ?

@bhack

I'll try to free some time and look again at git oddities next week

On Fri, Apr 8, 2016 at 10:16 AM, zsy372901 notifications@github.com wrote:

@poine https://github.com/poine I'm glad your code released, but the link may broken...anyway, thanks for your efforts

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/raulmur/ORB_SLAM2/issues/19#issuecomment-207302773

[zsy372901]

@poine aha, it is worked：D thanks a lot!

[shomin]

@poine Wow, I just gave myself a headache... There seems to be a bug in your PublishCurrentCamera function used in rgbd. I added the save subscriber from ros_mono and was trying it on a bag, but I was getting an exception after about 20 seconds of running. Very strange. I'll look into it later, but for now, the save functionality seems to be working. I'll add the loading to System.cc and try it through ros.

[shomin]

@poine Would you like me to package up the save/load functionality into commits and make a pull request, or do you plan to? Also, nice work!

[poine]

@Michael: Sure, whatever... i'm not fluent with git, so it would take me a long time figure it. Also the code is not really finished. It only works for monocular and i vaguely recall very inefficiently storing an array of pointers which were mostly null. But for now, it fills my need which was to have a georeferenced slam.

On Tue, Apr 12, 2016 at 4:39 AM, Michael Shomin notifications@github.com wrote:

@poine https://github.com/poine Would you like me to package up the save/load functionality into commits and make a pull request, or do you plan to?

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/raulmur/ORB_SLAM2/issues/19#issuecomment-208675784

[highlightz]

@poine Thanks for your great work! I have downloaded your codes and see that in Map.cc, you implemented a function called _ReadKeyFrame. And there is a comment 'mono only for now'. I have been running ORB-SLAM2 with a bumblebee2 stereo camera, that is, I need to save map in stereo mode. So could you tell me whether I can reuse your codes on my stereo camera. Thanks again.

[skylook]

Thanks @poine for the great job. But I still have an issue: If I just save map and next time load map, it loaded well but does not work on tracking stage.

It crashes on this line:

    mlRelativeFramePoses.push_back(mlRelativeFramePoses.back());<=== Crash Error: EXC_BAD_ACCESS
    mlpReferences.push_back(mlpReferences.back());
    mlFrameTimes.push_back(mlFrameTimes.back());
    mlbLost.push_back(mState==LOST);

because mlRelativeFramePoses is empty. Any ideas on how to fix this or did I miss something? Thanks.

[poine]

@skylook: the code i use to restart orbslam2 after loading a map is here: ORB_SLAM2/test/restart_slam_on_images.cc

On Tue, Apr 26, 2016 at 11:32 AM, skylook notifications@github.com wrote:

Thanks @poine https://github.com/poine for the great job. But I still have an issue of how to use map load function? If I just save map and next time load map, it loaded well but does not work on tracking because the value of mState is still NOT_INITIALIZED and some other parameters are not set. Any body can give a working example of Save&Load. Thanks!

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/raulmur/ORB_SLAM2/issues/19#issuecomment-214683526

[skylook]

Thanks for your reply @poine , I already tried what you did in restart_slam_on_images.cc, it still crashes on the following line in Tracking.cc:

mlRelativeFramePoses.push_back(mlRelativeFramePoses.back());

This line is touched when it is failed tracking on the first frame (mCurrentFrame.mTcw.empty()). When it runs without an initial map, it always has a mlRelativeFramePoses, but when it runs with a initial map, it is empty.

I go through your code but I haven't catch the point on how to fix this. Any idea? Thanks so much :)

[highlightz]

@dannyway03 Have you realized your goal of reusing the slam map?

[skylook]

I have found a problem of map load function. Because mspMapPoints is stored in std::set which means when using std::vector<MapPoint*> amp = GetAllMapPoints(); the return vector is not always in the same order (set is ordered by the pointer address). That makes _ReadKeyFrame() function gets a wrong result. @poine

[Severn-Vergil]

@poine Thanks for your great job! Could I ask about the modification of your package comparing with the original ORB-SLAM? For example, the map save/load in map.cc and the example codes. So nice of you to help me learn more about SLAM.

[dannyway03]

@highlightz not so much time for this lately...anyway, i have been trying with serialization, somehow inspired by @MathewDenny..

[chwimmer]

@poine 2 Questions:

1. Are you able to switch back from the Localisation Mode to the SLAM Mode after loading the Map?
2. Are you able to reset the System after loading the Map?

I'm also currently writing my code to save and load the map and it works fine, but still got problems with this 2 issues.

[MathewDenny]

Thanks @dannyway03 . Im sure you can devise something better.. @chwimmer on paper these should be definitely possible. If its of any use check out the monocular specific code i put up sometime back.

[lschaupp]

hey @poine: I have tried to run your code and while the LoadMap and SaveKITTIMap files work fine, the restartSLAMOnImage seems to be broken (at least for me). There seems to be a memory related bug as in my Virtual Machine the kernel kills the process after filling up the whole RAM. @skylook Do you have any fixes for the bugs u mentioned in the code - could be related!

edit: so far I've started the SaveKitti which outputs a map.bin file. ViewMap can load the map and show it. Now I wanted to load the map and localize myself with a given dataset by using restartSLAMOnImage. Maybe I'm missing a step here?

[poine]

Hello World

@lschaupp I am aware there is a problem with restarting SLAM (retarting in localization mode works, which is what i was doing) - I believe it is due to me not storing the last keyframe index with the map. So... restarting works sometimes, when the last keyframe has not been culled. I'm a bit overwhelmed at the moment, but i can fix that maybe next week.

On Tue, Jun 28, 2016 at 1:05 AM, lschaupp notifications@github.com wrote:

hey @poine https://github.com/poine: I have tried to run your code and while the LoadMap and SaveKITTIMap files work fine, the restartSLAMOnImage seems to be broken (at least for me). There seems to be a memory related bug as in my Virtual Machine the kernel kills the process after filling up the whole RAM. @skylook https://github.com/skylook Do you have any fixes for the bugs u mentioned in the code - could be related!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/raulmur/ORB_SLAM2/issues/19#issuecomment-228902300, or mute the thread https://github.com/notifications/unsubscribe/AAGTz30RpE7AoQhySUFtu3l3uCAg-BwFks5qQFc6gaJpZM4HVJUE .

[lschaupp]

@poine thx for your reply. So far I could fix my problem (mistake in the map path) and it runs really well with the localization. So thanks again! Thanks for all your effort.

[poine]

On Tue, Jun 28, 2016 at 9:30 PM, lschaupp notifications@github.com wrote:

@poine https://github.com/poine thx for your reply. So far I could fix my problem (mistake in the map path)

sorry... that code is not really written. We should write errors handling. It was just a quick and dirty proof of concept.

and it runs really well with the localization. So thanks again! Thanks for all your effort.

You're very welcome

Does anybody have ideas or even better has started to work on map management ( like georeferencing, offline optimization, map merging, collaborative mapping, etc... )? I'm thinking of a fleet of network connected robots using and maintaining a map, a bit like what the autonomous cars seem to be doing.

Oh, and on another topic, i have started to write a python binding (cython) to ORBSLAM2. I find it much easier to experiment and write test in python than in C++. Is anybody interested in that?

[chwimmer]

@MathewDenny Thank you so much. Tried your code and it works out of the box for me (only the tool folder from poine is missing in your repository so your build.sh is not working out of the box but thats not really the point). The only thing is that a reset is not working after loading the map but thats subsidiary for me.

[aseyfi]

@MathewDenny , I was looking at your last commit and the only modified file in ROS folder is ros_mono.cc. Does it mean the save/load functionality is only working for monocular?

[chwimmer]

@aseyfi No it also works with RGBD or Stereo. You just have to modifie the other files like ros_mono.cc

[colegleason]

Hey @poine, regarding your last comment, I would be interested in Python bindings!

[Etragas]

Hey @poine I'm also doing work on robotic fleets doing offline shared mapping. I'm both interested in developing the cython bindings and adding additional functionality to ORB-SLAM2 to support Octomap integration. If you'd like to work with me on this you can hit me up at tragas@cs.toronto.edu and we can talk further.

[qianwenjun0801]

@poine hi, I'm working on reload map. There is a problem like that : first i save the map using "Save" in "map.cc" and close the program; second I load ("Load in map.cc") the map and using it as initialization, then I save the map again, there is an error "segmentation fault" . I guess maybe the f.write((char*)&ckf->mnId, sizeof(ckf->mnId)); cause the error, but i don't know how to solve it. Do you have some good ideas?

[chwimmer]

you just have to check if it is a NULL Pointer and if it is don't write it into the File. But checkout the Code of @MathewDenny. Its working pretty good.

[qianwenjun0801]

@chwimmer thanks

[qianwenjun0801]

@chwimmer hi, I download the code of @MathewDenny , and it work well for just creating a new map, and when the bReuseMap=true, there is also a problem as follows terminate called after throwing an instance of 'boost::archive::archive_exception' what(): input stream error Aborted (core dumped)

I've checked the threads and the stack, and it doesn't have any wrong things, could you have some advise for me ?

[chwimmer]

This is normally the error Message when the Path to the Map is wrong. I'm not sure what is the default path from Mathew. I changed it that I give the constructor the path to the map so I have: LoadMap(path_map.c_str()); with the new constructor: System::System(const string &strVocFile, const string &strSettingsFile, const eSensor sensor,const string &path_map, const bool bUseViewer, const bool bReuse)

[MCoder1]

Any idea what's going wrong for me here?

When I clone the repo, I get an error when trying to build though I can build the latest ORB_SLAM2.git from raulmur fine and it runs, so I know my dependencies are right...

Uncompress vocabulary ... Configuring and building ORB_SLAM2 ... -- The C compiler identification is GNU 4.8.4 -- The CXX compiler identification is GNU 4.8.4 -- Check for working C compiler: /usr/bin/cc -- Check for working C compiler: /usr/bin/cc -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working CXX compiler: /usr/bin/c++ -- Check for working CXX compiler: /usr/bin/c++ -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done Build type: Release -- Performing Test COMPILER_SUPPORTS_CXX11 -- Performing Test COMPILER_SUPPORTS_CXX11 - Success -- Performing Test COMPILER_SUPPORTS_CXX0X -- Performing Test COMPILER_SUPPORTS_CXX0X - Success -- Using flag -std=c++11. -- Found Eigen3: /usr/include/eigen3 (Required is at least version "3.1.0") -- Boost version: 1.54.0 -- Found the following Boost libraries: -- serialization -- Configuring done CMake Error at CMakeLists.txt:105 (add_executable): Cannot find source file:

tools/bin_vocabulary.cc

Tried extensions .c .C .c++ .cc .cpp .cxx .m .M .mm .h .hh .h++ .hm .hpp .hxx .in .txx

-- Build files have been written to: directory... make: *\ No targets specified and no makefile found. Stop. Converting vocabulary to binary ./build.sh: line 35: ./tools/bin_vocabulary: No such file or directory

[poine]

Looks like the "bin_vocabulary" program failed to build ( the 'Cannot find source file line' ). Later build.sh tries to use that program to convert the text vocabulary into binary and fails. I don't know why the program failed to build in the first place, though. Do you have the source file (tools/bin_vocabulary.cc) ?

On Aug 11, 2016 9:04 PM, "MCoder1" notifications@github.com wrote:

Any idea what's going wrong for me here?

When I clone the repo, I get an error when trying to build though I can build the latest ORB_SLAM2.git from raulmur fine.

Uncompress vocabulary ... Configuring and building ORB_SLAM2 ... -- The C compiler identification is GNU 4.8.4 -- The CXX compiler identification is GNU 4.8.4 -- Check for working C compiler: /usr/bin/cc -- Check for working C compiler: /usr/bin/cc -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working CXX compiler: /usr/bin/c++ -- Check for working CXX compiler: /usr/bin/c++ -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done Build type: Release -- Performing Test COMPILER_SUPPORTS_CXX11 -- Performing Test COMPILER_SUPPORTS_CXX11 - Success -- Performing Test COMPILER_SUPPORTS_CXX0X -- Performing Test COMPILER_SUPPORTS_CXX0X - Success -- Using flag -std=c++11. -- Found Eigen3: /usr/include/eigen3 (Required is at least version "3.1.0") -- Boost version: 1.54.0 -- Found the following Boost libraries: -- serialization -- Configuring done CMake Error at CMakeLists.txt:105 (add_executable): Cannot find source file:

tools/bin_vocabulary.cc

Tried extensions .c .C .c++ .cc .cpp .cxx .m .M .mm .h .hh .h++ .hm .hpp .hxx .in .txx

-- Build files have been written to: /home/mchong5/Projects/ORB_SLAM2_MapSaveLoad/ORB_SLAM2_SaveLoad/build make: *\ No targets specified and no makefile found. Stop. Converting vocabulary to binary ./build.sh: line 35: ./tools/bin_vocabulary: No such file or directory

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[MCoder1]

@poine

I do not have the source file tools/bin_vocabulary.cc. It isn't included when I cloned the repo. It doesn't seem to be a part of raulmur's orb slam repo, so I'm not sure what it's for.

[poine]

@MCoder1

Where does the build.sh that builds and uses bin_vocabulary come from? I just checked in Raul's repo and i did not see the binary vocabulary things.

I assume you took the binary vocabulary from my repo ( https://github.com/poine/ORB_SLAM2 ) or from the tarball i released with the map save/load feature:

http://recherche.enac.fr/~drouin/slam/orbslam2/poine_orbslam2_04_07_16.tgz

On Thu, Aug 11, 2016 at 9:20 PM, MCoder1 notifications@github.com wrote:

@poine https://github.com/poine

I do not have the source file tools/bin_vocabulary.cc. It isn't included when I cloned the repo. It doesn't seem to be a part of raulmur's orb slam repo, so I'm not sure what it's for.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/raulmur/ORB_SLAM2/issues/19#issuecomment-239263557, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGTzzqXKqNPdP2Sk1kpR-tNWd8YxJxGks5qe3YWgaJpZM4HVJUE .

[MCoder1]

@poine

I took it from https://github.com/MathewDenny/ORB_SLAM2; some of the comments seemed to suggest it works well. If I clone your repo, will that build and run fine? If so, I'll try that.