Release for Kinetic / Xenial

[mgADx]

Any chance mongodb_store will be soon available for ROS Kinetic under Ubuntu 16.04 (Xenial)?

[hawesie]

I don't have much time at the moment or easy access to a 16.04 install. I'm very happy to support you through the process if you can create a release, or anyone else, e..g @m312z @furushchev

[cdondrup]

If I see that correctly, there is no mongo-cxx-driver package in ROS Kinetic anymore. I had to build the 26compat branch of the mongo-cxx-driver manually in order to build mongodb_store. Not sure if there is another way of doing this or if this is the right branch (usage of 26compat is discouraged) but at least it builds afterwards. Didn't get to test the functionality, yet.

[barryridge]

Hi all,

For convenience, I've created a fork with the minuscule necessary changes + instructions for building with Kinetic/Xenial as per @cdondrup's notes above. I too have yet to test it, but the build went fine.

I'd create a PR, but the changes are really tiny (just a matter of removing references to mongodb-dev in the rosdeps, so I suspect it would be best to wait for a proper update to the API.

Oh, and nice package @hawesie - we've been making good use of it :smiley: :+1:

[cdondrup]

Thank you for the effort @barryridge.

I think the main problem with a release for Kinetic is that usage of the 26compat branch of the mongo-cxx-drivers is discouraged and it is not longer supported. To have a viable solution for the future, mongodb_store should be migrated to use a newer version of the driver. I am not sure how much work that would be as I have never looked at the C++ implementation of it. I guess this would be up to @hawesie?! If I have time, I might give it a go as well.

Additionally, there does not seem to be a ubuntu package for the mongo-cxx-driver in 16.04 in the first place. This post also suggests that the newest version of the driver has non-standard dependencies like a higher version of libbson and mongoc. That would mean that package for those have to be provided as well.

[barryridge]

@cdondrup Yeah, I actually initially tried using the newest version of mongo-cxx-driver, and had installed libbson and mongoc and everything, but then realised that the API had changed significantly, and fearing a potential rabbit hole, I reverted to your solution! It might not be so bad to change over, but I haven't looked into the code, so I have no idea. There's also the possibility of trying to shift to the legacy branch instead of 26compat, but it might produce some breakages according to the docs. In any case, I'll probably avoid going deeper unless I absolutely have to- enough on my plate as it is!

[hawesie]

Ah, two of my past lives colliding. Hi @barryridge long time no see.

We have the final STRANDS review next week so my head isn't in the right place to deal with this yet, but I'll come back to it post-review and see what I can add.

[marc-hanheide]

Just to add: I also started porting STRANDS software to Kinectic, and there is a new fancy buildfarm that also supports Kinetic at https://lcas.lincoln.ac.uk/buildfarm/ that can do more than the old one. But, progress only to be expected when the review is out of the way.

[marc-hanheide]

In preparation for the kinetic release I opened this PR: https://github.com/ros/rosdistro/pull/15218 as mongodb-dev cannot be resolved for xenial.

[marc-hanheide]

@barryridge did you make any progress on this? I'm inclined of chucking the mongodb_log package out of this repo, as this one heavily relies on the "legacy" C++ Mongo interface. The mongodb_store C++ code probably needs significantly fewer changes as it only uses the BSON part, right @hawesie? And the Python parts should be easy anyway.

[barryridge]

Hi @marc-hanheide,

No, I'm afraid I haven't touched this since our last discussion and I haven't looked into the code beyond a cursory level, so I don't know how difficult the conversion process would be and would say you're probably much more informed on this front. Maybe I can lend a hand with the process at some point, but not such a priority right now seeing as it's possible to use the legacy version.

[hawesie]

@marc-hanheide that makes sense to me. mongodb_log would need to depend on mongodb_store since it uses some of the libraries, but otherwise it should be the easier approach.

[marc-hanheide]

@hawesie have you got anyone who could look into this ATM?

[hawesie]

It's on my TODO list...

[marc-hanheide]

I already created a kinetic-devel branch without the mongodb_log package. But then I noticed that even the new mongocxx drivers are not available on Ubuntu 😒 so, they need to be packaged up, too. Best might actually be to include them in our repo as third party and build them right as part of this. Maybe you have a better idea @hawesie

[hawesie]

Since this is used more generally by other people I think we need a solution which can be used from the ROS repos too. If this means rewriting some of the C++ parts I will do that. I still haven't looked closely, but will do so in the next week or so.

[marc-hanheide]

Oh, don't get me wrong. That's precisely what I meant. Using the new, proper mongocxx, not the legacy one. But that needs a newer libmongoc than available in repos and also libmongocxx which isn't available at all if I have seen this correctly. I suggested that, rather than trying to convince the official ros repos to include these, we provide them as part of this repo. Build as 3rd party libs.

[hawesie]

What is available in the standard repos? I haven't got a 16.04 VM on my laptop yet.

[hawesie]

(if the answer is "go get a VM" then that's fine)

[cdondrup]

In 16.04 you have libmongoc and libbson both version 1.3.1. The mongo-cxx-driver 3.0 requires libmongo 1.4.x. For newer versions even later libmongoc versions are required. I don't think there is any form of mongo-cxx-driver available at all. So whatever is done would need to include a release of the mongo-cxx-driver via the ros repos as well.

As I have written above, compiling the 26compat branch and using that still works on 16.04 even without changing any of the code. So that could easily be packaged and released but its use is highly discouraged by the developers.

[marc-hanheide]

Yeah, and the 26compat won't work with newer MongoDBs.

I started packaging up mongoc 1.6 and libbson and then also a most recent version of libmongocxx. But then it occurred to me that this would work on my own build farm, but not really for the ROS one. As they don't allow arbitrary non-catkin packages. Hence, my suggestion to embed them into this and build them as external libs using the cmake mechanism for it.

[marc-hanheide]

shall we go with the 26compat branch for now and embed that in mongodb_store as a submodule? The refactoring for the head of mongocxx might take longer or have you done anything yet, @hawesie ?

[marc-hanheide]

OK, I have made some progress on this, along the lines outlined above. It's in the kinetic-devel branch. I'll see if this builds on the build farm. It based on the release/legacy branch, which is somewhat stable at least and works with most mongdb versions still. And it contains that libmongoclient.so in its own package, build as an external project.

[marc-hanheide]

right, a few more fixed where required, but now the test build on kinetic ran through including the unit tests. New release is triggered for our buildfarm
Let's see how this work: https://github.com/LCAS/rosdistro/pull/43

[hawesie]

Cool, good work @marc-hanheide. I haven't got too close to doing anything (except installing a 16.04 VM), so let's go with this for now. @barryridge @mgADx @cdondrup are you able to test this?

[barryridge]

Nice work @marc-hanheide! Can deffo help with testing @hawesie, though we have our own project review coming up next week, so it would have to be after that for me.

[marc-hanheide]

can anyone else confirm that #194 works for them. It would be good to also test this on indigo or other versions, just to see if this solution can be the new main branch?

[cdondrup]

I don't have a robot at the moment so cannot really test it. Sorry. As soon as it is back, I'll try it on indigo and kinetic.

[barryridge]

@marc-hanheide - apologies for the delay - I was busy preparing a Humanoids submission!

Just tested building the kinetic-devel branch- that seems to work fine. I will get back to you once we've tested it with our robots here.

[barryridge]

@marc-hanheide - we have now lightly tested this on our setup here, though not extensively. We were able to write/read a database entry, so it seems to be working. Will let you know if we run into any issues.

[hawesie]

So @marc-hanheide shall we make this the main branch?

[marc-hanheide]

I think it already is, isn't it?

[hawesie]

Indeed!