Should there be separate libraries for each subsystem, or a single library for everything?

[jwonders]

Sternberg brought this idea up the other night, and I think we should at least give it some thought. Or, at least not have the discussion lost in gchat logs. The whole thing started because of ram_logging not being linked to ram_network and the associated tests which caused some problems that would be very hard to track down for someone who is not familiar with the code or not familiar with the more advanced parts of boost::serialization. If there was a single library for the C++ code, ram_tortuga, this problem would not have showed up in the first place. This is not an isolated incident. We have had problems with circular dependencies between libraries in the past and have had to do some gymnastics to avoid them. It is easy for someone that is not familiar with the current dependency graph to accidentally introduce them and this means that there is a higher learning curve for new people as they need to spend time understanding how things are allowed to fit together rather than writing code that actually does stuff.

I'll try and enumerate the possible conclusions of this thread so we can try and work towards one of them.

1. Separate libraries are better. We should not make this switch.
2. A single library is better. We should make the switch.
3. A single library is better, but it is too much work to switch at this point.
4. Neither approach is clearly better than the other. We should not make the switch.

[jsternberg]

This is something I've considered a lot, so I know fully well the road blocks to making it a reality (if done).

First of all, a shared library must be produced at some point because the robotics code must be loaded by python, which can only be done using a shared library. Furthermore, as much as I hate the number of namespaces we have, they're used everywhere and while I don't like the namespaces in C++, they're quite useful in Python.

The default method for creating C extension in Python is to have a shared library with the name of the module. Inner modules would then also require a shared library, so ext.vision and ext.math need to be separate shared libraries (this isn't actually true, but I'll get into that).

Since this is the default method, this is the one pygccxml is going to generate. With pygccxml, there are two options. One is to compile all of the wrappers into a single namespace (not practical for a number of reasons) and one is to continue having them as separate shared libraries per namespace and link to a single robotics shared library. The first one frankly won't work. The second version might work if the single library is a shared library, but I really want to make it a static library (it makes linking much easier and may make it possible for me to remove scripts/setenv in a practical way).

It IS possible to have a single shared library have multiple inner modules with a Python module. It's possible with both the Python C library and boost::python to create inner modules. So it would be possible to have an ext.so with all of the robotics wrappers.

This version would require manually writing the wrappers, which I personally like the idea of. The end result would be a single static library for all of robotics' code and a single shared library for the extension modules.

I'd also like to change the name of the extension modules from "ext" to "_ram" or something that's less... generic.

[jwonders]

Aside: scripts/setenv is the main roadblock standing in the way of using Eclipse as a development environment. There is no way to set the Eclipse environment through a script and as such, the eclipse projects generated by CMake will not build out of the box. It is a huge pain to manually set up the project. Even though the evil part of me wants to force everyone to learn how to use a freaking text editor, the possibility of using Eclipse might help member retention.

I think manually writing the wrappers makes a lot of sense. Most of the classes and functions that need to be wrapped are pretty stable at this point. It will also be much clearer to new members what needs to be done to extend the wrappers because there will be examples of wrapping similar classes for pretty much any use case we would hit. Right now, the wrappers are just voodoo magic. They still scare me.

I don't like _ram as a name (the leading underscore makes me think PRIVATE). I imagine you wouldn't like tortuga because it is too many characters. Changing the name would require modifying all of the files that currently use ext.{subsystem}. I think this should be the lowest priority item.

[jsternberg]

While I personally like using vim/emacs, getting Eclipse to work would be great. Can you arbitrarily set up any build system you want with Eclipse, or would we still be restricted to CMake? Honestly, I likely wouldn't have nearly as many problems with CMake as I currently do if we just removed the auto-generated wrappers.

The _ram as a name is due to a Python convention, where the C extension library is essentially private and the proper namespace (in this case, ram) imports the private extension module and sets up some Pythonic wrappers around it to make it easier to use.

Changing the name is one sed command, so I'm not too worried about that. It would be a much bigger deal if third-party code used our own code.

I like manually writing the wrappers as an idea because it takes gccxml and pygccxml out of the picture, which is an incredibly buggy project.

I would guess @jlisee should likely chime in, since he's the one who set up this system to begin with.

[jwonders]

Can you arbitrarily set up any build system you want with Eclipse, or would we still be restricted to CMake? I think the question you should ask is whether the build system that you want to use can generate Eclipse projects.

What build system did you have in mind? I know it isnt Boost.Build or Premake. Waf? Scons? ;)

I'm a bit fuzzy on the Eclipse stuff since I pretty much gave up on it due scripts/setenv troubles. I believe the build for Eclipse CDT projects generated with CMake is backed by Makefiles. Projects can live outside of the source tree which was something that was important to me. I want to say that I had to pass a special option to CMake in order to do that though. There might have also been a problem with the wrapper generation, but that was further down the pipeline than dealing with scripts/setenv and it would certainly be solved by manually written wrappers.

[jsternberg]

I was thinking Waf, but most build systems would work.

scripts/setenv isn't relevant to the wrapper generation. It'll require some other work to remove scripts/setenv, but a simpler structure where only one library has to be loaded will make it easier. We might want to make another issue in regards to scripts/setenv.

[jlisee]

Reducing the overall complexity of the wrapper system would be a plus. I also think switching to a single shared library would fix some of the dynamic_cast issues that are buried in the event system.

I think you should aim higher though. Part of the R@M system complexity is due to the tight coupling of the C++ and Python code. You could get away with a smaller and easier to maintain wrapper if you just had to wrap the math code and common messages/events. Each subsystem would become a program exchanging messages (leverage something like ROS).

I would favour a large re-architecture approach because if you are going to do a lot of grunt work, I would try and come away with a better, and more modern architecture.

[gsulliva]

A little late, but here's my thoughts:

From a design standpoint, I like having multiple separate libraries. However, the benefit doesn't make much sense in a system like ours, at least not any more, looking at the amount of things we've had to do to make a work-around that makes the multiple-library system happy. Ultimately, I think our systems are so closely linked that a single library makes more sense. But frankly, I can't confidently say how much effort it would take to switch over, and whether or not it would be "worth it". Although I think it's something we should work towards in the future regardless.

I also favor a re-structuring of at least a few of our sub-systems, which would probably snowball into a re-architecture, but that should be discussed in large detail (probably not no this thread).