Create "modules" for types not in the standard library

[AzothAmmo]

We already have support for boost::variant mostly due to needing it for a project when cereal was created, but there are many other common things cereal could have support for.

Instead of cluttering the cereal types directory with these, it would be better to have organized modules of support for things, e.g. boost, opencv, or eigen. I'd prefer if these weren't pulled by default with cereal, so they either need to be in another repository or potentially just in some directory that would be outside of the normal cereal install (the cereal include folder).

This relates to the pull requests #53 and #122.

[stevehickman]

Modules makes sense to me. For something as large as Boost, would it make sense to have submodules as well?

[stevehickman]

In terms of subcategories for Boost, it may make sense to use Boost categories or(http://www.boost.org/doc/libs/1_56_0/?view=categorized) or individual library names as the organizing principle to help users know which submodules they might need. In some cases (e.g., Containers and Data Structures), the individual libraries within the category may contain enough data structures to make creating a separate module for each worthwhile. In other cases (e.g. Algorithms), where the primary focus in not on data structures, there will still be data structures used by the libraries in the category, but there may not be enough in each individual library to make creating an entire submodule worthwhile.

The competing forces, as I see it right now, are 1) the more that is in a single module/ submodule, the more effort required to keep the module/submodule updated; 2) the more modules/submodules proliferate, the higher the effort required of each user to keep track of which ones are needed.

Net result: Probably makes sense to just start with a single Boost module, with the thought that breaking it into multiple submodules down the road as it grows may be needed. As each module, submodule grows "too large", it gets factored into smaller pieces based on some easily understood criteria (like Boost category or library name. Knowing that we'd need to leave room to refactor / break apart modules should be sufficient initially.

[stevehickman]

I did a quick review of Boost and created the list below of the libraries it makes sense to serialize. This list excludes libraries that are already in C++11 or those that don't have data structures that it would make sense to serialize. Note that this was a quick review - it may turn out that some of these don't need to be serialized. I have not (yet) attempted to identify which of these are used the most.

Any BiMap (maps and iterators after maps) Circular Buffer Compressed Pair Config (Standard Integer Types) - Container - maybe? mostly C++11 conformant Context - for thread switching - would this ever be serialized (debug logs?) Date_Time - could, but std already has Chrono - lower priority Dynamic Bitset Exception - maybe - dig deeper Filesystem - in TR2? Check Flyweight Fusion Geometry GIL (Generic Image Library) - might be big enough for a submodule Graph - might be big enough for a submodule Heap ICL Interprocess Interval Intrusive IO State Savers - duh Iterators - must work in conjunction with iterated collection? Locale Lockfree Math - 128 bit floating point structure. Others? Math Octonions Math Quaternion Meta State Machine Multi-Array Multi-Index Multiprecision Optional - done Parameter Pointer Container (along with the things it points to) Polygon Property Map Property Tree Range - but only if the referenced collection is serialized Rational Regex SmartPtr (if any aren’t in C++11) Spirit Statechart Tribool Tuple TypeIndex uBLAS Units - store value and units (might need own submodule) Uuid Variant - already done

[AzothAmmo]

My intention here will be for the majority of these modules to be community contributed once we have the infrastructure in place, and we likely won't implement anything ourselves unless we have an immediate need for it.

[patlecat]

Excellent idea, that will help immensely working with Cereal! But I wouldn't make the use of them modules too remote and complicated. A simple include or config entry in a header file should do to add what the developer needs.

[patlecat]

Besides I think especially Flyweight could profit a lot from this, since it added Boost::Serialization in 1.57 and has key-value types. With this and a fast serialization lib Cereal could rival with the use of pure key-value store databases like Redis! :dancer:

[pwm1234]

I just implemented cereal load/save functions for boost::posix_time::ptime. Is there someplace I can put this? In a comment above @AzothAmmo said "once we have the infrastructure in place". Since that comment was made two years ago and this is still an open issue, I suspect the intended infrastructure will not be in place. Can we adjust our intentions to get these kind of community contributions in place? (I am not being critical--I just found cereal and really like the work that you have done on it! I want it to continue to improve and grow.)

My preference would be a subdirectory in the cereal repo, perhaps include/cereal/contrib? I feel this would be better than a separate repo. I am, however, a newcomer to cereal; I would certainly defer to others.

[qh-huang]

I plan to implement boost::graph save/load functions. This thread seems ended in 2014, but I can't find the conclusion about 3rd party data structure serialization. There is no "module" nor "contrib" folder for community work.

Is there any ways to provide our code?

[AzothAmmo]

I recently created a Github project for this (see here: https://github.com/USCiLab/cereal/projects/1) to help organize all of the modules related work. This is still something I very much want to add to cereal, and would greatly appreciate input and assistance with.

I don't want contributions to reside in the main cereal repository if they 1) aren't part of the standard library or 2) can't be entirely self contained (for contributions that aren't serialization functions). For example, a new archive, assuming it was header only with no external dependencies, would be fine for inclusion into the main cereal repository.

I have two ideas for how to actually implement contributions such as these into cereal:

1. Each contribution would consist of a submodule in the appropriate location, along with a second submodule in the unittests directory. The upsides to this are that users can easily add features by just pulling a submodule and it will exist in a sensible location (e.g. cereal/types/boost). The downsides are that for development, each contribution is now split into at least two different repositories (e.g. boost and boost_unittest), making development more annoying.
2. Each contribution consists of a single submodule, residing in cereal/modules, and contains all necessary code and unit tests. As part of the installation of a module, the repository would be pulled and unpacked into the appropriate directories within cereal. This would be accomplished using some combination of CMake and/or scripts, such that a user could do something like make install_module_boost and end up with the appropriate code now residing in cereal/types/boost, unittests, or wherever else.

The second option is currently my favorite of the two though it is more work to set up. It will be easier on developers because everything can be contained in a single repository.

Ideally I would like repositories to be contained here, with ownership rights shared amongst maintainers of the module as well as cereal (though neither idea presented here requires this).

---

For now, feel free to add your contribution to cereal/types/boost/XXX/YYY (along with an appropriate unit test) until the above system is in place. I'll do my best to devote some time to cereal in the coming weeks to start to address this all.

[stevehickman]

The work was moved to the develop branch. See this pull request: https://github.com/USCiLab/cereal/pull/282

What you can do is pull from my repo: https://github.com/stevehickman/cereal/tree/develop

make your changes and then send me a pull request. I’ve integrated several other boost contributions. That way all the boost contributions will get added in pull request 282.

NOTE: I am 2 updates behind the USCIlab develop branch. I’ll get that taken care of no later than this weekend. That shouldn’t affect any work you do.

Also – when you pull from my develop branch, take a look at how I structured the tests. There are several things there you can take advantage of if you like. If you have any questions, ask. I’ll use your questions and update the readme with the answers.

Regards,

Steve H.

From: qiao [mailto:notifications@github.com] Sent: Tuesday, September 27, 2016 11:40 AM To: USCiLab/cereal cereal@noreply.github.com Cc: Hickman, Steve Steve.Hickman@honeywell.com; Comment comment@noreply.github.com Subject: Re: [USCiLab/cereal] Create "modules" for types not in the standard library (#123)

I plan to implement boost::graph save/load functions. This thread seems ended in 2014, but I can't find the conclusion about 3rd party data structure serialization. There is no "module" nor "contrib" folder for community work.

Is there any ways to provide our code?

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/USCiLab/cereal/issues/123#issuecomment-249957687, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AGVK-GC4YrkKs3UT2DhVzUfaaye_UBlZks5quWL-gaJpZM4CiLCV.

[AzothAmmo]

I've started working on #139 which is a core requirement for modules. I'm also going to be removing the dependency on boost test in favor of doctest.

[AzothAmmo]

A small update on this - the current plan is to use a cross platform CMake based module system (e.g., https://github.com/iauns/cpm, https://github.com/ruslo/hunter, or https://github.com/tamaskenez/cake) to "install" modules into an existing cereal installation.

The idea is that if you wanted boost support, this would simply be an option you specify to CMake, which would then automatically download the required files and update CMake flags as appropriate. Modules would have a CMake file that describes how their directory structure should be integrated into cereal proper.

Consider a hypothetical boost module, which might have an organization like:

|- license.txt
|- install.cmake
|- unittests
 |- boost_optional.hpp
 |- boost_optional.cpp
 |- etc
|- include
 |- cereal
  |- types
   |- boost_optional.hpp
   |- boost_chrono.hpp
   |- etc

This would get placed in the appropriate locations within cereal, allowing for things like unit testing to now additionally compile the unit tests from the module. CMake could also keep track of the installed modules and remove them if necessary.

This approach keeps a clean separation of cereal proper and modules. It allows us to easily support a mix of curated modules (hosted here, with shared ownership between a maintainer and cereal: https://github.com/cerealcpp) but still allow any repository to be a potential source, so long as it has an appropriate CMake module file that explains how it should be installed.