search

corot / world_canvas

ROS framework for storing and accessing semantic information about the world, with an initial emphasis on needs and use cases for mobile robots.
10 stars 19 forks source link

Pickle don't have a C++ package installable with apt-get; use ros::serialization instead #19

Closed corot closed 10 years ago

corot commented 10 years ago

I found this http://www.picklingtools.com tools, that look good but must be installed by hand.

In addition, I have noticed that ROS-serialized arrays takes 1/3 of the space that pickle arrays do!

stonier commented 10 years ago

Ecto does some serialisation between python and c++ too. I think they tackle this from the other direction though. Use boost c++ serialisation libraries and then python probably just calls on those libraries to do the serialisation.

1/3....! I'd start to sniff for something smelly...that's a big difference. At the end of the day, the final storage for bytes shouldn't be that different amongst implementations. It's serialisation....not compression. Or is that something that is often done under the hood to minimise time of flight across the wire, which would be important for ros? Serialisation doesn't necessarily have to be used for 'across the wire' applications (think data dumps).

corot commented 10 years ago

mmm... using ROS serialization turned to be a pain in the ass, but it's finally working. I wanted to avoid calling python from C++ (pickle) or C++ from python (boost???), and so on with Java, etc. That's why I thought that using ROS infrastructure will be the mos portable. But given I had lost a weekend making it work... well, well... maybe it was not that good idea. But now is fine :muscle:

Yes... suspicious. But I think I have an idea of why: pickle serializations contain ALL the structure of the message, with field names, hierarchy, etc. So you can rebuild the message with 0 extra info. ROS serialization, however, only stores the data, and the message structure must be provided on deserialization. But... I'm puzzled why an occupancy grid is much bigger, if almost all is data, no message structure! :confused:

2014-07-13 8:50 GMT+02:00 Daniel Stonier notifications@github.com:

Ecto does some serialisation http://plasmodic.github.io/ecto/ecto/usage/advanced/serialization.html between python and c++ too. I think they tackle this from the other direction though. Use boost c++ serialisation libraries and then python probably just calls on those libraries to do the serialisation.

1/3....! I'd start to sniff for something smelly...that's a big difference. At the end of the day, the final storage for bytes shouldn't be that different amongst implementations. It's serialisation....not compression. Or is that something that is often done under the hood to minimise time of flight across the wire, which would be important for ros? Serialisation doesn't necessarily have to be used for 'across the wire' applications (think data dumps).

— Reply to this email directly or view it on GitHub https://github.com/corot/world_canvas/issues/19#issuecomment-48833451.

stonier commented 10 years ago

But... I'm puzzled why an occupancy grid is much bigger, if almost all is data, no message structure!

Yes...quite smelly.

Hmm, serialisation comparisons: ros is not really that cross-platform (although it does kind of have java at least) and it may change soon. Boost serialisation of course would be less cross-platform, but at least stable. There really is no other serialisation options that have cross-platform-language support (protobufs?).

corot commented 10 years ago

Last sentence is an affirmation or an answer? If the later... 몰라

2014-07-14 11:18 GMT+02:00 Daniel Stonier notifications@github.com:

But... I'm puzzled why an occupancy grid is much bigger, if almost all is data, no message structure!

Yes...quite smelly.

Hmm, serialisation comparisons: ros is not really that cross-platform (although it does kind of have java at least) and it may change soon. Boost serialisation of course would be less cross-platform, but at least stable. There really is no other serialisation options that have cross-platform-language support (protobufs?).

— Reply to this email directly or view it on GitHub https://github.com/corot/world_canvas/issues/19#issuecomment-48879618.

stonier commented 10 years ago

Oh, meant to be a question! I know in the ros 2.0 serialisation docs they list a couple. Protobufs is always the first candidate for an implementation that spans languages and platforms.

stonier commented 10 years ago

I don't think it's a huge issue right now though :) Serialisation for this shouldn't be too hard to swap in and out right?

corot commented 10 years ago

right

2014-07-14 12:14 GMT+02:00 Daniel Stonier notifications@github.com:

I don't think it's a huge issue right now though :) Serialisation for this shouldn't be too hard to swap in and out right?

— Reply to this email directly or view it on GitHub https://github.com/corot/world_canvas/issues/19#issuecomment-48884114.

corot commented 10 years ago

To ensure easy migration to other serialization tools, I put all in a separated package: https://github.com/corot/world_canvas_libs/issues/3