Interest in google flatbuffers integration?

[ahundt]

I've implemented an AzmqFlatbuffers class to help provide google flatbuffers integration. I'm sure it needs code cleanup and perhaps some design and implementation changes, but I would be interested in contributing it if there is interest in adding this integration.

Since it does add a dependency on flatbuffers I would suggest it be explicitly included by users who wish to use it.

[rodgert]

I am not opposed to the idea of support for flatbuffers but I'm not sure as written, this is the way I would like go about it. Let me collect my thoughts and follow up more fully.

On Fri, Mar 27, 2015 at 8:10 PM, Andrew Hundt notifications@github.com wrote:

I've implemented an AzmqFlatbuffers class https://github.com/ahundt/robone/blob/develop/include/robone/AzmqFlatbuffer.hpp to help provide google flatbuffers http://github.com/google/flatbuffers integration. I'm sure it needs code cleanup and perhaps some design and implementation changes, but I would be interested in contributing it if there is interest in adding this integration.

Since it does add a dependency on flatbuffers I would suggest it be explicitly included by users who wish to use it.

— Reply to this email directly or view it on GitHub https://github.com/zeromq/azmq/issues/70.

[rodgert]

• Introducing a dependency on Google's FlatBuffers; It looks like there is a CMake Find module for it at -
  https://github.com/google/flatbuffers/blob/master/CMake/FindFlatBuffers.cmake So making builds and tests conditional on it's existence should be straightforward
• In regards to the approach, if you want to pool flat buffers, it might make more sense to do that pool as an asio::io_service::service which can be associated with the supplied io_service as needed (it is then accessible to any code that knows to ask for it). See - http://www.boost.org/doc/libs/1_57_0/doc/html/boost_asio/examples/cpp03_examples.html#boost_asio.examples.cpp03_examples.services http://www.boost.org/doc/libs/1_57_0/doc/html/boost_asio/reference/io_service__service.html

This is how azmq::sockets are (and indeed most of Asio is) implemented.

• I am not convinced that FlatBufferBuilder provides a suitably 'empty' state if moved from (though it does nothing to prevent you from moving them, and it uses the pre-C++11 trick of making copy/assignment private, yet claims to require C++11...O_o), so you can't store them by value. In the process of looking at the code in -

https://github.com/google/flatbuffers/blob/master/include/flatbuffers/flatbuffers.h

to figure out if this is possible leads to the general observation that the FlatBuffer C++ code is slightly nasty at least from an idiomatic C++11 standpoint.

• There is some desire to be able to make azmq compatible with standalone Asio, so you might consider not using boost::circular_buffer to manage your pool (std::vector<> should be fine this sort of thing).
• Do you really need to model shared ownership of these buffers/builders, or is it really just instances which move from pool to pool during their lifetimes?
• I still need to finish both the refactoring of the code and associated blog post that describes how async operations are handled by azmq (which is modeled after Asio's approach), but this functionality would likely be better expressed in terms of azmq::detail::receive_op/send_op derived types.

[ghost]

FlatBuffers is not idiomatic C++11 because we originally decided to use Visual Studio 2010 as our minspec on Windows. I'll re-open the discussion if we can move to a more recent version, and use delete.

Not sure if I understand your other concerns (not familiar with ZeroMQ), but I'd be happy to help resolve them.

[ahundt]

@gwvo may be able to correct me if I am mistaken, but flatbuffers are designed to be read/written at extremely high rates, so large performance benefits come from allocating FlatBufferBuilders once.

Ownership doesn't need to be shared, it was just convenient for me to work around move semantics not working for lambda functions in C++11 by just copying the shared pointer. If FlatBufferBuilder becomes properly movable (not 100% sure how to tell yet), moving them with no pointer at all would probably be the best way to handle them, but I'd still need to deal with the move/lambda problem. I know there is a bind and std::ref workaround, but that is also pretty ugly.

Creating an FBB service sounds like a good design to me since they are only needed during the serialization stage though downsides might be having to either lock or post to access them. Are there examples of safely moving data into and out of them with multiple threads around as well?

[rodgert]

The sentiment was motivated by requiring C++11 but not being idiomatic C++11...'nasty' was overstated (MSVC2010...grim, understandable though).

I think the main question was about flat buffer types being movable (so they can be stored in a container by value) but having a private ctor and assignment operator precludes that (also the custom vector type would require a bit of work before being movable I think).

On Wednesday, April 1, 2015, gwvo notifications@github.com wrote:

FlatBuffers is not idiomatic C++11 because we originally decided to use Visual Studio 2010 as our minspec on Windows. I'll re-open the discussion if we can move to a more recent version, and use delete.

Not sure if I understand your other concerns (not familiar with ZeroMQ), but I'd be happy to help resolve them.

— Reply to this email directly or view it on GitHub https://github.com/zeromq/azmq/issues/70#issuecomment-88640108.

[ghost]

Sure, I'll see what I can do.

[rodgert]

"Creating an FBB service sounds like a good design to me since they are only needed during the serialization stage though downsides might be having to either lock or post to access them. Are there examples of safely moving data into and out of them with multiple threads around as well?"

You could make an FBB pool per unique requesting thread (perhaps using Boost's thread_specific_ptr). The main goal here is (IIUC) to reduce allocations not necessarily to minimize memory footprint.