HW-Integration: Serializers

[eichelbe]

TSI: Check whether the official Kryo-Serializers can be used for HW Serialization? SUH: Include serialization/deserialization-Methods into data tuples.

serializeHw(HWStream stream) { stream.write...(field1); stream.write...(field2); }

deserialize(HWStream stream) { field1 = stream.read...(); field2 = stream.read...(); }

[ap0n]

The HW server is written in C. As far as I know, Kryo cannot be used. We should find an alternative (JSON, protocol buffers, ...)

[eichelbe]

Of course, we need a bridge between Java and C. The actual idea was to use a Kryo interface on the software side for doing the serialization for HW and optimized Storm execution with the same code while replacing the Storm Kryo serialization with something that is suitable for both, software and hardware. However, the required interfaces of Kryo explicitly refer to classes instead of interfaces (see below, Kryo is a class rather than an interface).

Style I: Class-external serlializers (implements KryoSerializer)

public void write (Kryo kryo, Output output, T object); public T read (Kryo kryo, Input input, Class type);

Style II: Serialization implemented in the respective data classes.

public void write (Kryo kryo, Output output); public void read (Kryo kryo, Input input);

Personally, I like style I as it does not pollute the data classes and their interfaces (although we initially sketched the style II as given above).

So I think that we need something like this but relying on own interfaces (in StormCommons). This allows that serialization can be described independently of the underlying format (protobuf, json) and depending on the actual performance, you may even change the implementation without need for regenerating pipeline code or the instantiation scripts.

Would you like to make an initial sketch for the design or shall we do that?

[eichelbe]

There is a potential relation to the discussed data storage metadata in #25

[ekateriniioannou]

Nick, Apostolos, Gregory: please discuss and suggest possible solutions (e.g., 1-2 slides)

[eichelbe]

Any news here?

[gregorychrysos]

No news yet! :-( I hope in the next week we will suggest a solution...

[eichelbe]

After no suggestions and several hours debugging in simple hardware algorithm for experiments... Initial suggested version for software side in DataManagementLayer/serialization. Please discuss.

[eichelbe]

updated for simple db keys

[eichelbe]

Intended application:

• For hardware integration, create an instance of HardwareSerialization streams and read/write objects via them. Needs implementation of software/hardware-side at TSI.
• For DataManagementLayer: DataManagementBolt passes declarer instance to DML so that schema can be checked / created. While creation, DataManagementBolt obtains write stream from DML and serializes objects step-by-step.
• To be done (if agreed): Create serializers during infrastructure instantiation.

Reassigned to remind about discussion.

[gregorychrysos]

Well, I have to ask some things about this HardwareSerialization :-) :-), as I am not familiar with that...

First, we need to define an Interface which will be common for all the sub-topologies that will work with hardware. In this topology, will implement a write function, which will receive a Serializing object (Kryo is for Storm, right?) from the pipeline, will parse it and will send the data "down" to hardware. Also, we are going to implement a read function, which will receive the results from the hardware and and will create a serializing object which will be sent to the pipeline. My worry is that this object is not going to be the same for all the algorithms(it will have different data and fields). So, what is going to be done in that case? Also, if we use a differently structured object for each algorithm, why are we going to use serializers(which are going to decrease the performance, too)? Is that because we are going to have a common interface for all the algorithms? Sorry about the size of the message! :-)

[eichelbe]

No problem - mine is longer ;)

Indeed, the idea is to have a common interface for all algorithms and to have a similar performance as Storm provides through kryo. Actually, the type-specific serializers in kryo (and also in the proposed set of classes) rely on the sequence of data and do not need generic metadata (as Java does), i.e., you write what you consume (in terms of types) in the same sequence. As you need to write the data anyhow, this is just a kind of visitor pattern on the objects typically avoiding all the overhead that you have with generic Java serialization. So there is a common interface, which is still specific for the individual types.

And yes, the "object" to be send may differ from algorithm to algorithm, depending on how the pipeline is defined. The individual algorithms then know through a family-specific interface on which types they can rely on. The nice thing is that we know how the objects look like and that we can derive the additional (singleton) classes for serialization (for kryo and for whatever we will do) directly from the pipeline configuration. So no worry for the software side ;)

Although we could use kryo so far, the problem on the DML side is that you have to deserialize an object explicitly before writing it and that you do not know its internal structure. This is the reason why we proposed to have extensible streams so that you can either turn the Java primitives in something that is more adequate (e.g., a direct call to HBase or whatever) and that you can embed additional information if needed. For the DML there is an additional declarer inspired by storm which provides the meta data so that a database schema with keys can be set up if required.

For hardware, you may have also different requirements on the actual format of the data (although you might not need the metadata). So far, we had the assumption that you follow the domain-specific interfaces that we use for the software side, i.e., there is a software-sided input/output serializer stream implementation that turns the data into a format that you like (and turns it back for software), embeds additional information if required ("here starts a tupe", "here ends a tuple") and supports the protocol you use for communication (as extended with Cui). This would need at least some similar framework on your side with specific (singleton) classes / records / function prototypes for serialization / deserialization and, finally, it is your choice whether you turn the data into records, classes or just write them accordingly into / read them from memory so that you can transfer them into / from the DFEs.

But as mentioned above, this is just a quick proposal as there was none so far and I'm not quite sure whether it really fits to hardware - but we shall figure out what is also adequate for your side ;)

[eichelbe]

... and we will need it for the state transfer (D4.2)...

[eichelbe]

.. current version does not handle arrays, containers and polymorphic types. Let's see whether we need them and in what form.

[ekateriniioannou]

Gregory will send the current code to check what can be done

[eichelbe]

Update: Gregory, Cui and I discussed the examples / suggestions and we will do a first round on the example algorithm used for measuring

[gregorychrysos]

Update: I searched for Kryo serializers and deserializers that work for C and unfortunately they do not exist... Also, the parsing and the decoding of Kryo serialized data is a really tough task. I found kryococoa, which seems to serialize and deserialize Kryo objects in Objective-C. I will try to make it run with Maxeler platform and I will inform you. 1) Is there any chance for changing serializing/deserializing scheme, if I do not manage? 2) Maybe I will need a Kryo serialized object, for example an array of integers, in order to test the deserializing scheme (if I manage to run it, of course:-) :-) )

[eichelbe]

Hi, it's not a big deal if it kryococoa does not work, as we anyway aim at a solution that also helps the data management layer (what kryo does not) ;)

[gregorychrysos]

That's great, as it does not seem to run! :-) Could we try Google protobuf? As, they seem to implement both Java and C++ solutions? Will it help you with the data management layer?

[eichelbe]

Protobuf does not seem to have the genericity we would like to have to make the generation of the Bolts easier, but at a glance it seems that we can add this on top and use protobufs below. This means that the infrastructure derivation well become a bit more complex (generation of .proto file and serializers) as well as the build process becomes more complex (but there seems to be a maven plugin). We should give it a try, because we don't have to worry for platform-independent serialization of several datatypes...

[eichelbe]

Added Patrick... could protobuf help the Data Management Layer (writing)? If you, we are about working with 3 mechanisms...

[gregorychrysos]

As, I missed that! Are we going to try protobufs or it is a ot of work from your side and we should think of another solution?

[eichelbe]

Patrick is in holidays... From our side, I think that we can wrap the protobufs behind a stream class so that they become transparent and do not interfer with whatever we need for the DML. The only potential issue that comes to my mind is that we may have to describe the communication messages in the .proto file while the data object needs to remain generic. But we should give it a trial, then we will see whether this leads to a real problem or whether we can implicitly solve it. Could you please do a first round on the hardware side, let us know, and we do then the corresponding software side based on your .proto file?

[gregorychrysos]

Yes, I will try it and as soon as I have any news, I will inform you! I hope I manage to run it on hardware!

[gregorychrysos]

Update: I installed the protobuf framework and managed to run the C code on a Maxeler server. Now, I am trying to integrate it in a hardware-based Maxeler project in order to be able to run it inside the server side of hardware-based Qualimaster side.... Did you try to run the protobuf framework via storm? If you try, just ask me to give you some hints as I ran it as java project, too! :-) :-) :-)

[eichelbe]

Fine, could you please send us / attach your .proto file ;)

[gregorychrysos]

The .proto file is just an example... I prefer first to integrate it on the Maxeler project and afterwards to make a proto file, which would be near to our algorithms.... I will send the proto file via email....

[eichelbe]

Ok... the one for the simple algorithm would be helpful for us to think about and to start...

[eichelbe]

Thanks for the person example. Compilation with Maven is working... Did you proceed with the simple experiment algorithm?

[eichelbe]

We have now a running Java version against your example... We need to discuss how to handle your hardware protocol with that and in particular algorithm-specific types...

(just generating the respective calls into the software side should not be difficult Cui said ;))

[gregorychrysos]

At last, we have a working C version for the google protobuf running on the Maxeler server. I tested it with the Java implementation that I sent you and it works fine. Now, I think that we need to move on testing the serializer on the pipeline infrastructure with a simple test... Please tell me whenever you are ready to start this testing...

[cuiqin]

@gregorychrysos good news:). have you also considered the algorithm-specific types (object types)?

[gregorychrysos]

One step at the time! :-) :-) No, actually I think that we need first to test it with a simple example, as we did the previous time and then define the exact interface of the message... Another good thing is that, out implementation is generic, which means that we need simply to define a new message(proto file) and it will work with very little changes...

[cuiqin]

The test pipeline as well as a simple test algorithm for the serialization are ready... to be tested with the hardware.

[gregorychrysos]

It seems that we have a small problem with the serialization... The C++ part of the google protobuf does not support the deserialization in all cases as Java does. I tested with simple messages and it worked but when the messages are more complicated, then the C++ part does not support all the needed functions. Thus, we have to define exactly the structure of the packet in order to deserialize the incoming packets using custom functions.

[eichelbe]

Update: Design discussion is going on by mail ... transparent/generic vs. explicit integration. If only the explicit integration provides the needed functionality on C++ side (last comment by Gregory), we may also go for that.

[eichelbe]

Update: we have a principal agreement on the design / message sending (via a mini-protocol), now working on a handcrafted example for pipeline type and algorithm-specific type... to clarify the mini-protocol ;)

[gregorychrysos]

Update: we managed to serialize-deserialize and integrate custom classes-objects into TCP packets. Also, "custom" packets were exchanged between SW (Java local) and HW (Maxeler) implementation... The next step is going to be the testing on the pipeline infrastructure.

[cuiqin]

Update: we generated the proto files along families as well as the protobuf serializers. They are now available in the PriorityPipelineInterfaces jar. The first version of the hardware algorithm generation will follow...

[cuiqin]

Update: Gregory and me managed to run the test pipeline with protobuf serialization in the cluster. The next step will be to evaluate the performance of the hardware integrated serialization.

[cuiqin]

Update: The hardware integration about protobuf serialization is done. As also evaluated, it has an improvement on the throughput. Further, the hardware algorithm can be generated based on the integrated serialization and the generated hardware example algorithm has been tested in the cluster. Next step: turn the correlation hardware algorithms to the generated version.

[eichelbe]

done