Serialization Factory mandatory?

[minimok]

Hi,

I'm experiencing a problem - Hz is insisting I need a Serializer factory and matching Java pojos for everything (other than a primitive) I want to store in a Map. Really?? I don't need to write my app in 2 languages...

What's the "common sense" solution for this?

[mustafaiman]

Hello @minimok Hazelcast needs to serialize objects in order to be able to keep them in server memory. For primitive types, it uses Hazelcast native serialization. For other complex types (e.g. JS objects), it uses JSON serialization. If you access Hazelcast only from Node.js client and not from Java you should not experience any serialization issues.

Can you give more information about your problem? Preferably;

• Your Node.js client version
• Your Hazelcast server version
• The error message
• Do you write/read objects using only Node.js client or do you also manipulate objects from Java(or any other native client language) side too?

[minimok]

Hi, The issue only arises when I try and do something 'active' with my data. By 'active', I mean e.g. querying with a Predicate. So, with e.g. an object like:

 class Order: {
    constructor(orderType, qty, price, item){ 
        this.orderType = orderType
        this. qty = qty
        this. price = price
        this.item = item
      }
}

There is no issue in storing this object in a map, or putting it on a queue...until...I try and retrieve all Orders where item == 'myItemId' or where price > 24.75 etc. A this point I get the standard error referring to no Serialization Factory for type -130.

The identical problem (type -130 error) also kicks in if I am trying to use a similarly simple object, not querying it, but reading/writing it to/from a MapStore impl (I'm using a Spring Data MongoDb for my MapStore).

But - when I go ahead and create a Pojo, plus Java Serialization Factory (using an IdentifiableData factory with Class and Factory ID of e.g. 42) the issue disappears completely, and I can query my Map perfectly as well as storing it in MapStore. This is great - but it requires maintaining 2 data models in 2 languages - not too 'elegant' really. It also becomes problematical if I add an array to my 'Order' object - e.g. class Order { ..... this.statusUpdates = [] }

At this point, I have 'nested' objects and the Java side reverts to type...saying it doesn't know what to do with the object. I've tried using Array and ArrayList - but both fail irritatingly. I was eventually reduced to storing my nested objects as JSON strings and using readUTF / writeUTF...but that's borderline "lunacy"!!! So my new solution is that no nested objects are allowed...and I use a separate Map for each nested object. But consider an object like: Employee: --empQualifications (array of qualification objs) --empWorkCalendar - a calendar --empPreformanceReviews - some reviews objects ...

This object might require 20-30 different Maps!!

Naturally this is an exaggeration - but these are the kind of issues that arise when the only available docs / guides are of the form:

myMap.put("A", 1)

If serialization is a key part of the product, then at least show people how to use it! Do this with a non "Mickey Mouse" example :-)) It took me around an hour to find the relevant docs on ObjectDataOutput on the Java side and I still cannot see what method to use that would successfully allow nested Objects to be serialized (writeObject() doesn't work). I can keep trying to work this out from first principles or I can use either Redis or Mongo (both of which are incidentally several order of magnitude faster!!) - as a potential user of Hz - that's the choice you leave your users with when you don't hire a technical author to do product documentation properly. There's a common belief among people who manage devs that "lots and lots" of documentation is the same as "good" documentation...the Hz website is an absolutely classic example of this problem. I've bookmarked the Javadocs because I know that I'll never, ever find them again....

[mustafaiman]

Hello again,

When you try to query your data using predicates, this querying is handled on the server side so hazelcast does not have to bring all data to client but only the relevant entries. Otherwise, there would be a lot of unneccessary data traffic between the client and the server and the performance would severely drop. Because predicates run on the server side, the server should be able to reason about your objects. That is why you need to implement serialization on the server side as you did.

A similar explanation is true for MapStore. The server should be able to deserialize your objects in order to store them in MapStore.

Regarding arrays in a serializable object, you can use methods like writeIntArray if the array is of a primitive type.

If you have nested objects, these nested objects also need to be serializable. You can register the serializers for nested objects just like you did for the encapsulating object. Then, writeObject method will not have any problem with finding suitable serializer for and writing/reading the nested object.

If you have arrays of custom objects, you can serialize/deserialize them like the following:

writeData(dataOutput) {
    ...
    dataOutput.writeInt(this.arrayOfCustomObjects);
    this.arrayOfCustomObjects.forEach(function(element) {
        dataOutput.writeObject(element);
    });
    ...
}

readData(dataInput) {
    ...
    var arrayOfCustomObjects = [];
    var lenOfArray = dataInput.readInt();
    for (i=0;i<lenOfArray;i++) {
        arrayOfCustomObjects.push(dataInput.readObject());
    }
    this.arrayOfCustomObjects = arrayOfCustomObjects;
    ...
}

Java server side implementation is similar.

I see your concerns about the documentation of serialization. You spent a lot of time finding the Javadocs but I also wonder if you were able to find Node.js API docs quickly at http://hazelcast.github.io/hazelcast-nodejs-client/api/0.5/docs/ Also you can take a look at the serialization of cutom objects in our tests. (e.g. https://github.com/hazelcast/hazelcast-nodejs-client/blob/master/test/serialization/AnIdentifiedDataSerializable.js )

For your convenience, you can also use the following alternatives when you think the documentation falls short. node.js client gitter channel: https://gitter.im/hazelcast/hazelcast-nodejs-client Hazelcast google groups: https://groups.google.com/forum/#!forum/hazelcast

And of course you are more than welcome to open github issues as you see fit.

[minimok]

Thanks! Yes - I have to say that the Node.js documentation (and ease of locating the right thing quickly) from your team is a very different quality from the material on the java side! In fairness once you find the java stuff it's often ok, but there's now so much of it that it's very difficult to find what you need. The best way to get people using a product, in my experience, is a proper set of 'worked' examples... This link is from a few years back - but to my mind represents the Gold Standard of how to get information across to a user (the originals were accompanied by detailed blog posts, and step-by-step drops of code): I think replicating this level of instruction would do no harm at all :-) Thanks again!