bakwc / PySyncObj

A library for replicating your python class between multiple servers, based on raft protocol
MIT License
704 stars 113 forks source link

logCompaction data restriction #67

Open SantjagoCorkez opened 7 years ago

SantjagoCorkez commented 7 years ago

As of https://github.com/bakwc/PySyncObj/blob/master/pysyncobj/syncobj.py#L1282 logCompaction dumps all the properties of a SyncObj instance that are not listed in SyncObj.__properies (note a typo). That dataset, being collected via self.__properies surely includes subclass'es own properties.

Since the base SyncObj class is designed to be subclassed and some business logic to be implemented on top of this, there surely might be many properties related to that logic. In my case they include some data receiving/emitting sockets. Since sockets can't be pickled or serialized another way that leads to logCompaction failure.

  1. Note an attribute typo
  2. Can the data included to logCompaction mechanism restricted to not include some data programmatically? Another (and I think more preferrable) way: restrict logCompaction to collect only the data fields that are specially registered to be included.

For now I have to use an ugly method of self._dict__['_SyncObj__properies'].update({'myfield1', 'myfield2', ...})

bakwc commented 7 years ago

Thanks for report. There is several possibilities to achieve skipping members serialization. First one is declare all members that should not be serialized before running init of SyncObj:

class SomeClass(SyncObj):
    def __init__(self):
        self.a = 10 # will NOT be serialized
        super(SomeClass, self).__init__(...)
        self.b = 20 # will be serialized

Second approach is to declare your own serializer and deserializer function. And the last one is to stop inheriting from SyncObj, instead write your own class which will have SyncObj as a member, and also have one or many SyncObjConsumers. Consumers will store only state-machine data and logic, and all members that don't need to be serialized should be moved out of consumers.

Another (and I think more preferrable) way: restrict logCompaction to collect only the data fields that are specially registered to be included.

May be it is a good idea, but I don't want to make any significant changes into interface and break backward compatibility. But I'll try to think what we can do here. Current way is not very obvious.