In-memory state persistence

[araobp]

In-memory state persistence

A very simple proprietary data base will be implemented for in-memory state persistency as a tentative solution (You could make them stateless and use SQL, Cassandra or Hazelcast instead).

The enhancement:

• ODENOS appends every POST/PUT/DELETE operation(Request and Reponse classes) and EVENT(Event class) to an append-only file on a disk.
• When restarting ComponentManager, every Odenos RemoteObject re-loads every CRUD operation and EVENT from the log file and re-construct the in-memory state including a topology and flows.
• As an option, the re-load process stops at a certain timestamp to check a snapshot of in-memory state for a debugging purpose.
• Or it provides a management command to append a tag (not timestamp) to the log, and the re-load process stops at the tag.
• Log file compaction or snapshot feature??? Maybe...

  Sequence consistency

Since ODENOS runs in multiple threads, you might think that the sequence can be different between the logging time and a restarting time.

However, every RemoteObject runs in a single thread (Actor model) and every request is triggered by an incoming event (Request and Event, i.e., event-driven), so the sequence is always consistent.

While resuming the in-memory state, every RemoteObject instance as an actor is completely isolated from the others.

The only message that is not initiated by an incoming event is "PUT /systemmanager/component_managers/...".

MessageDispatcher instantiates OdenCookBook for each target RemoteObject instance for the resume process:

[OdenCookBook 1] <-----> [RemoteObject instance 1]
[OdenCookBook 2] <-----> [RemoteObject instance 2]
[OdenCookBook 3] <-----> [RemoteObject instance 3]
   :

The Oden cook makes delicious Oden for the guests.

That is something like RANCID using "expect" on Redis to configure ODENOS... OdenCookBook uses Java Iterator.next() (a list of Oden recipes) to make a next action (to make delicious Oden!).

"expect"-like method in OdenCookBook.java

"expect"-like method is used for synchronization between OdenCookBook and a target RemoteObject instance:

• ComponentManager sends Request autonomously at the beginning => "expect" this.
• All other RemoteObject instances never send Request autonomously.
• Sending Request is always triggered by incoming Event => "expect" this.

OdenCookBook can resume the state if the target RemoteObject instance follows the rule above.

Synchronization with other RemoteObjects (local or remote)

Use ZooKeeper for global locking: while restarting RemoteObject, the other RemoteObject instances cannot send a request or an event to the restarting one. Maybe, all the incoming requests from drivers or RESTTranslator must be blocked while restarting.

Future direction

[Idea 1]

Make all RemoteObject instances stateless and store all states in open source data bases.

• SQL: transaction supported
• Cassandra: tunable consistency, global deployment on cloud
• Hazelcast: Java only, embeddable, tunable consistency
• Any others.

[Idea 2]

• System restart and re-initialization (synchronization with external components) ==> From my experience implementing this tentative "Oden" data base, I think such synchronization is really hard to implement and can not be proved to work correctly (may not be able to resume the previous state correctly).

[araobp]

I have almost completed the coding, but found a problem. Some of ODENOS REST APIs are not transactional: returns 200 OK even if the request has not been confirmed yet.

All the REST APIs and internal APIs should be transactional to keep state consistency among distributed objects.

And it may not be a good idea to keep state of a distributed object in another distributed object. Use a data base supporting HA and Ephemeral mode instead, if it is for a data caching purpose.

Use asynchronous events for state update only.

[araobp]

This feature is very hard to implement, because it is impossible to prove that reconstructing in-memory state from log works for any kinds of ODENOS RemoteObject.

For example, Aggregator behaves very differently from Logic components.

So I stop implementing this.

[araobp]

I have come up with a different method, so I reopen this issue.

[araobp]

It has turned out that a request sending sequence of "operator component" such as Aggregator is inconsistent, so this tool (expect-like tool) does not work very well.

This issue is closed.