Save and load state - persistence

[ArjunVachhani]

order-matcher should have methods to save state/serialize and load state de-serialize. so that state can be persisted and restored after restart. Also it can be used to save state and move and load it inside different machine. which can help in maintenance also.

[avatar-92]

Hello! I understand that matcher must keep the state in memory for high performance without any communications with database or file system. This matcher not thread safe, it means you must do save state on the another thread or sacrifice performance. If, while saving the state, we save the entire order book, we will spend more time with a larger number of orders and may not keep up with the match. If we will save changes only for changed orders, there is a possibility that the state will not update for one order if the application crashes. What are your thoughts / plans for keeping the orderbook state?

[ArjunVachhani]

@avatar-92 All events that modify state of matcher should be logged to persistent storage before they modify the state. This is similar to Write-Ahead-Log(WAL) that well known databases (PostgreSQL, Redis etc) perform. In case of restart apply the WAL logs. WAL will keep on growing. Time to re-initialize order matcher from WAL will depend on WAL size. To reduce re-initialize time we could create order matcher snapshots and on restart only apply WAL changes happed after snapshots.

To achieve higher performance we could have separate threads to perform IO and order matching. IO thread will read incoming message and pass to order matcher thread via queue. Order matcher thread can execute order and pass result to another IO thread which will update database (persistence storage) or pass to Kafka/RabbitMQ accordingly.

[avatar-92]

@ArjunVachhani, thank you. It is great solution. I will try to do it.

[Amenocy]

in this case, if we do update persistent storage ( DB ), the bottleneck would be our second IO thread. how should we ensure that the previous change was committed into DB? because we will check DB records to manage user balances and accept or reject orders from users. I mean, if that second IO thread has noticeable latency(even in rare cases), we may face double-spending.

[Amenocy]

in my opinion service state shouldn't be different from the application state.

[ArjunVachhani]

in this case, if we do update persistent storage ( DB ), the bottleneck would be our second IO thread. how should we ensure that the previous change was committed into DB? because we will check DB records to manage user balances and accept or reject orders from users. I mean, if that second IO thread has noticeable latency(even in rare cases), we may face double-spending.

If your persistent storage is database and your database provide strong ACID property. then you could parallelize order and balance update.

in my opinion service state shouldn't be different from the application state.

could you explain a bit?

[Amenocy]

If your persistent storage is database and your database provide strong ACID property. then you could parallelize order and balance update.

let's assume we updated the Order status in the order-matcher service, which means that two orders matched, price changed, and order book changed. we will show these to the end-user(and they may make some decisions ). now we should update balances and orders in the database for two users with a transaction and even function ( in PostgreSQL) if there is any problem with these jobs, and transaction rolled back, can we undo the order-matcher service state? no. how can we maintain data consistency between the database and order-matcher service?
one solution is to add strict controls on adding new order. but there is no 100% warranty.

[ArjunVachhani]

Agree, transient errors can occur due to network, overloaded db, OS updates, etc. Proper retry mechanism can help there.

Also, order-matcher is stateless, restart and add orders in same order it will have proper state.