If matching server crash, orders will be lost ?

[KabDeveloper]

Hi,

Is there a way to recover orders in case of server crashed for some reasons ?

300k orders processed per second is this right ? What are the characteristics of the server used for that (RAM, Processor etc.) ?

Thank you

[nilber]

The engine is just part of an exchange, all the logic around you needs to work out and persist as best you can

[KabDeveloper]

@nilber Thank you

So the best way, is to store new orders directly to MySQL/Postgres and in same time process them in the orderbook (In Memory).

[nilber]

Hi @bcashier,

this you need to store and then in case of application restart you load your engine again with the data recorded in the database. In the last Exchange project I set up I got to use redis so as not to lose too much engine efficiency.

[KabDeveloper]

@nilber Thank you for your reply.

Using Redis is risky, since the datas are stored for a limited time only or if the server crash you will lose them too from Redis.

Since you have experience on exchanges, can you let me know how you managed the process to check balance of users, update them, add new orders and update them too etc. ?

Do you do this from Redis only or do you have database like MySQL (RDBMS) ?

Redis is used for caching market datas only no ?

But the important thing, is how the balance and orders are added to database from the matching server in a faster way without losing all the performances, aswell as websocket messages to notify users about the changes on the browser.

Thank you again

[nilber]

@nilber Thank you for your reply.

Using Redis is risky, since the datas are stored for a limited time only or if the server crash you will lose them too from Redis.

Since you have experience on exchanges, can you let me know how you managed the process to check balance of users, update them, add new orders and update them too etc. ?

Do you do this from Redis only or do you have database like MySQL (RDBMS) ?

Redis is used for caching market datas only no ?

But the important thing, is how the balance and orders are added to database from the matching server in a faster way without losing all the performances, aswell as websocket messages to notify users about the changes on the browser.

Thank you again

It is a very complex product, it is difficult to explain in text everything that is needed.

Redis has this downside, but with the right configurations and proper infrastructure you can solve this crash problem.

Briefly,

The management of balances is like a bank, you increase and remove as the user uses.

Before sending to the book, you remove the balance and send it to the engine. When the engine responds to you with the trade, you pay the bills their respective trade balance results.

There are other approaches, such as working with a locked balance as well, instead of just removing it, you move to a locked balance column.

What exactly are you intending to create? or is it just for study?

[KabDeveloper]

@nilber

About balance you didn't got me right, I know that it is stored on RDBMS datanase and updated as the user trade, deposit, withdraw etc ...

The question is: Since this order matching program can handle up to 300k orders per second, adding a line of code to update the balance of let's say around 15000 traders per second inside RDBMS this will make the program too slower and will handle only few orders per second...

In addition to the balance updating, there is too:

• Storing order history of actual trade to RDBMS
• Sending notification to browser using websocket for real time results
• Updating market data inside Redis

There many additional operations that will make this order matching server slower ... so how this is managed ?

Using Apache Kafka (That support around 1M messages per second) will help to send new trades from web server to the engine, but how about other operations ?

I am entending to learn how a cryptocurrency trading platform works and study the possibility to make it faster with the micro architecture application.

Waiting your answer, thank you.

[nilber]

@bcashier

My current engine supports 2.5M inserts per second, in fact it is complicated to set up an infrastructure to support all this load. (go + redis + ws + amqp)

But I would recommend you not worry now, you first need to learn and model / build all exchange behaviors and then try to optimize and see where you can work to make it faster.

Try a beta so you can understand how it works, then see where it is slower and work on these to optimize your platform.

[KabDeveloper]

@nilber

I didn't found an appropriate one for that... my only issue for now is to look for a faster way to fetch users balances from database and then update them quickly when order is completely or partially filled.

Do your solution open source ? Can you share the modal used please ? Are you refering to this: https://github.com/nilber/go-exchange-order-book

Honestely, I am on this for up to 7 months now without a concrete solution.

Can you share your email for more infos with you please ?

Your help will be appreciated.

[nilber]

@bcashier

https://github.com/nilber/go-exchange-order-book is just test, It was a fork project that I was just testing, I don't recommend using.

I understand your timing is really complicated.

Relational database operations won't support such a high load per second, if that's what you really want, you'll have to go to a disk-persistent in-memory database.

In fact, there are still other strategies, such as distributing the API / Book on more than one server to support such a high load.

My exchange project has many separate services, each working at a specific point.

I can recommend you to use redis, read its documentation, there are precautions for sudden falls, and how to back up.

It was the best solution I found for my systems.

In redis beyond orders, you can manipulate user balances.

[KabDeveloper]

@nilber

Even distributing them in a different servers, there will be many inserts into the database (Problem of matching server will be solved but not the database case since all of them will be connected to the same database)... I am stuck on this. (Using MySQL as database will be complicated for my case, since with MySQL cluster using 16 data nodes, I can handle 2.5M statments per second ... each node 2x Intel Haswell E5-2697 v3 CPUs, infos come from here: https://www.mysql.com/why-mysql/benchmarks/mysql-cluster/, and I can't afford to spend that much money on this and for micro architecture that is supposed to get awesome result with low cost and server performance).

What about 24hours volume / Change ? Do you use Redis for this aswell ?

Can you give me tips on how to process or documentation you followed for the Redis solution ?

Thank you again and sorry for your time :)

[nilber]

@bcashier

see https://redis.io/topics/persistence

My engine test

https://ibb.co/qC0dwdP https://ibb.co/RvwfwWY

Server 8 cores x Intel(R) Xeon(R) CPU E5-1630 v3 @ 3.70GHz 4Gb RAM

[nilber]

@bcashier

api/ws --> redis || api/ws --> redis || api/ws --> redis || => AMQP cluster => Engine => AMQP => services => callback to api/ws api/ws --> redis || api/ws --> redis ||

[KabDeveloper]

@nilber

As AMQP solution, you are using RabbitMQ ?

[nilber]

yes

@nilber

As AMQP solution, you are using RabbitMQ ?

[KabDeveloper]

@nilber Between Engine and AMQP, do you use Apache Kafka or directly passing message to RabbitMQ ?

Sorry for all my questions, It's a while I didn't found a qualified person on this field :)

[nilber]

@nilber Between Engine and AMQP, do you use Apache Kafka or directly passing message to RabbitMQ ?

Sorry for all my questions, It's a while I didn't found a qualified person on this field :)

I do it via golang, I tried to use Kafka, but it didn't work very well on my macbook, when trying to send 100k of messages it stopped working or didn't deliver all the messages, I just gave up (didn't try to solve the problem), just changed for rabbitmq and had no more problems.

But if you can configure it well, I recommend using it, I saw good recommendations.

[KabDeveloper]

Ok :) I will take note.

api/ws --> redis || => AMQP cluster => Engine => AMQP => services => callback to api/ws

Why not passing datas directly from Redis to Engine (Using PubSub) ?

[nilber]

Ok :) I will take note.

api/ws --> redis || => AMQP cluster => Engine => AMQP => services => callback to api/ws

Why not passing datas directly from Redis to Engine (Using PubSub) ?

At the time I didn't use it because pub / sub do redis has risks, if the connection goes down it loses the messages.

[nilber]

hi @bcashier This part of infra was with another team, I do not have many details, I recommend testing to try to get the best results.

my workflow ws => Redis (storage/update balance) => AMQP => Engine => AMQP => services => ws (callback to client)

[KabDeveloper]

@nilber

Thank you again sir :)

What about passing data from Redis to AMQP ?

ws => Redis (storage/update balance) => AMQP => Engine => AMQP => services => ws (callback to client)

How is it done please ? Sorry again for all my questions :(

[nilber]

@bcashier Websocket application in Go, it connects to redis, writes and then sends it to AMQP if all went well.

[KabDeveloper]

@nilber

Sorry for taking so much of your time... I really appreciate your efforts for answering my questions :)

[nilber]

@bcashier You're welcome, But I recommend testing a lot, every case so you can understand and find the best performance.

[KabDeveloper]

@nilber

I will ! Your advice is awesome I will surely take note and follow it :)

You are the best, thank you again :)

[tpfwrz]

@nilber may I ask what data type is most efficient for storing orderbook in redis?

It would not just be for caching either, so would want to make it as efficient as possible.

My first approach to the problem would have to be key1 - instrument:asks -> set or list of ask IDS key2 - instrument:bids -> set or list of ask IDS

key3: ask:$id -> hash of the ask data (price,qty,id) key4: bid:$id -> hash of the bid data (price,qty,id)

I fear that the latency from resolving the set / list of ids to then getting each individual hash data would be too much, even with using pipelines.

The other option I guess would be instead of storing ask ids in the set, store the actual JSON stringified object of the hash

This would mean its only one query to get all asks / bids.

Crazy other option would be to just store whole orderbook as json stringified object in a string. but that is a terrible solution.

Hopefully what I described makes sense, I may be missing something.