How to understand cloud-oriented design? OLTP?

[drmingdrmer]

Try best to utilize cloud service to break the limits that are normally found in a program designed for the local machine, such as:

• WAL can be replaced with Raft or Kafka or other queue/log service that provides strong consistency.
• Data blocks(SSTable) can be stored in S3 for reliability and replication.

An application on the cloud has almost unlimited resources to use if you got enough money.

Thus money-oriented optimization will be considered: Data storage is cheap, while data IO is expensive. Data structures should be friendly with object-store. The data index has to be as small as possible.

Globally synchronization is expensive: locking is difficult to be done correctly in a distributed environment. Snapshot-based designs are more considered.

Message transmission becomes more expensive across nodes. Data layout has to be considered if cross-region deploy involves.

Algorithms that are friendly in a distributed system will be considered. Such as relatively relaxed consensus protocol(CRDT or else), non-linear WAL(log) structure.

[bytesleak]

Thank you for your reply, I get the idea. Will it consider scalability under multi-core?

[drmingdrmer]

Thank you for your reply, I get the idea. Will it consider scalability under multi-core?

It sounds like a has-to-do. :)

The internal sharding will make it generally friendly to multi-core env: https://datafuselabs.github.io/openkv/arch/sharding.html

And most of the data in an LSM are static SSTable(on disk or cached in memory), this makes data sharing across cores more efficient. No need to worry about cache line invalidation or synchronization.