Open joepio opened 2 years ago
Perhaps there is some wisdom to borrow from 4store - or maybe ask its author, Steve Harris, to provide input here?
Thanks!
A different approach to multi-node, is FAAS. Convert all logic to stateless functions. Has some interesting benefits to scalability - no longer needed to copy instances and all their logic.
@AlexMikhalev has made some interesting progress on having a multi-node Atomic-Server setup. See the aws-ops
repo.
He suggests:
atomic-server
instance per userpulumi
(i.e. terraform that you can use with any language, like JS)fargate
to minimize costs. It currently uses 1 VCPU with 1GB RAMSome thoughts about this approach:
atomic-server
doesn't depend on an ENV for BASE_URL.Another approach:
Check out https://github.com/atomicdata-dev/atomic-data-rust/pull/463 - the sync between nodes was the whole point of using fluvio. The bit which I didn't finish - is to remap atomic data URLs inside the smart module before the record reaches the streaming platform.
Another approach:
Wherever we're currently using sled
, we could use tikv
(for example). This means instead of using an embedded key-value store, we'll use an external one that supports multi-node setups.
The big advantage is that Atomic-Server instances are no longer tied to specific data. In other words, tikv deals with the complexities of having a multi-node system.
However, there are still some stateful resources that are currently linked to machines. Plugins, specifically. Also, we'll still need to make sure that the watched_queries
are scoped to specific workspaces, but that's an easily fixable problem.
For now, let's ignore plugins. Because even if we fix that problem, I'm not sure the solution will be very performant. There is quite a bit of overhead required when our storage system is on a different machine. Where we now get nanosecond responses, this will definitely become milliseconds. However, tikv still is very fast - on average one response takes less than 1 ms as well. This only becomes a bit worrisome if we do multiple consecutive gets that require multiple round trips. There are probably ways to work around this. I'm assuming there are ways we can do batch requests in their client.
2024-01 EDIT: This doesn't solve all issues:
I think this approach will not be sufficient. Back to the drawing board.
https://github.com/lnx-search/datacake/tree/main/examples/replicated-kv
Data cake also seems interesting, I'll take a closer look soon
This is a load balancer / proxy server / rust server (actix / axum?) that forwards requests to AtomicServer instances, depending on the used subdomain (drive
). Note that it needs to be aware of which node hosts which drive.
This means:
Atomic-server has been designed to run on low-end hardware, as a self-hosted server. This was why I decided to use an embedded database (sled) and search engine (tantivy).
However, this introduces a problem. What do you do when the physical constraints of a single machine are exceeded by the demands? For example, when a single CPU is not fast enough to host your data, or if the RAM is bottlenecking performance, or if the disk size is insufficient? This is where distributed setups come in handy, but the current architecture is not designed to deal with this.
This thread is for exploring what might need to happen to facilitate a multi-node setup.
Thoughts
Commits
to all nodes.registering
new nodes. They should probably tell the network that they would like to receive some range of resources that they are responsible for.subject-property-value
store and thevalue-property-subject
stores should probably be distributed. Should nodes specialize in one or the other?Interesting tools
monolith
project, might be inspring)