Tutorial

[tom-adsfund]

This looks like a really interesting project, but I'd like to see a tutorial that covers transitioning through from 1 node to several, along with a transaction across actors. My use case would be having at least two actors that have (user, balance), and the ability to transfer amounts between users on different actors.

I'd like to know whether this can be done without downtime.

I think a tutorial like this would make things much clearer to me.

[SergejJurecko]

Yes adding nodes/clusters can be done without downtime.

Nodes are divided into clusters. A cluster is a replication unit. If you start out with a single node, then you most likely should add 2 nodes to the existing group. If you wish to expand further, create a new group.

To add a node to an existing group named grp1 (actordb on that node should be running)

use config insert into nodes values ('mynode@192.168.2.1','grp1'); commit

To create a new cluster:

use config insert into groups values ('grp3','cluster'); insert into nodes values ('mynode1@192.168.3.1','grp3'); insert into nodes values ('mynode2@192.168.3.2','grp3'); insert into nodes values ('mynode3@192.168.3.3','grp3'); commit

This is documented at: http://www.actordb.com/docs-configuration.html

[tom-adsfund]

Very interesting!

When you add an actor, is it allocated to a cluster, and then stays within that cluster? Can we remove nodes online using SQL to remove them from the 'nodes' table? (So maybe remove bad hardware, or scale down hardware.) I guess that we can't remove clusters/groups once they are made?

[SergejJurecko]

Yes the actor stays in the cluster. Unless you add clusters, then it might get moved to a new one.

You do not need to remove nodes when changing hardware. Steps to change nodes:

1. Create a copy of the lmdb file from an existing node using actordb_tool (backup option).
2. Turn off one of the nodes
3. Setup new hardware or upgrade
4. Configure actordb on it normally, copy lmdb file to /var/actordb/lmdb (or wherever you have it configured)
5. Start actordb on that node
6. Connect actordb_console to an active node and run use config UPDATE nodes SET name='newname@newip' WHERE name='oldname@oldip'

Step 6 is not required if you use the same name and ip as the old hardware.

We support deleting nodes from clusters, but not deleting clusters (yet).

[tom-adsfund]

Sounds good.

When you say 'might get moved', is that automatic?

[SergejJurecko]

Yes it is automatic. Every actor belongs to a shard. When adding new clusters, existing shards get split in half. A new node will take the top half of the shard from an existing node in another cluster. So if the actor is in the top half of such a shard, it will get moved.

[tom-adsfund]

I guess that during that update, the actors will be locked (preventing writes until the shards have been transferred), and if there is a failure it will recover?

(Sorry for all the questions! This is a very clever project.)

[SergejJurecko]

No problem ask away.

Actors are moved relatively slowly. One at a time generally. Actor data is split into two parts. The main block (basically an sqlite file inside lmdb), and the write-ahead-log. Main block can be copied over while executing reads and writes. Once it gets to the WAL, the actor is locked. If there were a lot of writes since start of copy WAL will be larger and it will take longer.

[tom-adsfund]

That's a smart way to minimize locking time. I guess that if there's a failure during the lock, the WAL will still be on at least one side, and so it will be easy to recover? Is the movement restarted automatically if a failed node comes back online?

[SergejJurecko]

Yes copy is restarted if it fails at any time (during lock or no).

[tom-adsfund]

Great.

What would be the SQL for a transaction transferring an amount from a balance in one actor to a balance in another actor?

[SergejJurecko]

If balance actor has a table that stores the value and a transaction log. It would look something like this:

actor balance(b1); update balancetable set b=b-Amount where id=X; insert into log (id,user,amount) values (TransID,u1,X); actor user(u1); update usertable set b=b+Amount where id=X;

A transaction log table is required somewhere. It can be in balance, user or a specific log actor (depending on your use case). Mostly because a transaction may succeed, but a client may loose connectivity before getting a successful response.

[tom-adsfund]

Excellent. Thanks for all your time. I'll come back if there's anything else. Great work on what seems like a very solid distributed system.

[federico-razzoli]

Hello. I have questions related to the ones that I read here, and I can't find a mailing list, so I'll write them here.

1) What are minimum and maximum sizes for a cluster? 2) If a 1-node cluster is allowed, can I disconnect it from the others, add data to a single actor (no multi-actor transactions) and then reconnect it? 3) What filesystem(s) do you recommend for nodes?

Thanks in advance.

[SergejJurecko]

1) 1 - 5 2) Theoretically yes it should work. 3) If performance is of critical importance, ext2 will actually perform best. Because it does not do journaling. Otherwise I would stick to ext3/ext4.

[federico-razzoli]

Thank you for your answers. The project is definitely interesting.

I have another doubt. I got this watning on startup: !!!! WARNING: ulimit -n is 1024; 32768 is the recommended minimum.

It is not a problem itself. But while reading documentation, I didn't realize that ActorDB opens so many files. Perhaps the documentation should explain what files are used, and how they are used.

Or am I missing something?

[SergejJurecko]

System default is ridiculously low. ActorDB used to use a lot more files, it no longer does so because everything is in the lmdb file. But every client connection is also a file descriptor and counts as an open file. We kept the warning because one should increase that limit. Unless you are running a small setup that will only use a few connections. In that case ignore the warning.

[federico-razzoli]

I agree that the default is too low. I asked just because I never seen that warning with other databases.

Thanks for the clarification.