search

creationix / nstore

nStore is a simple, in-process key/value database for node.js
MIT License
392 stars 31 forks source link

multi-node #11

Closed dvv closed 13 years ago

dvv commented 14 years ago

Hi!

Is it possible to open a db file several times, to be able to run several node processes under http://github.com/kriszyp/multi-node?

TIA, --Vladimir

creationix commented 14 years ago

No, nStore only allows a single process accessing a single db file at once. You can have a master process and cause it to make it's services available over network or some other IPC.

creationix commented 14 years ago

Though, this isn't a bad idea, it would just require a change in how the program is architected.

dvv commented 14 years ago

Couldn't you outline the needed changes?

creationix commented 13 years ago

I don't know of a production database in the world that can have multiple processes accessing and modifying the same data file on disk safely without killing performance. sqlite sortof supports this, but it's highly discouraged. I still think the best solution is to have nStore be a single dedicated process and have it talk to the other processes via sockets.

If your reason for wanting this is to have the dedicated nStore core have multiple processes itself, then you would have to implement locks so that two processes can't write at the same time to the file. Also, the other processes would need to know somehow when a new record was added to the file on disk. I guess if there were no in-memory caches, then every read could start with the end of the file and work backwards.

I'm not sure the overhead of all this syncing and IPC would outweigh the benefits of having more CPU power available.

Better would be to shard your data into separate nStore processes and have those all available via sockets to the multiple processes needing the data.

The largest bottleneck (other than drive seek time) is serialization and deserialization of data. The more processes you have, the more data to serialize and pass around. and the more CPU you need.

dvv commented 13 years ago

I see. I came to the same conclusions. Sharding...

Thanks for extended answer.