search

fastmonkeys / stellar

Fast database snapshot and restore tool for development
MIT License
3.86k stars 119 forks source link

How does stellar work? #13

Closed lukecyca closed 10 years ago

lukecyca commented 10 years ago

This looks great!

My question is how stellar interacts with the database and where it stores the snapshots. Does it do so by creating multiple copies of the database on the RDBMS? Or does it dump the data out as SQL and manage snapshots locally on the filesystem? Are the snapshots stored in full or is there an efficient scheme to store just the diffs?

Apologies if this is documented somewhere. I didn't see it in the README and haven't dug through the code to find out. It would be great to have a section of the README explaining how it works.

asenchi commented 10 years ago

Might also want to add the limitations of the method used to copy PostgreSQL databases, from the documentation:

Although it is possible to copy a database other than template1 by specifying its name as the template, this is not (yet) intended as a general-purpose "COPY DATABASE" facility. The principal limitation is that no other sessions can be connected to the template database while it is being copied. CREATE DATABASE will fail if any other connection exists when it starts; otherwise, new connections to the template database are locked out until CREATE DATABASE completes. See Section 21.3 for more information.

Teemu commented 10 years ago

Hey Luke, good question (and probably should have been explained in README)!

Stellar stores two copies of the database (for instant restore and for storing a copy) in the RDBMS. I decoded against SQL dumping because that approach was too slow to begin with as the goal was to enable quick snapshot restoring. There is no efficient schema to store the diffs so you probably don't want to make too many snapshots or you will run out of storage space.

It might be possible in the future to be smarter about the duplicates but at the moment I'm prioritizing speed over storage use.

Teemu commented 10 years ago

The limitations of the method haven't really been an issue when using Stellar in development. I suspect it might be a problem in production use but I hope no-one is using this with production databases (I should add a warning for that as soon as I get back to computer).

Teemu commented 10 years ago

The explanation has been added to README so I consider this issue to be closed.

volkanunsal commented 10 years ago

Just a suggestion. Might be a good idea to make the warning more prominent and higher on the page since data loss is kind of a big deal. Right now, it's hardly noticeable under "How it works" section.

Teemu commented 10 years ago

I'm planning to add another warning to configuration initialization wizard just in case somebody doesn't read the bolded warning.