search

bassu / bzman

Bassu's ZFS (replication) Manager for SmartOS / Unix / Linux
GNU Lesser General Public License v3.0
38 stars 7 forks source link

Send to different target pool than the original #2

Open pongraczi opened 10 years ago

pongraczi commented 10 years ago

Hi,

Thank you for your great tool for ZFS! I use my own solution for nearly the same reason, but my target is different, because I use it for backup purposes:

Instead of creating two identical pools, it would be a nice option to send to under an existing pool for backup purposes, like in this example:

Source pool: 'tank' on host activeserver Backup pool: 'ztank/backup/activeserver/tank'

So, the backup should replicate the source pool into the backup pool: tank -> ztank/backup/activeserver/tank tank/private/101 -> ztank/backup/activeserver/tank/private/101

It should be an option, called target_prefix:

bzman -s tank darwin --target_prefix ztank/backup/activeserver

So, with this option you will be able to run one big fat server to store as many live server's replication as you want.

What do you think? Of course, backup and replication are more or less equivalent, but we assume replication means, if the master server fails, the standby server can start on the replicated filesystem, while the backup only stores the replicated filesystems. Thanks.

hellweiss commented 10 years ago

Perhaps you can try this: http://www.bolthole.com/solaris/zrep/

bassu commented 10 years ago

@pongraczi Hi. That seems a good and viable option. Since this version of bzman implements rudimentary pool replication requirement (that I had at the time) so I didn't think of the advanced features. However, I am considering implementing pool selection in the version currently I am running (which already includes major enhancements like Unix support, separate dataset replication, improved error handling in initial replication and stuff like that), if there's a need for the said feature, which I guess there will be at some time soon! I am still testing that in production and will have to release it after it gets golden. May be call it 0.5.

@hellweiss I remember looking at zrep quite some time ago but it is a mere shell script and lacks reporting and automation features.

hellweiss commented 10 years ago

@bassu I mentioned it, just as another option.

bassu commented 10 years ago

There is now an option to select individual datasets for sending replication streams. Does that satisfy the need? If you are willing to do some testing, I'll create a new branch with the requested feature.

treydock commented 10 years ago

+1 for being able to send to custom remote pool. Sending client-host:/tank to backup-host:/tank/backups/client-host/tank would be of great use.

treydock commented 10 years ago

Looking at my existing zfs send / receive commands or using an alternative receiving pool, it would be a huge benefit to allow the existing options '-s' and '-d' to behave the same but if some delimiter plus path is appended to the host, the receive command uses the '-d' flag and the provided path.

# bzman -d tank/myzvol 172.16.1.5:/tank/client/tank

This would then shift the command to be the following:

zfs send -R -I tank/myzvol@<N> tank/myzvol@<X> | ssh <ARGS> 172.16.1.5 zfs receive -F -d tank/client/tank

That would create on the receiving side tank/client/tank/myzvol with associated snapshots.

bassu commented 10 years ago

@treydock One of the reasons why I did not think of sending to a different dataset, is how the snapshot diffs are calculated here https://github.com/bassu/bzman/blob/master/bzman#L506-L518. Which of course is through their names and human readable time stamps. Since, sending to a different dataset usually means that the snapshot names will change so the algorithm will need to be rewritten with some other sort of identifier in the snapshot names.

On top of that, this may also require current users to redo the initial rep. So I am thinking we should roll out a new major beta version like 1.0b or something??

With that being said, I have created a new experimental branch to test this out and see how it goes.

About the host:dataset format for -s/-d options, I think it is cool and because it is makes better sense I have implemented in the new branch, already.

job commented 8 years ago

Has any progress been made on this feature? :)

bassu commented 8 years ago

@job I partially implemented the feature last year but since nobody replied, I didn't carry on. This year, I have a plan to port it to a compiled programming language (e.g. Go or C++) and in the wake of recent high-availability discussion on SmartOS mailing list, I want to add auto-failover option, plus improvements like SQLite / Postgresql backend and a web interface, preferably in a new project.