Sync convoy volumes between hosts?

[pwFoo]

Move also the convoy volume during container migration to another host? Sync convoy volumes between (RancherOS) hosts to be more fail-save?

Maybe convoy could work together with flocker plugin? But convoy should keep the simple usage (set driver and use "-v" option), which should be .

[yasker]

Hi @pwFoo ,

For sync purpose, you can use NFS mount to the same NFS server.

For migrate, you can use backup/restore feature.

Could you be more specific about "container migration"? I know Flocker supports migration, but haven't heard Docker supports it officically.

[pwFoo]

Would be great if the convoy volume could move to another host if the app container is moved. But maybe it isn't needed. I'll do some more tests also with data only container with Rancher service.

[jaroslavl1]

Support for moving volume to another host is quite critical feature. I cannot use convoy due to this.

Using backup/restore functionality of convoy is not a good idea for this, as it has different semantics. Your service may start on the same or different host than last time. It may be quite difficult to make correct choice - the volume may be available locally, but if it isn't then you would think its correct to use a backup. But what if backup wasn't created because there was a host crash and previous backup is too old? In such case you want to have the latest data which is on the host that crashed/was restarted. It may still be available. On amazon its more apparent - you restart a host and want to start the service on another host. But the new host is in different AZ so you can't reattach the volume. First a snapshot needs to be created from the volume, then volume created from it in new AZ and the snapshot deleted. In this case you do not want to use some old backup(snapshot). This needs to be automated and happening behind the scenes.

I guess the device mapper is only for local dev environment (vagrant etc) since all volumes end up on the same block device, making it difficult to move without moving the rest. While the amazon backend supports creation of EBS volumes for docker volumes so they can be managed per container. NFS is easiest since it will be available on all nodes and no migration is needed. But at the cost of performance so its not suitable for services that need high performance storage.

So the solution of this issue is further complicated in convoy by supporting multiple backends that work differently.

[jaroslavl1]

Also note that even if you run machines in the same region and availability zone, convoy on machine2 cannot see volumes created on machine1, since metadata is kept per machine. Such volumes could be easily mounted if metadata was global. To migrate a volume across availability zones you can use volume snapshots on amazon but again convoy on machine2 will not see snapshots created from machine1. You also cannot create volume from a snapshot so snapshots are not really useful except for backup.

The conclusion is convoy at the moment cannot be used in amazon since there is no real support for volume migration except through the workaround of backup feature which is unnecessary performance hit and complication.

More work is needed to make convoy easily usable on cluster.

[yasker]

Hi @jaroslavl1

Thanks for comments!

1. Current the design of Convoy is limited to one host, and we're working on higher level(e.g. cluster) management using Rancher.
2. On EBS, you can reuse the volume created by other machine by specifying the AWS volume ID when creating the volume, as long as they're not "in use" and in the same availability zone. And you're right that even with specifying AWS volume ID you cannot use the volume in the different availability zone, due to the AWS limitation
3. I guess you already know user can create volume from AWS snapshot, as long as they're in the same region. We don't support different region because copying from different region is very time consuming. We don't want to block the volume creation process for hours.
4. I don't quite understand why backup feature is a performance hit? It's basically just AWS snapshot.

We're working on cluster level support. Currently we're integrating Convoy with Rancher, which would deploy Convoy and setup it on demand. And it would support move volume around, and as the first step we would support NFS for this.

[pwFoo]

We're working on cluster level support. Currently we're integrating Convoy with Rancher, which would deploy Convoy and setup it on demand. And it would support move volume around, and as the first step we would support NFS for this.

Great news, @yasker. Thanks. Can't wait to test it with RancherOS :)

[lavvy]

Wow so great. Any timeline please ? On Oct 14, 2015 1:42 PM, "pwFoo" notifications@github.com wrote:

We're working on cluster level support. Currently we're integrating Convoy with Rancher, which would deploy Convoy and setup it on demand. And it would support move volume around, and as the first step we would support NFS for this.

Great news, @yasker https://github.com/yasker. Thanks. Can't wait to test it with RancherOS :)

— Reply to this email directly or view it on GitHub https://github.com/rancher/convoy/issues/31#issuecomment-148131204.

[yasker]

No exact timeline so far, but probably sooner than you think. :smile:

[pwFoo]

At the moment I use RancherOS / Rancher for testing only and will start use it in production with a stable release. Convoy could be a better alternative to data only containers which not work as needed with services / sidekicks at the moment. Snapshot, backup, restore and move to another host would be great features for persistent volumes.

Convoy with the listed features would be my favorite compared to other volume plugins! It would be perfect if optional ceph backend (#15) and maybe volume quotas (#27) could also be added. ;) Volume quota could also be implemented by ceph / device mapper / lvm volume size, but should be automated during size option during volume creation.

[lavvy]

Wow, gonna be really cool. Nfs too, wondering if it's going to be a form of unlimited volume like swift and gluster On Oct 14, 2015 3:37 PM, "pwFoo" notifications@github.com wrote:

At the moment I use RancherOS / Rancher for testing only and will start use it in production with a stable release. Convoy could be a better alternative to data only containers which not work as needed with services / sidekicks at the moment. Snapshot, backup, restore and move to another host would be great features for persistent volumes.

Convoy with the listed features would be my favorite compared to other volume plugins! It would be perfect if optional ceph backend (#15 https://github.com/rancher/convoy/issues/15) and maybe volume quotas (

27 https://github.com/rancher/convoy/issues/27) could also be added.

;) Volume quota could also be implemented by ceph / device mapper / lvm volume size, but should be automated during size option during volume creation.

— Reply to this email directly or view it on GitHub https://github.com/rancher/convoy/issues/31#issuecomment-148171031.

[jaroslavl1]

@yasker

The data volume may be big, tens to hundreds of gigabytes. If new service instance is started in the same availability zone, its possible to just mount the volume on new machine. Trying to archive and then create volume from the archive could mean significant delay. Moving service to other AZ should be avoided if possible due to the problem of having to create snapshot first. Since attaching volume to new machine can be done in two ways (simple and the complicated one with backup) this is something that should be solved by Convoy.

There are many people out there who don't use RancherOS but for example CoreOS. So you may want to minimize external dependencies to make convoy still usable for other OSes.

What I liked about Convoy is that it was simple to setup, worked well, documented, didn't try to do too much like Flocker.

[drewrobb]

Is there any timeline for this, and is the cluster support integration going to be rancher specific?

Thanks!

[flypenguin]

I thought using EBS would move the volume attachment along with a moving container between hosts, at least in the same av. zone. Is this not the case? I really need a storage solution for moving containers right now, and I thought this was it. Can someone comment please?

[drewrobb]

@flypenguin we are using rexray with EBS volumes for this, and it is working for us.

[flypenguin]

@drewrobb could you tell me a bit more? don't even know rexray :)