Support Storage vMotion while vDVS volumes are attached

[madjam002]

I'm aware that currently Storage vMotion isn't supported by this plugin and results in strange behaviour, but it would be nice if this could be supported somehow?

I'm not sure how it would work with volumes created on specific datastores (e.g MyVolume@datastore1 - datastore1 would be available on one host but not another) but for those with no datastore constraint (e.g just MyVolume) this currently resolves to the same datastore where the current VM resides, so it could theoretically be migrated to the same target datastore for the VM during Storage vMotion?

At the moment my workflow for executing a Storage vMotion is to stop all running containers on the host so that the vDVS disks are detached from the VM, run Storage vMotion on the VM, manually move the VMDKs for the vDVS disks from within the vSphere Client and then start the containers again, but it would be nice if the containers could stay running somehow.

Thanks for the great work on this project :smile:

[govint]

@madjam002, thanks for letting us know of your requirement, I'm curious why the VDVS volumes would need to be migrated. Storage vmotion applies to VMs where the VM's VMDKs (dependent disks) are migrated to a target datastore. The VDVS volumes are completely separate from VM disks and are managed wholly by the plugin and via the admin CLI on the ESX host. The datastores where the VDVS volumes get placed is also related but not limited to the datastore where the VM is located (although that's the default location).

I'd like to know the use case that requires migrating the VDVS volumes to a different location. The VDVS volumes are all tracked in a config database (if initialized) and that data would go stale if the volumes are migrated using the vSphere client. Possibly a config database isn't initialized in your setup and hence the issues isn't observed.

[madjam002]

@govint When I attempted a Storage vMotion earlier I noticed that the vDVS VMDKs were migrated but to the VM folder on the target datastore, not the dockvols directory.

My use case is being able to vMotion VMs off a host when it needs to be taken down for maintenance and currently it doesn't seem to be possible to migrate VMs with running Docker Containers with 100% uptime to another host.

Maybe there's another way to do this that I'm missing?

[govint]

@madjam002, presently, VM migration scenarios aren't supported with VDVS disks attached. One option is to stop the workload and then migrate the VM. Assuming a docker swarm, another option is to drain services (containers) deployed to the VM first (via docker node command), the services get redeployed, then migrate the VM and finally re-enable the VM (make node active in docker swarm) to redeploy services.

Having shared storage as the location for the VDVS volumes is definitely preferred, as local storage would be very limiting on what container deployments can be reasonably implemented, especially if you are running a swarm with VMs on multiple ESX hosts.

Possibly, could try the vFile plugin (which layers on top of the vDVS plugin) to be able to serve volumes across VMs/hosts, irrespective of the datastore location of the volumes themselves (yes?). Which should preclude the need to drain/activate services on the VM during a migrate.

@luomiao, could you update if the vFile plugin can be used with VMs on multiple hosts and the docker volumes located on a specific host local datastore.

[govint]

Closing this as presently we don't support migration scenarios for container volumes, plus, should also rethink if migration really makes sense with container workloads. The VM is very much part of the infrastructure and the workload (container) migrates across VMs. Balancing of volumes across (not supported yet) is something that can be addressed as also being able to migrate container volumes over datastores. Migration of VMs really doesn't apply, if anything

1. Simply drain the node (VM) of container workloads.
2. Perform maintenance of the VM (migrate).
3. Resume scheduling workloads to the node (VM).

[madjam002]

@govint Can this not stay open? One of the benefits of vMotion is that you can reschedule a VM without downtime, and currently there isn't a way to migrate a VM with associated containers without downtime? The solution you proposed involves stopping the container on the VM.

[govint]

@madjam002, this is an enhancement to how migration of VMs handle virtual disks and I already had an internal issue to address that on the ESX platform. But that will be addressed in subsequent releases of the ESX software. This issue need not be kept open as such. That said, draining containers off a VM only implies that those will now be scheduled on other VMs on the ESX hosts.or the same ESX host as the case may be. The containers shouldn't have to be stopped.

[madjam002]

@govint Ah that's great so a future version of ESX will migrate attached virtual disks to the right location when doing vMotion?

And by draining containers off a VM still means that the container will be stopped and started on a different VM. With some container workloads which require persistent storage will result in downtime when doing this.

[govint]

@madjam002, yes, thats the current idea to be able to address the needs of container workloads during migration.

[govint]

FWIW, opened Moby issue 35715 to allow an app to shutdown gracefully on container stop/pause so the app gets a chance to bring its data to a sane state before exiting or pausing. Can even allow the app to implement a hand-off to another instance if needed. Helps when instances are drained off a docker host for example.