search

hashicorp / nomad

Nomad is an easy-to-use, flexible, and performant workload orchestrator that can deploy a mix of microservice, batch, containerized, and non-containerized applications. Nomad is easy to operate and scale and has native Consul and Vault integrations.
https://www.nomadproject.io/
Other
14.95k stars 1.96k forks source link

Allow updating Volume topology #23291

Open lukaszo opened 5 months ago

lukaszo commented 5 months ago

Proposal

Allow to update Volume CSI topology.

Use-cases

I'm writing CSI plugin for LVM. I took https://github.com/metal-stack/csi-driver-lvm and replaced Kubernets with Nomad and it works. Using LVM is not a typical use for CSI as it's not a network file system. Volume is available on a specific node only, and I'm setting a proper topology so nomad schedules allocation on a correct node. This works as expected.

The issue is when for some reason I have to move volume to another node. Why would I do that? Because of maintenance of the current node or simply when I want to replace the node.

The process would be like this:

  1. Stop the job
  2. Migrate the data
  3. Update Volume topology
  4. Start the job and allow nomad to schedule allocation on the new node.

This is not possible though because updating topology is not allowed. I'm not sure why, though.

Attempted Solutions

My current solution is:

  1. Stop the job
  2. Create a new volume
  3. Migrate data.
  4. Update Job with new volume
  5. Delete old volume.

This is not a perfect as there is a risk of doing it in a wrong order and deleting a wrong volume :)

What are your thoughts on allowing updating topology?

jrasell commented 5 months ago

Hi @lukaszo and thanks for raising this issue. This is something that would be possible to add, but wouldn't be exactly trivial given then current implementation. I'll add it to our backlog for further discussion. Below are a couple of quotes from the CSI spec for future reference to readers.

Recreating a volume:

This operation MUST be idempotent. If a volume corresponding to the specified volume name already exists, is accessible from accessibility_requirements, and is compatible with the specified capacity_range, volume_capabilities, parameters and mutable_parameters in the CreateVolumeRequest, the Plugin MUST reply 0 OK with the corresponding CreateVolumeResponse.

"already exists" error:

Indicates that a volume corresponding to the specified volume name already exists but is incompatible with the specified capacity_range, volume_capabilities, parameters, accessibility_requirements or volume_content_source.

lukaszo commented 5 months ago

Hi @jrasell, if I can contribute in any way, please let me know.