Allow for user to create a node with attached disks

[kannon92]

User Story

As an operator, I would like to create a node with a root partition and attached disks.

The API should be optional and allow for an array of disks in case a user wants to add more than 1 disk to the node.

Detailed Description

AWS, GCP and Azure CAPI providers allow one to create a node with additional disks. It would be great for this provider to also allow this functionality.

Anything else you would like to add:

[Miscellaneous information that will assist in solving the issue.]

/kind feature

[metal3-io-bot]

This issue is currently awaiting triage. If Metal3.io contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance. The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.

[dtantsur]

AWS, GCP and Azure CAPI providers allow one to create a node with additional disks.

It's infinitely easier for a cloud VM-based provider to add/remove disks than it is for baremetal. Which mechanism do you have in mind when you say "attach"? Why cannot it be solved by a first boot script (cloud-init, etc)?

[dtantsur]

/triage needs-information

[kannon92]

The main use cases we have are when a user booted a node and then they realize that they want to add a disk rather than recreate the node.

A few usecases I have in mind would be supporting https://kubernetes.io/blog/2024/01/23/kubernetes-separate-image-filesystem/.

Generally we find users realize that they are running out of space on their cluster. So they would resort to adding a new disk to a node to help alleviate the situation.

Another one that we would be interested to support is https://github.com/kubernetes-sigs/cluster-api/blob/main/docs/proposals/20200423-etcd-data-disk.md.

Adding etcd to a separate disk.

This may be a better one to do on a creation of the node but we are also exploring adding swap to a node where swap memory is paged out to a designated disk so we can avoid I/O contention on the main filesystem.

[jayofdoom]

@kannon92 is an old co-worker of mine and asked me to take a look. My concerns line up mostly with Dmitry, we need to figure out "attach" means to you in context of a physical machine -- what should be the other side of that "attach" API. Seems to me in order for "attach" to make sense, you kinda need a source (what do I attach?) and a destination (where do I attach it?).

If we're talking physical disks; those cannot generally be dynamically added/removed from a machine via an API. If we're talking network-attached disks, you'd likely need some coordination with metal3/ironic around when the node reboots (Ironic handles this via our existing StorageInterface; which AFAICT doesn't have native support for any kubernetes-flavored way of storage management -- if such a thing exists).

[dtantsur]

If we're talking physical disks; those cannot generally be dynamically added/removed from a machine via an API.

Yeah, except for very exotic composable hardware (and even it will probably require a reboot). I'm still not sure what can be done from the hardware perspective.