Add support for bulk import method when commissioning hosts

usbenny commented 12 months ago

Code of Conduct

[X] I have read and agree to the project's Code of Conduct.
[ ] Vote on this issue by adding a 👍 reaction to the original issue initial description to help the maintainers prioritize.
[ ] Do not leave "+1" or other comments that do not add relevant information or questions.
[ ] If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Description

The current provider runs all host commissioning tasks in parallel. Currently, SDDC Manager is limited to supporting 2 host commissioning tasks (https://kb.vmware.com/s/article/92494) and 10 tasks of any type

Use Case(s)

Commissioning more than 2 hosts in parallel

Potential Configuration

@dimitarproynov demonstrated on a call the ability to have each host be deployed serially which would solve the issue, however it would take longer to deploy many hosts.

Using the host commission API in SDDC Manager and passing it all hosts that need to be commissioned in one JSON payload would only create one task in SDDC Manager and then it controls the marshalling for commissioning those hosts.

I'm not sure how you would aggregate the hosts in terraform for this to happen though.

References

KB for max parallel tasks: https://kb.vmware.com/s/article/92494

Host Commission API: https://developer.vmware.com/apis/vcf/latest/hosts/#_usecase_hostCommission

raj-chelur commented 12 months ago

Currently, SDDC Manager is limited to supporting 2 host commissioning tasks (https://kb.vmware.com/s/article/92494) and 10 tasks of any type

This KB is describing a situation where a variety of operations are submitted in parallel.

If there are no other active operations running on SDDC Manager, and you want to commission a large number of hosts, that limit is described in configmax:

Concurrent Operations ESXi Host Commission 10 Maximum number of concurrent host commission operations recommended per SDDC manager instance are 10. Maximum number of hosts recommended per commission operation is 40. Note: This limit is supported when this is the only operation type submitted in parallel.

Per SDDC Manager Hosts in a Commission Operation 40 Maximum number of input ESXi hosts for each host commission operation in SDDC manager instance

So, you can commission up to 400 hosts in parallel.

Even in the scenario described in the KB, it is 2 host commission operations that are supported, and each operation can have 40 hosts in that API request.

usbenny commented 12 months ago

Great, while that's a good start you're still limited to 10 at a time then. The way the Terraform VCF Provider currently works, at least from what I saw, was by creating a commission task for EACH host defined in the TF. This was the reason for my suggestion, I've seen plenty of commission operations with over 10 hosts. Simply need a way to batch them up to get to the 400# you reference.

raj-chelur commented 12 months ago

Agree, we need to integrate batching into the terraform provider somehow - @dimitarproynov and @tenthirtyam - thoughts?

I was providing the context for what is possible in SDDC Manager APIs.

github-actions[bot] commented 10 months ago

'Marking this issue as stale due to inactivity. This helps us focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!'

jmac-vmw commented 9 months ago

I also saw this in my testing of the provider. I really do think that we should make use of the bulk import function as a part of my work for PS, as it is a great possibility that we will hit the 10 limit. The way i instantiate my workflow is that i use a Variable map object that has all the host details and then just do a for each to go through the list of hosts. I will have to think through this approach, and do a wait so only 10 can happen at a time if we do not use a bulk function.

If we could add the bulk function, it would be as simple as providing the object map as the reference, so it it sends it all in one request (very much like in the GUI where you specify a json of hosts).

github-actions[bot] commented 5 months ago

'Marking this issue as stale due to inactivity. This helps us focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!'

github-actions[bot] commented 2 months ago

'Marking this issue as stale due to inactivity. This helps us focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!'

spacegospod commented 2 months ago

I did some research on this one. Even now we use the bulk API to commission hosts. The technical challenge here is to get all of the host specs into the same API call. This is difficult because the individual resources don't know about each other. We could figure out some sort of utility that orchestrates this.

A simpler solution would be to define a dedicated resource for managing hosts in bulk

vmware / terraform-provider-vcf