Deploying OVF from content library results in HTTP error 500

sofixa commented 3 years ago

Terraform Version

0.14.6

vSphere Provider Version

1.24.3

Affected Resource(s)

vsphere_virtual_machine

Terraform Configuration Files

resource "vsphere_virtual_machine" "vm" {
  name             = var.name
  num_cpus         = var.cpu
  memory           = var.memory * 1024
  resource_pool_id = data.vsphere_compute_cluster.compute_cluster.resource_pool_id
  datastore_id     = data.vsphere_datastore.datastore[0].id
  host_system_id   = var.host_system_id
  folder           = var.folder
  annotation       = var.notes

  dynamic "disk" {
    for_each = var.disk_size
    content {
      label            = "${var.name}-${disk.key}.vmdk"
      size             = disk.value
      eagerly_scrub    = var.eagerly_scrub
      thin_provisioned = var.thin_provisioned
      unit_number      = disk.key
    }
  }

  clone {
    template_uuid = data.vsphere_content_library_item.image.id
  }

  lifecycle {
    ignore_changes = [clone[0].template_uuid, disk[0].key]
  }

  cpu_hot_add_enabled         = true
  memory_hot_add_enabled      = true
  wait_for_guest_net_routable = false

  dynamic "network_interface" {
    for_each = data.vsphere_network.network
    content {
      network_id   = network_interface.value.id
      adapter_type = var.network_interface_type
    }
  }
  custom_attributes = {
    (data.vsphere_custom_attribute.project_id.id) = (var.project_id)
  }

  extra_config = {
    "guestinfo.vmname" = var.name
  }
}

Debug Output

https://gist.github.com/sofixa/4cd7740a8628d5a43ec4db5fb75a5594 Note: this is with a custom build of the provider with extra logs and most notably the deployment spec of the template.

Expected Behavior

The content library OVF template should have been successfully deployed since there's nothing weird or special about it. The exact same template, with almost the same options gets deployed just fine with PowerCLI or govc, so it's something in terraform ( the differences are in the number of network interfaces and disks since we can't tell PowerCLI or govc to make multiple of them). The gist contains the specs generated by govc and terraform.

Actual Behavior

Error: POST https://sv-test.ad/rest/com/vmware/vcenter/ovf/library-item/id:d18e2fba-16cb-4d83-9ca7-85b360f07d16?~action=deploy: 500 Server Error

As per usual, a completely useless error from VMware. After going through the logs, there doesn't seem to be anything suspicious.

Steps to Reproduce

Try deploying a content library OVF template??? Not really sure what's causing the issue

How can we proceed about debugging this further? I found this issue in govmomi which seemed like a potential cause, but it was fixed in ~0.23, and when i rebuilt the provider with govmomi v0.24.0 the problem is still there, and govc 0.21 doesn't have any issues, so it's not that.

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

sofixa commented 3 years ago

So, after some more digging, we've narrowed down the cause for this problem.

https://github.com/hashicorp/terraform-provider-vsphere/blob/9d5570c76b48899b64601c52ad3dda5bb8c552d5/vsphere/internal/helper/contentlibrary/content_library_helper.go#L553-L563

ovf_mapping is used for the key of the storage_mapping struct ( spec here ). If it isn't set ( why would it be, the terraform docs mention it once and it says (optional), terraform sends an empty key, like so:

 NetworkMappings: ([]vcenter.NetworkMapping) (len=2 cap=2) {
    (vcenter.NetworkMapping) {
     Key: (string) "",
     Value: (string) (len=15) "dvportgroup-128"
    },

which the vCenter doesn't appreciate, with the resulting vapi log:

com.vmware.vapi.bindings.convert.ConverterException: Could not convert field 'network_mappings' of structure 'com.vmware.vcenter.ovf.library_item.resource_pool_deployment_spec'
    at com.vmware.vapi.internal.bindings.convert.impl.JavaClassStructConverter.fromValue(JavaClassStructConverter.java:89)
    at com.vmware.vapi.internal.bindings.convert.impl.JavaClassStructConverter.fromValue(JavaClassStructConverter.java:33)
    at com.vmware.vapi.internal.bindings.TypeConverterImpl$ValueToJavaVisitor.visit(TypeConverterImpl.java:281)
    at com.vmware.vapi.bindings.type.StructType.accept(StructType.java:372)
    at com.vmware.vapi.internal.bindings.TypeConverterImpl$ValueToJavaVisitor.visit(TypeConverterImpl.java:305)
    at com.vmware.vapi.bindings.type.TypeReference.accept(TypeReference.java:20)
    at com.vmware.vapi.internal.bindings.TypeConverterImpl.convertToJava(TypeConverterImpl.java:661)
    at com.vmware.vapi.internal.bindings.StructValueExtractor.valueForField(StructValueExtractor.java:46)
    at com.vmware.vcenter.ovf.LibraryItemApiInterface$DeployApiMethod.doInvoke(LibraryItemApiInterface.java:44)
    at com.vmware.vapi.internal.bindings.ApiMethodSkeleton.invoke(ApiMethodSkeleton.java:175)
    at com.vmware.vapi.provider.ApiMethodBasedApiInterface.invoke(ApiMethodBasedApiInterface.java:82)
    at com.vmware.vapi.provider.local.LocalProvider.invokeMethodInt(LocalProvider.java:471)
    at com.vmware.vapi.provider.local.LocalProvider.invoke(LocalProvider.java:290)
    at com.vmware.vapi.admin.interposer.impl.Invoker.execute(Invoker.java:46)
    at com.vmware.vapi.admin.interposer.impl.PreInterposerHandler.execute(PreInterposerHandler.java:57)
    at com.vmware.vapi.admin.interposer.impl.VetoInterposerHandler.execute(VetoInterposerHandler.java:51)
    at com.vmware.vapi.admin.impl.InterposerImpl.invoke(InterposerImpl.java:277)
    at com.vmware.vdcs.activation.ActivationFilter.invoke(ActivationFilter.java:123)
    at com.vmware.vapi.core.DecoratorApiProvider.invoke(DecoratorApiProvider.java:37)
    at com.vmware.vsphere.common.impl.SecurityContextInterceptorProvider.invoke(SecurityContextInterceptorProvider.java:72)
    at com.vmware.vapi.cis.authz.impl.AuthorizationFilter.invoke(AuthorizationFilter.java:219)
    at com.vmware.vapi.provider.introspection.ErrorAugmentingFilter.invoke(ErrorAugmentingFilter.java:74)
    at com.vmware.vapi.security.AuthenticationFilter$1.setResult(AuthenticationFilter.java:180)
    at com.vmware.vapi.security.AuthenticationFilter$1.setResult(AuthenticationFilter.java:166)
    at com.vmware.vsphere.common.sessions.impl.SessionAuthnHandlerImpl.authenticate(SessionAuthnHandlerImpl.java:42)
    at com.vmware.vapi.security.AuthenticationFilter.invoke(AuthenticationFilter.java:165)
    at com.vmware.vapi.core.DecoratorApiProvider.invoke(DecoratorApiProvider.java:37)
    at com.vmware.vsphere.vcde.diagnostics.interceptor.ApiDiagnosticsInterceptorProvider.invoke(ApiDiagnosticsInterceptorProvider.java:50)
    at com.vmware.vapi.protocol.server.msg.json.JsonServerConnection.processApiRequest(JsonServerConnection.java:275)
    at com.vmware.vapi.protocol.server.msg.json.JsonServerConnection.requestReceived(JsonServerConnection.java:200)
    at com.vmware.vapi.protocol.server.rpc.http.impl.HttpStreamingServlet.doPostImpl(HttpStreamingServlet.java:124)
    at com.vmware.vapi.protocol.server.rpc.http.impl.HttpStreamingServlet.doPost(HttpStreamingServlet.java:92)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:543)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:615)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:818)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1627)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)
Caused by: com.vmware.vapi.bindings.convert.ConverterException: Element already present in the map.
    at com.vmware.vapi.internal.bindings.convert.impl.JavaUtilMapMapConverter.fromValue(JavaUtilMapMapConverter.java:62)
    at com.vmware.vapi.internal.bindings.convert.impl.JavaUtilMapMapConverter.fromValue(JavaUtilMapMapConverter.java:30)
    at com.vmware.vapi.internal.bindings.TypeConverterImpl$ValueToJavaVisitor.visit(TypeConverterImpl.java:264)
    at com.vmware.vapi.bindings.type.MapType.accept(MapType.java:42)
    at com.vmware.vapi.internal.bindings.TypeConverterImpl.convertToJava(TypeConverterImpl.java:661)
    at com.vmware.vapi.internal.bindings.convert.impl.NullableReferenceOptionalConverter.fromValue(NullableReferenceOptionalConverter.java:33)
    at com.vmware.vapi.internal.bindings.convert.impl.NullableReferenceOptionalConverter.fromValue(NullableReferenceOptionalConverter.java:23)
    at com.vmware.vapi.internal.bindings.TypeConverterImpl$ValueToJavaVisitor.visit(TypeConverterImpl.java:239)
    at com.vmware.vapi.bindings.type.OptionalType.accept(OptionalType.java:24)
    at com.vmware.vapi.internal.bindings.TypeConverterImpl.convertToJava(TypeConverterImpl.java:661)
    at com.vmware.vapi.internal.bindings.convert.impl.JavaClassStructConverter.fromValue(JavaClassStructConverter.java:86)
    ... 51 more

Once we know that, it's easy, just set ovf_mapping on the network_interface block:

network_interface {
   network_id     = data.vsphere_network.network_bkp.id
   adapter_type   = "vmxnet3"
   ovf_mapping    = "nic0"
  }
  network_interface {
   network_id     = data.vsphere_network.network_adm.id
   adapter_type   = "vmxnet3"
   ovf_mapping    = "nic1"
  }

IMHO, at the very least, that needs to be documented much better. I'd even go as far as setting a default value if ovf_mapping is empty.

It's even worse with storage:

https://github.com/hashicorp/terraform-provider-vsphere/blob/9d5570c76b48899b64601c52ad3dda5bb8c552d5/vsphere/internal/helper/contentlibrary/content_library_helper.go#L534-L551

ovf_mapping is used for the key, but there is no such thing :

https://github.com/hashicorp/terraform-provider-vsphere/blob/9d5570c76b48899b64601c52ad3dda5bb8c552d5/vsphere/internal/virtualdevice/virtual_machine_disk_subresource.go#L77-L235

How can we proceed? I'm open to writing a PR with the documentation change and sane defaults for network ovf_mapping, but i have no idea about storage mapping.

slarimore02 commented 3 years ago

I'm running into this same issue. I'm trying to deploy an OVF from a content library and when I have 1 NIC specified it deploys normally. When I add additional NICs it fails with the same 500 error response from vSphere.

clementbernet commented 3 years ago

Did you specify ovf_mapping for each NIC? By using ovf_mapping = var.ovf_map[network_interface.key] in the dynamic network_interface mentioned above we are not running the same problem anymore.

We use it with the variable below:

variable "ovf_map" {
  type    = list(string)
  default = ["eth0", "eth1", "eth2", "eth3"]
}

It should work even if you're not using a dynamic for your NIC by just specifying eth0, eth1 anyway.

tenthirtyam commented 2 years ago

Per the VMware vSphere product documentation, which compares the use of VM Templates vs OVF Templates in Content Library:

"During the deployment of an OVF template, only guest OS customization is supported. Hardware customization is not supported."

Reference: The VM Template as a Content Library Item

Hardware guest customization for OVF templates would need to be supported in vSphere to enable this capability.

Regards, Ryan

Similar to #1441

cc @appilon and @iBrandyJackson for review and closure.

appilon commented 2 years ago

Closing as per @tenthirtyam recommendation

sofixa commented 2 years ago

@appilon @tenthirtyam I don't see the link between hardware customisation and the fact that it's pretty non-obvious how to deploy an OVF template because terraform has optional parameters that are in fact required by VMware ( and rather useless i might say). Having that clearer in the docs and/or having defaults on the terraform side would make for an easier UX.

sofixa commented 2 years ago

So i checked the linked issue and it's an unrelated problem :) Manually mapping nics and disks fixes the 500 from this one. @appilon could you please reopen this one?

tenthirtyam commented 2 years ago

@appilon, we can retest and pick it up in a doc update in the least. @sofixa, feel free to suggest a resolution in a pull request.

Ryan

appilon commented 2 years ago

I'll re-open and have it close once we at least provide a documentation update

hashicorp / terraform-provider-vsphere