search

hashicorp / terraform-provider-vsphere

Terraform Provider for VMware vSphere
https://registry.terraform.io/providers/hashicorp/vsphere/
Mozilla Public License 2.0
612 stars 449 forks source link

v1.16 ERROR:ServerFaultCode: NoPermission #966

Closed Bandyman closed 4 years ago

Bandyman commented 4 years ago

Hi, We ran into an issue this morning with not being able to create any new nodes on our vsphere. Not a lot of usefull output with only error showing when applying a plan being Error: ServerFaultCode: NoPermission

Setting the output to trace revealed a little more, getting the following output during the plan stage.

4261 Error: ServerFaultCode: NoPermission 4262 2020-02-04T12:45:39.115Z [DEBUG] plugin: plugin process exited: path=/build/terraform/terraform-windows-vm/projects/.terraform/plugins/linux_amd64/terraform-provider-vsphere_v1.16.0_x4 pid=202 4263 Error: ServerFaultCode: NoPermission 4264 2020-02-04T12:45:39.115Z [DEBUG] plugin: plugin exited 4265 Error: ServerFaultCode: NoPermission 4266 Error: ServerFaultCode: NoPermission 4267 Error: ServerFaultCode: NoPermission 4268 Error: ServerFaultCode: NoPermission 4269 Error: ServerFaultCode: NoPermission 4270 Error: ServerFaultCode: NoPermission 4271 Error: ServerFaultCode: NoPermission 4272 Error: ServerFaultCode: NoPermission 4273 2020-02-04T12:45:39.116Z [DEBUG] plugin: plugin process exited: path=/builds/terraform/terraform-windows-vm/projects/.terraform/plugins/linux_amd64/terraform-provider-vsphere_v1.16.0_x4 pid=189 4274 2020-02-04T12:45:39.116Z [DEBUG] plugin: plugin exited 4275 Error: ServerFaultCode: NoPermission 4276 ERROR: Job failed: exit code 1

Terraform Version: 0.12.18 vSphere Provider Version : 1.16.0

Our account according to IT has full admin privalges on vsphere, reverting back to vsphere plugin version 1.15.0 and hard locking to that version fixed it for us.

Let me know if you need more information and I try and help Thanks, Tristan

f4nha commented 4 years ago

Same here, Failed on 1.16 works fine on 1.14 and 1.15

Error: disk.0: validation failed (ServerFaultCode: NoPermission)

  on machines.tf line 11, in resource "vsphere_virtual_machine" "tftest":
  11: resource "vsphere_virtual_machine" "tftest" {

Thanks

JosephHobbs commented 4 years ago

Seeing the same issue here as well...

8uachaille commented 4 years ago

We are experiencing the same issue with v1.16 - Please see our error in context of DEBUG log

2020-02-04T15:15:37.988Z [DEBUG] plugin.terraform-provider-vsphere_v1.16.0_x4: 2020/02/04 15:15:37 [DEBUG] VM “/“path/to/our/template/ISO found for UUID "4201b507-7907-3b1a-55d3-cdef9f4264cd" 2020-02-04T15:15:38.007Z [DEBUG] plugin.terraform-provider-vsphere_v1.16.0_x4: 2020/02/04 15:15:38 [DEBUG] queryAssociatedProfile: Retrieving storage policy of server object of type [virtualDiskId] and key [vm-1092382:2000]. 2020/02/04 15:15:38 [ERROR] root: eval: terraform.EvalDiff, err: disk.0: validation failed (ServerFaultCode: NoPermission) 2020/02/04 15:15:38 [ERROR] root: eval: terraform.EvalSequence, err: disk.0: validation failed (ServerFaultCode: NoPermission) 2020/02/04 15:15:38 [TRACE] [walkPlan] Exiting eval tree: vsphere_virtual_machine.vm 2020/02/04 15:15:38 [TRACE] dag/walk: upstream errored, not walking "meta.count-boundary (count boundary fixup)" 2020/02/04 15:15:38 [TRACE] dag/walk: upstream errored, not walking "provisioner.local-exec (close)" 2020/02/04 15:15:38 [TRACE] dag/walk: upstream errored, not walking "provisioner.remote-exec (close)" 2020/02/04 15:15:38 [TRACE] dag/walk: upstream errored, not walking "provider.vsphere (close)" 2020/02/04 15:15:38 [TRACE] dag/walk: upstream errored, not walking "provisioner.file (close)" 2020/02/04 15:15:38 [TRACE] dag/walk: upstream errored, not walking "root" 2020/02/04 15:15:38 [DEBUG] plugin: waiting for all plugin processes to complete... 2020-02-04T15:15:38.225Z [DEBUG] plugin.terraform: remote-exec-provisioner (internal) 2020/02/04 15:15:38 [ERR] plugin: plugin server: accept unix /tmp/plugin930112039: use of closed network connection 2020-02-04T15:15:38.225Z [DEBUG] plugin.terraform: remote-exec-provisioner (internal) 2020/02/04 15:15:38 [DEBUG] plugin: waiting for all plugin processes to complete... 2020-02-04T15:15:38.225Z [DEBUG] plugin: plugin process exited: path=/home/terraform/bin/terraform

arsiesys commented 4 years ago

Hello,

Same issue here.

After debug, it's related the following change: https://github.com/terraform-providers/terraform-provider-vsphere/pull/881/commits/12e2fc95bdfbfccdcad245919ebd09158c266746

Could we know which access/role name is missing to be able to fix it ? :p Thanks!

aareet commented 4 years ago

Thank you for filing this issue - we're investigating the problem

bill-rich commented 4 years ago

I'm working on tracking down the potential causes of this issue. There are a few data points I could use that would help make sure I cover all the cases.

1) What vCenter/vSphere version are you using? 2) Does the user Terraform is running as have "Profile-driven storage" permissions at the vCenter level?

Thanks, and I'll provide updates shortly.

arsiesys commented 4 years ago

Hello @bill-rich,

We are running in Vcenter 6.5.

The user running terraform had some specific RW access on ressources pools/Datastore and was running fine in 1.15. Also, the user had a global read only access on the vcenter. However, it seems that the global read only do not cover the profile-driven storage. With the complementary access "profile-driven storage view", it work!

I guess it could be good to document (or catch the error and print a detailed output) it as we will not be the only ones to get impacted :p.

vkmellon commented 4 years ago

The same we have: err: disk.0: validation failed (ServerFaultCode: NoPermission)

kavson commented 4 years ago

Also just started getting this error:

Error: Error running plan: 2 errors occurred:

8uachaille commented 4 years ago

Previously:

We were experiencing the above (v1.16 ERROR:ServerFaultCode: NoPermission #966) error

We were running Terraform v0.11.11 with v1.16 vsphere provider against vCenter 6.7 / ESXi 6.5

We got the following error: err: disk.0: validation failed (ServerFaultCode: NoPermission)

Now:

I tried to work-around this problem I still get the following error running terraform plan

 upgraded Terraform to v0.12.20 
 ran terraform 0.12upgrade
 Allocated Profile-driven storage (view) privilege to Terraform-related user role

 Error: disk.0: validation failed (ServerFaultCode: NoPermission)

   on config.tf line 32, in resource "vsphere_virtual_machine" "vm":
   32: resource "vsphere_virtual_machine" "vm" {

   [terraform@nohost ]$ /var/tmp/terraform --version
   Terraform v0.12.20
   + provider.vsphere v1.16.0
   [terraform@nohost ]$

I would like to have a working approach for vCenter 6.7 if possible

It would also help to know how to select a specific vSphere provider version say v1.15.0

I tried the following stanza which seemed to agree with the provider documentation at https://github.com/terraform-providers/terraform-provider-vsphere

provider "vsphere" { version = "~> 1.15" user = “not” password = “working” vsphere_server = “server” allow_unverified_ssl = true }

However, my terraform run continues to use 1.16.0

[terraform@nohost]$ /var/tmp/terraform init Initializing the backend... Initializing provider plugins...

I looked for a state file to determine whether I needed to purge that but there isn't one in the pwd after the terraform plan nor is there one anywhere else on the host

jgrancell commented 4 years ago

It would also help to know how to select a specific vSphere provider version say v1.15.0

I tried the following stanza which seemed to agree with the provider documentation at https://github.com/terraform-providers/terraform-provider-vsphere

provider "vsphere" { version = "~> 1.15" user = “not” password = “working” vsphere_server = “server” allow_unverified_ssl = true }

You need to specify the version string correctly:

provider "vsphere" {
  version = "< 1.16.0"
  ...
}

The key being the < symbol which means you want a version less than 1.16.0. By using ~> you're specifying you want a release equal to or greater than 1.15, but below 2.0.

8uachaille commented 4 years ago

That worked - thanks for your help Josh

bill-rich commented 4 years ago

Thanks for testing that @arsiesys!

For everyone still experiencing this issue, it looks like is is due to new permissions being required for the addition of SPBM support in v1.16.0. Please check that the user Terraform is running as has "Profile-driven storage" permissions at the vCenter.

I will get the changelog updated with notes about the additional permissions.

8uachaille commented 4 years ago

Still fails with v0.12.20 and v1.16.1

$ egrep -i 'terraform|1.16' terraform.log | head 2020/02/19 09:24:00 [INFO] Terraform version: 0.12.20 ... 2020/02/19 09:24:00 [DEBUG] fetching provider location from "https://registry.terraform.io/v1/providers/hashicorp/vsphere/1.16.1/download/linux/amd64" [terraform@terraform ece02.vh.iot.ed.ac.uk]$

+

Error: WARNING: There was an error performing post-clone changes to virtual machine "/MY Datacenter/vm/YY/ Servers/my.f.q.d.n": error processing disk changes post-clone: disk.0: ServerFaultCode: NoPermission: RESOURCE (vm-1215521:2000), ACTION (queryAssociatedProfile): RESOURCE (vm-1215521), ACTION (PolicyIDByVirtualDisk) Additionally, there was an error removing the cloned virtual machine: error destroying virtual machine: ServerFaultCode: Permission to perform this operation was denied.

The virtual machine may still exist in Terraform state. If it does, the resource will need to be tainted before trying again. For more information on how to do this, see the following page: https://www.terraform.io/docs/commands/taint.html

If the virtual machine does not exist in state, manually delete it to try again.

on config.tf line 35, in resource "vsphere_virtual_machine" "vm": 35: resource "vsphere_virtual_machine" "vm" {

aareet commented 4 years ago

@glenfiddich have you ensured this - Please check that the user Terraform is running as has "Profile-driven storage" permissions at the vCenter.?

stevenklar commented 4 years ago

Upgrade to 1.16.1 and provide the mentioned permissions "Profile-driven storage" fixed it for us.

Ekallatum commented 4 years ago

Plugin version 1.16.2.

It's seems that it is necessary to set "Profile-driven storage" policy at the root group of vcenter server.

Fixed for us.

aareet commented 4 years ago

Closing this issue - please create a new issue if this recurs in current or future versions of the provider.

ghost commented 4 years ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!