search

vmware / vic

vSphere Integrated Containers Engine is a container runtime for vSphere.
http://vmware.github.io/vic
Other
640 stars 173 forks source link

hostd error on container that we created: Invalid transition requested (VM_STATE_RECONFIGURING -> VM_STATE_RECONFIGURING) #5647

Open mhagen-vmware opened 7 years ago

mhagen-vmware commented 7 years ago

Generated from: docker %{VCH-PARAMS} create ${busybox} /bin/top

This is from 1-10-Docker-PS test.

It actually causes an exception that the server ends up catching from the looks of it, but I don't think we should be causing exceptions in the server in our standard day to day ops.

2017-07-06T13:02:58.250Z info hostd[33240B70] [Originator@6876 sub=Vimsvc.TaskManager opID=c3cfeabc user=droneci] Task Created : haTask-50-vim.VirtualMachine.reconfigure-26096
2017-07-06T13:02:58.252Z error hostd[335C2B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/58af7eb5-d9fc5174-6127-0cc47a9be492/bbf8e32d8ac3b32b972757053620b355d7215e30cad124fbf4294a68a1c10576/bbf8e32d8ac3b32b972757053620b355d7215e30cad124fbf4294a68a1c10576.vmx opID=c3cfeabc user=droneci] Invalid transition requested (VM_STATE_RECONFIGURING -> VM_STATE_RECONFIGURING): Tasks running
2017-07-06T13:02:58.252Z warning hostd[335C2B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/58af7eb5-d9fc5174-6127-0cc47a9be492/bbf8e32d8ac3b32b972757053620b355d7215e30cad124fbf4294a68a1c10576/bbf8e32d8ac3b32b972757053620b355d7215e30cad124fbf4294a68a1c10576.vmx opID=c3cfeabc user=droneci] Failed to start Reconfiguring: vim.fault.TaskInProgress
2017-07-06T13:02:58.252Z info hostd[335C2B70] [Originator@6876 sub=Default opID=c3cfeabc user=droneci] AdapterServer caught exception: vim.fault.TaskInProgress
2017-07-06T13:02:58.252Z info hostd[335C2B70] [Originator@6876 sub=Vimsvc.TaskManager opID=c3cfeabc user=droneci] Task Completed : haTask-50-vim.VirtualMachine.reconfigure-26096 Status error
mhagen-vmware commented 7 years ago

vmsupport-bundle.zip

corrieb commented 7 years ago

Mike should this be in "New Issues"? Don't want to question the master but...

dougm commented 7 years ago

I'm going to start on #6039 first, @cgtexmex you want to take this one?

In general, VIC triggering a TaskInProgress "error" is not going to cause any harm. We could check the target resource for any task in progress via the recentTask property, but I think it'll always be a race since any VM resource could be transitioned to the RECONFIGURING state outside of VIC (such as via the UI). But we may be able to reduce the chance of this case from the VIC perspective.

mdubya66 commented 7 years ago

based on the above info and after conversations with @mhagen-vmware, pulling this from 1.2

hickeng commented 7 years ago

It's likely that this will result in a new condition in the pkg/vsphere/tasks for immediate retry, along with the VM_STATE_GUEST_OPERATION (not necessarily the actual state name!) and maybe VM_MIGRATION.

matthewavery commented 7 years ago

@mhagen-vmware I will look through the logs for this as I am working on #6370 as well. Looks like this should get handled by the task package and if it is not we will need to wrap this in the task retry which is designed for all of the intermittent failures due to VC state/config fluctuations.

matthewavery commented 7 years ago

@mhagen-vmware did this occur on a specific CI run? I am curious if you also have the VCH logs.

matthewavery commented 7 years ago

I am grabbing this for Sprint 18. I intend to atleast confirm that the tasks package is handling this properly. Or figure out where the task package is not being used where it should be.