search

turbonomic / kubeturbo

138 stars 75 forks source link

The IP_address of Move action may be improper sometimes. #95

Closed songbinliu closed 6 years ago

songbinliu commented 7 years ago

Problem: when kubeturbo executing a move action for Container_POD, it fails with the error log: "Cannot find node with IPs 10.128.0.1".

Env: Openshift Cluster 3.4 OpsMgr 6.0.0, with Market1. Kubeturbo PR93

Possible Reason The newSE field of the ActionItem is a virtual machine: enMaster, whose official IP address is: 10.10.173.119, and Kubernetes uses this IP address to find the host. However, the ActionItem only has an IP of "10.128.0.1".

So this maybe because the vCenter probe fails to get the correct IP address of the virtual machine. (stitching type is "uuid" in kubeturbo, and the stitching has successed).

screen shot 2017-09-07 at 1 33 27 pm
songbinliu commented 7 years ago

Here is the content of the move action:

I0907 10:16:36.030798 3249 client_protobuf_endpoint.go:129] [ServerRequestEndpoint][[waitForServerMessage] : received message is: &{ActionRequest:probeType:"OpenShift" accountValue:<key:"targetIdentifier" stringValue:"https://10.10.173.119:8443" > accountValue:<key:"password" stringValue:"****" > accountValue:<key:"username" stringValue:"developer" > actionExecutionDTO:<actionType:MOVE actionItem:<actionType:MOVE uuid:"_COlS0pPXEeeNpdCIShQVzw" targetSE:<entityType:CONTAINER_POD id:"ed86c7fd-73dc-11e7-abf9-00505680effd" displayName:"default/saturated.memory-256-2" commoditiesSold: commoditiesSold:<commodityType:VMEM reservation:262144 capacity:8.010812e+06 limit:0 > commoditiesSold:<commodityType:VMPM_ACCESS reservation:0 capacity:1e+10 limit:0 > commoditiesBought:<providerId:"4200088e-b4ed-7bb8-3a67-ce10a1d447ca" bought: bought:<commodityType:VMEM reservation:262144 capacity:8.388608e+06 limit:0 > bought: bought: bought: providerType:VIRTUAL_MACHINE > entityProperties:<namespace:"https://10.10.173.119:8443" name:"LocalName" value:"ed86c7fd-73dc-11e7-abf9-00505680effd" > entityProperties:<namespace:"https://10.10.173.119:8443" name:"TargetType" value:"OpenShift" > > hostedBySE:<entityType:VIRTUAL_MACHINE id:"4200088e-b4ed-7bb8-3a67-ce10a1d447ca" displayName:"enOpenShiftSlave1" commoditiesSold: commoditiesSold:<commodityType:VMEM reservation:827392 capacity:8.388608e+06 limit:-1 > commoditiesSold: commoditiesSold: commoditiesSold: commoditiesSold: commoditiesSold: commoditiesSold:<commodityType:APPLICATION reservation:0 capacity:1e+09 limit:0 > commoditiesSold:<commodityType:VMPM_ACCESS reservation:0 capacity:1e+10 limit:0 > commoditiesSold:<commodityType:VMPM_ACCESS reservation:0 capacity:1e+10 limit:0 > commoditiesSold:<commodityType:VMPM_ACCESS reservation:0 capacity:1e+10 limit:0 > commoditiesSold:<commodityType:VMPM_ACCESS reservation:0 capacity:1e+10 limit:0 > commoditiesSold:<commodityType:VMPM_ACCESS reservation:0 capacity:1e+10 limit:0 > commoditiesSold:<commodityType:VMPM_ACCESS reservation:0 capacity:1e+10 limit:0 > commoditiesSold:<commodityType:VMPM_ACCESS reservation:0 capacity:1e+10 limit:0 > commoditiesSold:<commodityType:VMPM_ACCESS reservation:0 capacity:1e+10 limit:0 > commoditiesSold:<commodityType:VMPM_ACCESS reservation:0 capacity:1e+10 limit:0 > commoditiesSold:<commodityType:CLUSTER reservation:0 capacity:4.857142272e+09 limit:0 > commoditiesBought:<providerId:"30333436-3638-5355-4532-313058394a43" bought: bought: bought:<commodityType:MEM reservation:0 capacity:5.36860256e+08 limit:0 > bought:<commodityType:BALLOONING reservation:0 capacity:5.36860256e+08 limit:0 > bought:<commodityType:SWAPPING reservation:0 capacity:5e+06 limit:0 > bought: bought:<commodityType:MEM_PROVISIONED reservation:0 capacity:5.368602624e+09 limit:0 > bought:<commodityType:DATASTORE reservation:0 capacity:1e+09 limit:0 > bought:<commodityType:IO_THROUGHPUT reservation:0 capacity:1.024e+06 limit:0 > bought: bought:<commodityType:NETWORK reservation:0 capacity:1e+09 limit:0 > bought:<commodityType:DATACENTER reservation:0 capacity:1e+09 limit:0 > bought:<commodityType:CLUSTER reservation:0 capacity:1e+09 limit:0 > bought:<commodityType:SOFTWARE_LICENSE_COMMODITY reservation:0 capacity:1e+09 limit:0 > providerType:PHYSICAL_MACHINE > commoditiesBought:<providerId:"45e99dd021432110b216c35eb133ffec7605c79b" bought: bought:<commodityType:MEM_ALLOCATION reservation:0 capacity:1.043268608e+09 limit:0 > providerType:VIRTUAL_DATACENTER > commoditiesBought:<providerId:"91f0448f-90170688" bought:<commodityType:STORAGE_CLUSTER reservation:0 capacity:1e+09 limit:0 > bought:<commodityType:STORAGE_AMOUNT reservation:0 capacity:2.815145e+06 limit:0 > bought:<commodityType:STORAGE_PROVISIONED reservation:0 capacity:5.63029e+06 limit:0 > bought: bought: bought:<commodityType:DSPM_ACCESS reservation:0 capacity:1e+09 limit:0 > providerType:STORAGE > entityProperties:<namespace:"DEFAULT" name:"numElasticIps" value:"0" > entityProperties:<namespace:"DEFAULT" name:"numCpus" value:"0" > entityProperties:<namespace:"DEFAULT" name:"ProductDescription" value:"" > entityProperties:<namespace:"DEFAULT" name:"KubernetesNodeName" value:"enslave1.octurbo.org" > entityProperties:<namespace:"DEFAULT" name:"Proxy_VM_UUID" value:"4200088e-b4ed-7bb8-3a67-ce10a1d447ca" > entityProperties:<namespace:"DEFAULT" name:"BillingType" value:"ONDEMAND" > entityProperties:<namespace:"10.10.150.251" name:"LocalName" value:"vm-322" > entityProperties:<namespace:"10.10.150.251" name:"TargetType" value:"vCenter" > entityProperties:<namespace:"https://10.10.173.119:8443" name:"LocalName" value:"84719c09-2902-11e7-a3d7-00505680effd" > entityProperties:<namespace:"https://10.10.173.119:8443" name:"TargetType" value:"OpenShift" > virtual_machine_data:<ipAddress:"10.130.0.1" guestName:"Red Hat Enterprise Linux 7 (64-bit)" 6:0 9:0 > > currentSE:<entityType:VIRTUAL_MACHINE id:"4200088e-b4ed-7bb8-3a67-ce10a1d447ca" displayName:"enOpenShiftSlave1" commoditiesSold: commoditiesSold:<commodityType:VMEM reservation:827392 capacity:8.388608e+06 limit:-1 > commoditiesSold: commoditiesSold: commoditiesSold: commoditiesSold: commoditiesSold: commoditiesSold:<commodityType:APPLICATION reservation:0 capacity:1e+09 limit:0 > commoditiesSold:<commodityType:VMPM_ACCESS reservation:0 capacity:1e+10 limit:0 > commoditiesSold:<commodityType:VMPM_ACCESS reservation:0 capacity:1e+10 limit:0 > commoditiesSold:<commodityType:VMPM_ACCESS reservation:0 capacity:1e+10 limit:0 > commoditiesSold:<commodityType:VMPM_ACCESS reservation:0 capacity:1e+10 limit:0 > commoditiesSold:<commodityType:VMPM_ACCESS reservation:0 capacity:1e+10 limit:0 > commoditiesSold:<commodityType:VMPM_ACCESS reservation:0 capacity:1e+10 limit:0 > commoditiesSold:<commodityType:VMPM_ACCESS reservation:0 capacity:1e+10 limit:0 > commoditiesSold:<commodityType:VMPM_ACCESS reservation:0 capacity:1e+10 limit:0 > commoditiesSold:<commodityType:VMPM_ACCESS reservation:0 capacity:1e+10 limit:0 > commoditiesSold:<commodityType:CLUSTER reservation:0 capacity:4.857142272e+09 limit:0 > commoditiesBought:<providerId:"30333436-3638-5355-4532-313058394a43" bought: bought: bought:<commodityType:MEM reservation:0 capacity:5.36860256e+08 limit:0 > bought:<commodityType:BALLOONING reservation:0 capacity:5.36860256e+08 limit:0 > bought:<commodityType:SWAPPING reservation:0 capacity:5e+06 limit:0 > bought: bought:<commodityType:MEM_PROVISIONED reservation:0 capacity:5.368602624e+09 limit:0 > bought:<commodityType:DATASTORE reservation:0 capacity:1e+09 limit:0 > bought:<commodityType:IO_THROUGHPUT reservation:0 capacity:1.024e+06 limit:0 > bought: bought:<commodityType:NETWORK reservation:0 capacity:1e+09 limit:0 > bought:<commodityType:DATACENTER reservation:0 capacity:1e+09 limit:0 > bought:<commodityType:CLUSTER reservation:0 capacity:1e+09 limit:0 > bought:<commodityType:SOFTWARE_LICENSE_COMMODITY reservation:0 capacity:1e+09 limit:0 > providerType:PHYSICAL_MACHINE > commoditiesBought:<providerId:"45e99dd021432110b216c35eb133ffec7605c79b" bought: bought:<commodityType:MEM_ALLOCATION reservation:0 capacity:1.043268608e+09 limit:0 > providerType:VIRTUAL_DATACENTER > commoditiesBought:<providerId:"91f0448f-90170688" bought:<commodityType:STORAGE_CLUSTER reservation:0 capacity:1e+09 limit:0 > bought:<commodityType:STORAGE_AMOUNT reservation:0 capacity:2.815145e+06 limit:0 > bought:<commodityType:STORAGE_PROVISIONED reservation:0 capacity:5.63029e+06 limit:0 > bought: bought: bought:<commodityType:DSPM_ACCESS reservation:0 capacity:1e+09 limit:0 > providerType:STORAGE > entityProperties:<namespace:"DEFAULT" name:"numElasticIps" value:"0" > entityProperties:<namespace:"DEFAULT" name:"numCpus" value:"0" > entityProperties:<namespace:"DEFAULT" name:"ProductDescription" value:"" > entityProperties:<namespace:"DEFAULT" name:"KubernetesNodeName" value:"enslave1.octurbo.org" > entityProperties:<namespace:"DEFAULT" name:"Proxy_VM_UUID" value:"4200088e-b4ed-7bb8-3a67-ce10a1d447ca" > entityProperties:<namespace:"DEFAULT" name:"BillingType" value:"ONDEMAND" > entityProperties:<namespace:"10.10.150.251" name:"LocalName" value:"vm-322" > entityProperties:<namespace:"10.10.150.251" name:"TargetType" value:"vCenter" > entityProperties:<namespace:"https://10.10.173.119:8443" name:"LocalName" value:"84719c09-2902-11e7-a3d7-00505680effd" > entityProperties:<namespace:"https://10.10.173.119:8443" name:"TargetType" value:"OpenShift" > virtual_machine_data:<ipAddress:"10.130.0.1" guestName:"Red Hat Enterprise Linux 7 (64-bit)" 6:0 9:0 > > newSE:<entityType:VIRTUAL_MACHINE id:"4200562d-ad69-923a-fc43-a8c2253f740f" displayName:"enOpenShiftMaster" commoditiesSold: commoditiesSold:<commodityType:VMEM reservation:622592 capacity:1.048576e+07 limit:-1 > commoditiesSold: commoditiesSold: commoditiesSold: commoditiesSold: commoditiesSold:<commodityType:APPLICATION reservation:0 capacity:1e+09 limit:0 > commoditiesSold:<commodityType:VMPM_ACCESS reservation:0 capacity:1e+10 limit:0 > commoditiesSold:<commodityType:VMPM_ACCESS reservation:0 capacity:1e+10 limit:0 > commoditiesSold:<commodityType:VMPM_ACCESS reservation:0 capacity:1e+10 limit:0 > commoditiesSold:<commodityType:VMPM_ACCESS reservation:0 capacity:1e+10 limit:0 > commoditiesSold:<commodityType:VMPM_ACCESS reservation:0 capacity:1e+10 limit:0 > commoditiesSold:<commodityType:VMPM_ACCESS reservation:0 capacity:1e+10 limit:0 > commoditiesSold:<commodityType:VMPM_ACCESS reservation:0 capacity:1e+10 limit:0 > commoditiesSold:<commodityType:VMPM_ACCESS reservation:0 capacity:1e+10 limit:0 > commoditiesSold:<commodityType:VMPM_ACCESS reservation:0 capacity:1e+10 limit:0 > commoditiesSold:<commodityType:CLUSTER reservation:0 capacity:4.857142272e+09 limit:0 > commoditiesBought:<providerId:"30333436-3638-5355-4532-313058394a43" bought: bought: bought:<commodityType:MEM reservation:0 capacity:5.36860256e+08 limit:0 > bought:<commodityType:BALLOONING reservation:0 capacity:5.36860256e+08 limit:0 > bought:<commodityType:SWAPPING reservation:0 capacity:5e+06 limit:0 > bought: bought:<commodityType:MEM_PROVISIONED reservation:0 capacity:5.368602624e+09 limit:0 > bought:<commodityType:DATASTORE reservation:0 capacity:1e+09 limit:0 > bought:<commodityType:IO_THROUGHPUT reservation:0 capacity:1.024e+06 limit:0 > bought: bought:<commodityType:NETWORK reservation:0 capacity:1e+09 limit:0 > bought:<commodityType:DATACENTER reservation:0 capacity:1e+09 limit:0 > bought:<commodityType:CLUSTER reservation:0 capacity:1e+09 limit:0 > bought:<commodityType:FLOW reservation:0 capacity:8.9999999500288e+14 limit:0 > bought: bought: bought:<commodityType:SOFTWARE_LICENSE_COMMODITY reservation:0 capacity:1e+09 limit:0 > providerType:PHYSICAL_MACHINE > commoditiesBought:<providerId:"45e99dd021432110b216c35eb133ffec7605c79b" bought: bought:<commodityType:MEM_ALLOCATION reservation:0 capacity:1.043268608e+09 limit:0 > providerType:VIRTUAL_DATACENTER > commoditiesBought:<providerId:"91f0448f-90170688" bought:<commodityType:STORAGE_CLUSTER reservation:0 capacity:1e+09 limit:0 > bought:<commodityType:STORAGE_AMOUNT reservation:0 capacity:2.815145e+06 limit:0 > bought:<commodityType:STORAGE_PROVISIONED reservation:0 capacity:5.63029e+06 limit:0 > bought: bought: bought:<commodityType:DSPM_ACCESS reservation:0 capacity:1e+09 limit:0 > providerType:STORAGE > entityProperties:<namespace:"DEFAULT" name:"numElasticIps" value:"0" > entityProperties:<namespace:"DEFAULT" name:"numCpus" value:"0" > entityProperties:<namespace:"DEFAULT" name:"ProductDescription" value:"" > entityProperties:<namespace:"DEFAULT" name:"KubernetesNodeName" value:"enmaster.octurbo.org" > entityProperties:<namespace:"DEFAULT" name:"Proxy_VM_UUID" value:"4200562d-ad69-923a-fc43-a8c2253f740f" > entityProperties:<namespace:"DEFAULT" name:"BillingType" value:"ONDEMAND" > entityProperties:<namespace:"10.10.150.251" name:"LocalName" value:"vm-341" > entityProperties:<namespace:"10.10.150.251" name:"TargetType" value:"vCenter" > entityProperties:<namespace:"https://10.10.173.119:8443" name:"LocalName" value:"84123937-2902-11e7-a3d7-00505680effd" > entityProperties:<namespace:"https://10.10.173.119:8443" name:"TargetType" value:"OpenShift" > virtual_machine_data:<ipAddress:"10.128.0.1" guestName:"Red Hat Enterprise Linux 7 (64-bit)" 6:0 9:0 > > providers:<entityType:VIRTUAL_MACHINE ids:"4200088e-b4ed-7bb8-3a67-ce10a1d447ca" > > > }

enlinxu commented 7 years ago

in the case of K8s + VCenter, since we already knew the stitching doesn't work well with IP and we chose to use UUID, can we also use uuid for actions?

songbinliu commented 7 years ago

Yes, we can.

A even better way is to make sure that the entityProperities of VirtualMachine are all sent back. For example, here we can use the property: entityProperties:<namespace:"DEFAULT" name:"KubernetesNodeName" value:"enmaster.octurbo.org" > to get the node name.

But currently, even we have set some entityProperties, some of the properties are not sent back in the actionItem.

songbinliu commented 6 years ago

This is solved by PR113.