Closed JasonChinsen closed 1 year ago
vsiddharth
commented
5 years ago @JasonChinsen Thanks a lot for reporting this issue. Could you please describe the actions performed to end up in this situation? Could you also provide logs from the vpc-resource-controller pod?
vsiddharth
commented
5 years ago @JasonChinsen Have you tried restarting the vpc-resource-controller pod?
JasonChinsen
commented
5 years ago @vsiddharth This is a stip of the logs before I restarted the vpc-resource-controller:
I0405 14:02:32.443824 1 reconciler.go:102] Reconciler worker 1 starting processing node ip-192-168-71-112.us-west-2.compute.internal.
I0405 14:02:32.443834 1 reconciler.go:123] Reconciler checking resource vpc.amazonaws.com/PrivateIPv4Address warmpool size 3 desired 3 on node ip-192-168-71-112.us-west-2.compute.internal.
I0405 14:02:32.443840 1 reconciler.go:123] Reconciler checking resource vpc.amazonaws.com/ENI warmpool size 0 desired 0 on node ip-192-168-71-112.us-west-2.compute.internal.
I0405 14:02:32.443845 1 reconciler.go:106] Reconciler worker 1 completed processing node ip-192-168-71-112.us-west-2.compute.internal.
I0405 14:02:32.461741 1 watcher.go:236] Pod watcher completed processing pod <redacted>.
I0405 14:07:06.886385 1 watcher.go:247] Pod watcher processing deleted pod <redacted> on node ip-192-168-71-112.us-west-2.compute.internal.
I0405 14:07:06.886439 1 manager.go:279] Node manager freed resource vpc.amazonaws.com/PrivateIPv4Address 192.168.125.127 on node ip-192-168-71-112.us-west-2.compute.internal for pod <redacted>, pool &{Capacity:14 InUse:map[192.168.71.112:node 192.168.112.14:windows-server-iis-7dcfc7c79b-zxqgj] Warm:[192.168.84.132 192.168.108.247 192.168.86.56 192.168.125.127] Pending:0}.
I0405 14:07:06.886484 1 reconciler.go:102] Reconciler worker 1 starting processing node ip-192-168-71-112.us-west-2.compute.internal.
I0405 14:07:06.886495 1 reconciler.go:123] Reconciler checking resource vpc.amazonaws.com/ENI warmpool size 0 desired 0 on node ip-192-168-71-112.us-west-2.compute.internal.
I0405 14:07:06.886501 1 reconciler.go:123] Reconciler checking resource vpc.amazonaws.com/PrivateIPv4Address warmpool size 4 desired 3 on node ip-192-168-71-112.us-west-2.compute.internal.
I0405 14:07:06.886505 1 reconciler.go:106] Reconciler worker 1 completed processing node ip-192-168-71-112.us-west-2.compute.internal.
I0405 14:08:13.092459 1 watcher.go:194] Pod watcher processing pod apr-dbmigration-85bf6d544-nqmw6 on node ip-192-168-212-195.us-west-2.compute.internal.
I0405 14:08:13.092502 1 manager.go:234] Node manager allocated resource vpc.amazonaws.com/PrivateIPv4Address 192.168.253.100 for pod <redacted> on node ip-192-168-212-195.us-west-2.compute.internal.
I0405 14:08:13.092512 1 watcher.go:295] Pod watcher annotating pod <redacted> with resource vpc.amazonaws.com/PrivateIPv4Address 192.168.253.100.
I0405 14:08:13.092674 1 reconciler.go:102] Reconciler worker 1 starting processing node ip-192-168-212-195.us-west-2.compute.internal.
I0405 14:08:13.092688 1 reconciler.go:123] Reconciler checking resource vpc.amazonaws.com/ENI warmpool size 0 desired 0 on node ip-192-168-212-195.us-west-2.compute.internal.
I0405 14:08:13.092693 1 reconciler.go:123] Reconciler checking resource vpc.amazonaws.com/PrivateIPv4Address warmpool size 2 desired 3 on node ip-192-168-212-195.us-west-2.compute.internal.
I0405 14:08:13.092698 1 reconciler.go:106] Reconciler worker 1 completed processing node ip-192-168-212-195.us-west-2.compute.internal.
I0405 14:08:13.110191 1 watcher.go:236] Pod watcher completed processing pod <redacted>.
I0405 14:29:24.752556 1 watcher.go:190] Pod watcher ignoring pod vpc-admission-webhook-deployment-66c679d896-zfs2m on unmanaged node ip-192-168-175-63.us-west-2.compute.internal.
I0405 14:29:37.713197 1 watcher.go:247] Pod watcher processing deleted pod vpc-admission-webhook-deployment-66c679d896-2nk8l on node ip-192-168-237-38.us-west-2.compute.internal.
I0405 14:31:33.638817 1 watcher.go:247] Pod watcher processing deleted pod aws-node-68mlh on node ip-192-168-104-142.us-west-2.compute.internal.
I0405 14:31:33.663910 1 watcher.go:190] Pod watcher ignoring pod aws-node-5gd4r on unmanaged node ip-192-168-104-142.us-west-2.compute.internal.
I0405 14:32:18.984697 1 watcher.go:247] Pod watcher processing deleted pod aws-node-nrlct on node ip-192-168-175-63.us-west-2.compute.internal.
I0405 14:32:19.001268 1 watcher.go:190] Pod watcher ignoring pod aws-node-6mp6n on unmanaged node ip-192-168-175-63.us-west-2.compute.internal.
I0405 14:32:27.716990 1 watcher.go:247] Pod watcher processing deleted pod aws-node-dzx27 on node ip-192-168-237-38.us-west-2.compute.internal.
I0405 14:32:27.724400 1 watcher.go:190] Pod watcher ignoring pod aws-node-6khfw on unmanaged node ip-192-168-237-38.us-west-2.compute.internal.After restart
I0405 16:14:20.767935 1 watcher.go:295] Pod watcher annotating pod windows-test-5d469bdcb9-xzmpw with resource vpc.amazonaws.com/PrivateIPv4Address 192.168.108.247.
I0405 16:14:20.777301 1 watcher.go:236] Pod watcher completed processing pod windows-test-5d469bdcb9-xzmpw.
I0405 16:14:25.362866 1 reconciler.go:30] Node manager reconciler started.
I0405 16:14:25.362910 1 reconciler.go:102] Reconciler worker 3 starting processing node ip-192-168-212-195.us-west-2.compute.internal.
I0405 16:14:25.362921 1 reconciler.go:123] Reconciler checking resource vpc.amazonaws.com/PrivateIPv4Address warmpool size 2 desired 3 on node ip-192-168-212-195.us-west-2.compute.internal.
I0405 16:14:25.362927 1 reconciler.go:123] Reconciler checking resource vpc.amazonaws.com/ENI warmpool size 0 desired 0 on node ip-192-168-212-195.us-west-2.compute.internal.
I0405 16:14:25.362932 1 reconciler.go:106] Reconciler worker 3 completed processing node ip-192-168-212-195.us-west-2.compute.internal.
I0405 16:14:25.362943 1 reconciler.go:102] Reconciler worker 1 starting processing node ip-192-168-71-112.us-west-2.compute.internal.
I0405 16:14:25.362949 1 reconciler.go:123] Reconciler checking resource vpc.amazonaws.com/ENI warmpool size 0 desired 0 on node ip-192-168-71-112.us-west-2.compute.internal.
I0405 16:14:25.362954 1 reconciler.go:123] Reconciler checking resource vpc.amazonaws.com/PrivateIPv4Address warmpool size 2 desired 3 on node ip-192-168-71-112.us-west-2.compute.internal.
I0405 16:14:25.362959 1 reconciler.go:106] Reconciler worker 1 completed processing node ip-192-168-71-112.us-west-2.compute.internal.
I0405 16:14:25.362968 1 reconciler.go:102] Reconciler worker 2 starting processing node ip-192-168-131-121.us-west-2.compute.internal.
I0405 16:14:25.362974 1 reconciler.go:123] Reconciler checking resource vpc.amazonaws.com/PrivateIPv4Address warmpool size 3 desired 3 on node ip-192-168-131-121.us-west-2.compute.internal.
I0405 16:14:25.362978 1 reconciler.go:123] Reconciler checking resource vpc.amazonaws.com/ENI warmpool size 0 desired 0 on node ip-192-168-131-121.us-west-2.compute.internal.
I0405 16:14:25.362999 1 reconciler.go:106] Reconciler worker 2 completed processing node ip-192-168-131-121.us-west-2.compute.internal.Moreover, the pods in pending state started to run.
netsh advfirewall set allprofiles state offshoutdown \rvsiddharth
commented
5 years ago @JasonChinsen Thanks a lot for getting back with the relevant information.
Did restarting the vpc-resource-controller update the node resources vpc.amazonaws.com/ENI and vpc.amazonaws.com/PrivateIPv4Address correctly?
JasonChinsen
commented
5 years ago @vsiddharth there is nothing new when I describe the node ... but the pods did start to run.
Resource Requests Limits
-------- -------- ------
cpu 0 (0%) 0 (0%)
memory 0 (0%) 0 (0%)
ephemeral-storage 0 (0%) 0 (0%)
vpc.amazonaws.com/ENI 0 0
vpc.amazonaws.com/PrivateIPv4Address 1 1@vsiddharth this may be out of context for this issue ... but ...
JasonChinsen
commented
5 years ago @vsiddharth I have spent a little more time on this issue and it seems like the node drops the network every so often (sorry I can't be more clear on that) but restarting pods, vpc-admission-webhook-deployment, aws-node, and vpc-resource-controller seem to bring it back into line ...
Do you want me to close this ticket for now?
aarongooch
commented
5 years ago I am also experiencing issues with Insufficient vpc.amazonaws.com/PrivateIPv4Address please let me know if you'd like another issue.
Tell us about your request I am running a mixed k8s cluster based on eks-windows-preview
Which service(s) is this request for? EKS
Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
I've noticed when restarting a pod running on the Windows EC2 instance that has an assigned IP the recreated pod can not get a new IP. I've seen the reported solution to restart the vpc-resource-controller and doing so sometimes helps assign the pod an IP. However, in my case the pod is attempting to communicate with AWS Elasticache instance through the assigned IP but is unable to do so. In this event I must reboot the EC2 instance and restart the vpc-resource-controller.
Are you currently working around this issue?
Combination of rebooting vpc-resource-controller and rebooting the Windows EC2 instance
Additional context We have a pub sub architecture running on k8s mixed cluster. Pub/Sub Communication is through AWS Elasticache. Pub from linux pods, sub by windows pods.
vsiddharth
commented
5 years ago Thanks a lot for reporting this. There are two different issues here. We have provided a workaround at https://github.com/aws/containers-roadmap/issues/236 for one of the issue.
somujay
commented
5 years ago
- connection
I am also experiencing issues with
Insufficient vpc.amazonaws.com/PrivateIPv4Addressplease let me know if you'd like another issue.Tell us about your request I am running a mixed k8s cluster based on eks-windows-preview
Which service(s) is this request for? EKS
Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
I've noticed when restarting a pod running on the Windows EC2 instance that has an assigned IP the recreated pod can not get a new IP. I've seen the reported solution to restart the
vpc-resource-controllerand doing so sometimes helps assign the pod an IP. However, in my case the pod is attempting to communicate with AWS Elasticache instance through the assigned IP but is unable to do so. In this event I must reboot the EC2 instance and restart thevpc-resource-controller.Are you currently working around this issue? Combination of rebooting
vpc-resource-controllerand rebooting the Windows EC2 instanceAdditional context We have a pub sub architecture running on k8s mixed cluster. Pub/Sub Communication is through AWS Elasticache. Pub from linux pods, sub by windows pods.
Hey Aaron, for the issue that POD couldn't communicate to ElasticCache, can you share the details little more? Such as when POD couldn't communicate to ElasticCache, 1) Is it able to communicate to Cluster services by service ip? 2) Is it able to communicate to Internet? 3) Also can you share the CNI log (C:\ProgramData\amazon\eks\cni\logs\ --> Log file of the repro time) 4) Send kube-proxy logs from windows Event log (Application and Service Logs -> EKS (filter Event source by kube-proxy))
aarongooch
commented
5 years ago @somujay Re-created the issue this morning and below is the log output.
Is it able to communicate to Cluster services by service ip?
No, output of requests below
Is it able to communicate to Internet?
No, output of requests below
Preview is showing some logs cut off. I've created a gist that has all logs as well for your reference: https://gist.github.com/aarongooch/c7e484bc41936bb81050d9fa07909294
Connectivity Tests:
Sanity from linux pod:
# curl http://portaladmin:8000 -v * Rebuilt URL to: http://portaladmin:8000/
* Trying 172.20.234.19...
* TCP_NODELAY set
* Connected to portaladmin (172.20.234.19) port 8000 (#0)
> HEAD / HTTP/1.1
> Host: portaladmin:8000
> User-Agent: curl/7.52.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Thu, 09 May 2019 18:57:31 GMT
< Server: WSGIServer/0.2 CPython/3.5.7
< Content-Length: 16348
< X-Frame-Options: SAMEORIGIN
< Content-Type: text/html
Check on Windows pod
PS C:\microengine> Invoke-WebRequest -Uri http://portaladmin:8000
Invoke-WebRequest : The remote name could not be resolved: 'portaladmin'
At line:1 char:1
+ Invoke-WebRequest -Uri http://portaladmin:8000
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:Htt
pWebRequest) [Invoke-WebRequest], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShe
ll.Commands.InvokeWebRequestCommand
PS C:\microengine>
PS C:\microengine>
PS C:\microengine>
PS C:\microengine> Invoke-WebRequest -Uri http://google.com
Invoke-WebRequest : The remote name could not be resolved: 'google.com'
CNI Logs:
2019-05-09T18:41:39Z [INFO] Plugin vpc-shared-eni version executing CNI command.
2019-05-09T18:41:39Z [INFO] Executing DEL with netconfig: &{NetConf:{CNIVersion:0.3.1 Name:vpc Type:vpc-shared-eni Capabilities:map[] IPAM:{Type:} DNS:{Nameservers:[172.20.0.10] Domain: Search:[] Options:[]}} ENIName: ENIMACAddress:0e:a1:55:1f:9f:9a ENIIPAddress:10.0.2.237/24 VPCCI
DRs:[{IP:10.0.0.0 Mask:ffff0000}] BridgeNetNSPath: IPAddress:10.0.2.247/24 GatewayIPAddress:10.0.2.1 InterfaceType: TapUserID:0 Kubernetes:{Namespace:default PodName:enginebackends-xvirus-58b9f76dcf-4q85t PodInfraContainerID:6465a0fa29446f9b9591419bf5aa52a863675c4c7567871805ce222a5
9495e07 ServiceCIDR:172.20.0.0/16}} ContainerID:6465a0fa29446f9b9591419bf5aa52a863675c4c7567871805ce222a59495e07 Netns:none IfName:eth0 Args:IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NAME=enginebackends-xvirus-58b9f76dcf-4q85t;K8S_POD_INFRA_CONTAINER_ID=6465a0fa29446f9b9591
419bf5aa52a863675c4c7567871805ce222a59495e07.
2019-05-09T18:41:39Z [INFO] Detaching HNS endpoint 48873B83-40CE-418F-9D02-44C1F1AA8E88 from container 6465a0fa29446f9b9591419bf5aa52a863675c4c7567871805ce222a59495e07 netns.
2019-05-09T18:41:40Z [INFO] Deleting HNS endpoint name: cid-6465a0fa29446f9b9591419bf5aa52a863675c4c7567871805ce222a59495e07 ID: 48873B83-40CE-418F-9D02-44C1F1AA8E88
2019-05-09T18:41:40Z [INFO] Plugin vpc-shared-eni version executing CNI command.
2019-05-09T18:41:40Z [INFO] Executing ADD with netconfig: &{NetConf:{CNIVersion:0.3.1 Name:vpc Type:vpc-shared-eni Capabilities:map[] IPAM:{Type:} DNS:{Nameservers:[172.20.0.10] Domain: Search:[] Options:[]}} ENIName: ENIMACAddress:0e:a1:55:1f:9f:9a ENIIPAddress:10.0.2.237/24 VPCCI
DRs:[{IP:10.0.0.0 Mask:ffff0000}] BridgeNetNSPath: IPAddress:10.0.2.247/24 GatewayIPAddress:10.0.2.1 InterfaceType: TapUserID:0 Kubernetes:{Namespace:default PodName:enginebackends-xvirus-58b9f76dcf-4q85t PodInfraContainerID:e19f978e4b01a6623edefdd298a89cc458e36c1e1fe4c1a11c5bf2784
d9994cc ServiceCIDR:172.20.0.0/16}} ContainerID:e19f978e4b01a6623edefdd298a89cc458e36c1e1fe4c1a11c5bf2784d9994cc Netns:container:6465a0fa29446f9b9591419bf5aa52a863675c4c7567871805ce222a59495e07 IfName:eth0 Args:IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NAME=enginebackends-x
virus-58b9f76dcf-4q85t;K8S_POD_INFRA_CONTAINER_ID=e19f978e4b01a6623edefdd298a89cc458e36c1e1fe4c1a11c5bf2784d9994cc.
2019-05-09T18:41:40Z [INFO] Running on HNS version: {Major:9 Minor:1}
2019-05-09T18:41:40Z [INFO] Found existing HNS network vpcbr0ea1551f9f9a.
2019-05-09T18:41:40Z [INFO] Container e19f978e4b01a6623edefdd298a89cc458e36c1e1fe4c1a11c5bf2784d9994cc shares netns of container 6465a0fa29446f9b9591419bf5aa52a863675c4c7567871805ce222a59495e07
2019-05-09T18:41:40Z [ERROR] Failed to find endpoint cid-6465a0fa29446f9b9591419bf5aa52a863675c4c7567871805ce222a59495e07 for container e19f978e4b01a6623edefdd298a89cc458e36c1e1fe4c1a11c5bf2784d9994cc.
2019-05-09T18:41:40Z [ERROR] Failed to create endpoint: failed to find endpoint cid-6465a0fa29446f9b9591419bf5aa52a863675c4c7567871805ce222a59495e07: Endpoint cid-6465a0fa29446f9b9591419bf5aa52a863675c4c7567871805ce222a59495e07 not found.
2019-05-09T18:41:40Z [ERROR] CNI command failed: failed to find endpoint cid-6465a0fa29446f9b9591419bf5aa52a863675c4c7567871805ce222a59495e07: Endpoint cid-6465a0fa29446f9b9591419bf5aa52a863675c4c7567871805ce222a59495e07 not found
2019-05-09T18:41:40Z [INFO] Plugin vpc-shared-eni version executing CNI command.
2019-05-09T18:41:40Z [INFO] Executing ADD with netconfig: &{NetConf:{CNIVersion:0.3.1 Name:vpc Type:vpc-shared-eni Capabilities:map[] IPAM:{Type:} DNS:{Nameservers:[172.20.0.10] Domain: Search:[] Options:[]}} ENIName: ENIMACAddress:0e:a1:55:1f:9f:9a ENIIPAddress:10.0.2.237/24 VPCCI
DRs:[{IP:10.0.0.0 Mask:ffff0000}] BridgeNetNSPath: IPAddress:10.0.2.137/24 GatewayIPAddress:10.0.2.1 InterfaceType: TapUserID:0 Kubernetes:{Namespace:default PodName:enginebackends-xvirus-58b9f76dcf-ffvjf PodInfraContainerID:fc66311b306448e4a82e5fae0a736174e6db47a9ae07ccc284ba84bb9
52fab2d ServiceCIDR:172.20.0.0/16}} ContainerID:fc66311b306448e4a82e5fae0a736174e6db47a9ae07ccc284ba84bb952fab2d Netns:none IfName:eth0 Args:IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NAME=enginebackends-xvirus-58b9f76dcf-ffvjf;K8S_POD_INFRA_CONTAINER_ID=fc66311b306448e4a82e
5fae0a736174e6db47a9ae07ccc284ba84bb952fab2d.
2019-05-09T18:41:40Z [INFO] Running on HNS version: {Major:9 Minor:1}
2019-05-09T18:41:40Z [INFO] Found existing HNS network vpcbr0ea1551f9f9a.
2019-05-09T18:41:40Z [INFO] Creating HNS endpoint: {"Name":"cid-fc66311b306448e4a82e5fae0a736174e6db47a9ae07ccc284ba84bb952fab2d","VirtualNetworkName":"vpcbr0ea1551f9f9a","Policies":[{"Type":"OutBoundNAT","ExceptionList":["10.0.0.0/16","172.20.0.0/16"]},{"Type":"ROUTE","Destination
Prefix":"172.20.0.0/16","NeedEncap":true},{"Type":"ROUTE","DestinationPrefix":"10.0.2.237/32","NeedEncap":true}],"IPAddress":"10.0.2.137","DNSServerList":"172.20.0.10","PrefixLength":24}
2019-05-09T18:41:40Z [INFO] Received HNS endpoint response: &{Id:CE26F3A2-76C9-4739-8E1F-180409F04401 Name:cid-fc66311b306448e4a82e5fae0a736174e6db47a9ae07ccc284ba84bb952fab2d VirtualNetwork:29A3E292-5591-4E7A-AE21-C7F2239149E5 VirtualNetworkName:vpcbr0ea1551f9f9a Policies:[[123 34
69 120 99 101 112 116 105 111 110 76 105 115 116 34 58 91 34 49 48 46 48 46 48 46 48 47 49 54 34 44 34 49 55 50 46 50 48 46 48 46 48 47 49 54 34 93 44 34 84 121 112 101 34 58 34 79 117 116 66 111 117 110 100 78 65 84 34 125] [123 34 68 101 115 116 105 110 97 116 105 111 110 80 114
101 102 105 120 34 58 34 49 55 50 46 50 48 46 48 46 48 47 49 54 34 44 34 78 101 101 100 69 110 99 97 112 34 58 116 114 117 101 44 34 84 121 112 101 34 58 34 82 79 85 84 69 34 125] [123 34 68 101 115 116 105 110 97 116 105 111 110 80 114 101 102 105 120 34 58 34 49 48 46 48 46 50 4
6 50 51 55 47 51 50 34 44 34 78 101 101 100 69 110 99 97 112 34 58 116 114 117 101 44 34 84 121 112 101 34 58 34 82 79 85 84 69 34 125] [123 34 84 121 112 101 34 58 34 76 50 68 114 105 118 101 114 34 125]] MacAddress:00-15-5D-FB-12-67 IPAddress:10.0.2.137 DNSSuffix: DNSServerList:1
72.20.0.10 GatewayAddress:10.0.2.1 EnableInternalDNS:false DisableICC:false PrefixLength:24 IsRemoteEndpoint:false Namespace:<nil>}.
2019-05-09T18:41:40Z [INFO] Attaching HNS endpoint CE26F3A2-76C9-4739-8E1F-180409F04401 to container fc66311b306448e4a82e5fae0a736174e6db47a9ae07ccc284ba84bb952fab2d.
2019-05-09T18:41:40Z [INFO] Writing CNI result to stdout: Interfaces:[{Name:eth0 Mac:00:15:5d:fb:12:67 Sandbox:none}], IP:[{Version:4 Interface:0xc000332bf0 Address:{IP:10.0.2.137 Mask:ffffff00} Gateway:10.0.2.1}], DNS:{Nameservers:[] Domain: Search:[] Options:[]}
2019-05-09T18:41:40Z [INFO] Plugin vpc-shared-eni version executing CNI command.
2019-05-09T18:41:41Z [INFO] Executing ADD with netconfig: &{NetConf:{CNIVersion:0.3.1 Name:vpc Type:vpc-shared-eni Capabilities:map[] IPAM:{Type:} DNS:{Nameservers:[172.20.0.10] Domain: Search:[] Options:[]}} ENIName: ENIMACAddress:0e:a1:55:1f:9f:9a ENIIPAddress:10.0.2.237/24 VPCCI
DRs:[{IP:10.0.0.0 Mask:ffff0000}] BridgeNetNSPath: IPAddress:10.0.2.137/24 GatewayIPAddress:10.0.2.1 InterfaceType: TapUserID:0 Kubernetes:{Namespace:default PodName:enginebackends-xvirus-58b9f76dcf-ffvjf PodInfraContainerID:fc66311b306448e4a82e5fae0a736174e6db47a9ae07ccc284ba84bb9
52fab2d ServiceCIDR:172.20.0.0/16}} ContainerID:fc66311b306448e4a82e5fae0a736174e6db47a9ae07ccc284ba84bb952fab2d Netns:none IfName:eth0 Args:IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NAME=enginebackends-xvirus-58b9f76dcf-ffvjf;K8S_POD_INFRA_CONTAINER_ID=fc66311b306448e4a82e
5fae0a736174e6db47a9ae07ccc284ba84bb952fab2d.
2019-05-09T18:41:41Z [INFO] Running on HNS version: {Major:9 Minor:1}
2019-05-09T18:41:41Z [INFO] Found existing HNS network vpcbr0ea1551f9f9a.
2019-05-09T18:41:41Z [INFO] Found existing HNS endpoint cid-fc66311b306448e4a82e5fae0a736174e6db47a9ae07ccc284ba84bb952fab2d.
2019-05-09T18:41:41Z [INFO] HNS endpoint cid-fc66311b306448e4a82e5fae0a736174e6db47a9ae07ccc284ba84bb952fab2d is already attached to container ID fc66311b306448e4a82e5fae0a736174e6db47a9ae07ccc284ba84bb952fab2d.
2019-05-09T18:41:41Z [INFO] Writing CNI result to stdout: Interfaces:[{Name:eth0 Mac:00:15:5d:fb:12:67 Sandbox:none}], IP:[{Version:4 Interface:0xc000089cf8 Address:{IP:10.0.2.137 Mask:ffffff00} Gateway:10.0.2.1}], DNS:{Nameservers:[] Domain: Search:[] Options:[]}
2019-05-09T18:41:41Z [INFO] Plugin vpc-shared-eni version executing CNI command.
2019-05-09T18:41:41Z [INFO] Executing ADD with netconfig: &{NetConf:{CNIVersion:0.3.1 Name:vpc Type:vpc-shared-eni Capabilities:map[] IPAM:{Type:} DNS:{Nameservers:[172.20.0.10] Domain: Search:[] Options:[]}} ENIName: ENIMACAddress:0e:a1:55:1f:9f:9a ENIIPAddress:10.0.2.237/24 VPCCI
DRs:[{IP:10.0.0.0 Mask:ffff0000}] BridgeNetNSPath: IPAddress:10.0.2.137/24 GatewayIPAddress:10.0.2.1 InterfaceType: TapUserID:0 Kubernetes:{Namespace:default PodName:enginebackends-xvirus-58b9f76dcf-ffvjf PodInfraContainerID:fc66311b306448e4a82e5fae0a736174e6db47a9ae07ccc284ba84bb9
52fab2d ServiceCIDR:172.20.0.0/16}} ContainerID:fc66311b306448e4a82e5fae0a736174e6db47a9ae07ccc284ba84bb952fab2d Netns:none IfName:eth0 Args:IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NAME=enginebackends-xvirus-58b9f76dcf-ffvjf;K8S_POD_INFRA_CONTAINER_ID=fc66311b306448e4a82e
5fae0a736174e6db47a9ae07ccc284ba84bb952fab2d.
2019-05-09T18:41:41Z [INFO] Running on HNS version: {Major:9 Minor:1}
2019-05-09T18:41:41Z [INFO] Found existing HNS network vpcbr0ea1551f9f9a.
2019-05-09T18:41:41Z [INFO] Found existing HNS endpoint cid-fc66311b306448e4a82e5fae0a736174e6db47a9ae07ccc284ba84bb952fab2d.
2019-05-09T18:41:41Z [INFO] HNS endpoint cid-fc66311b306448e4a82e5fae0a736174e6db47a9ae07ccc284ba84bb952fab2d is already attached to container ID fc66311b306448e4a82e5fae0a736174e6db47a9ae07ccc284ba84bb952fab2d.
2019-05-09T18:41:41Z [INFO] Writing CNI result to stdout: Interfaces:[{Name:eth0 Mac:00:15:5d:fb:12:67 Sandbox:none}], IP:[{Version:4 Interface:0xc0002bcea8 Address:{IP:10.0.2.137 Mask:ffffff00} Gateway:10.0.2.1}], DNS:{Nameservers:[] Domain: Search:[] Options:[]}
2019-05-09T18:41:43Z [INFO] Plugin vpc-shared-eni version executing CNI command.
2019-05-09T18:41:43Z [INFO] Executing ADD with netconfig: &{NetConf:{CNIVersion:0.3.1 Name:vpc Type:vpc-shared-eni Capabilities:map[] IPAM:{Type:} DNS:{Nameservers:[172.20.0.10] Domain: Search:[] Options:[]}} ENIName: ENIMACAddress:0e:a1:55:1f:9f:9a ENIIPAddress:10.0.2.237/24 VPCCI
DRs:[{IP:10.0.0.0 Mask:ffff0000}] BridgeNetNSPath: IPAddress:10.0.2.137/24 GatewayIPAddress:10.0.2.1 InterfaceType: TapUserID:0 Kubernetes:{Namespace:default PodName:enginebackends-xvirus-58b9f76dcf-ffvjf PodInfraContainerID:cb1c44c1c15ad37abb2af782c9f5c86b9fca10bed796b1c457eaed9d9
3882089 ServiceCIDR:172.20.0.0/16}} ContainerID:cb1c44c1c15ad37abb2af782c9f5c86b9fca10bed796b1c457eaed9d93882089 Netns:container:fc66311b306448e4a82e5fae0a736174e6db47a9ae07ccc284ba84bb952fab2d IfName:eth0 Args:IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NAME=enginebackends-x
virus-58b9f76dcf-ffvjf;K8S_POD_INFRA_CONTAINER_ID=cb1c44c1c15ad37abb2af782c9f5c86b9fca10bed796b1c457eaed9d93882089.
2019-05-09T18:41:43Z [INFO] Running on HNS version: {Major:9 Minor:1}
2019-05-09T18:41:43Z [INFO] Found existing HNS network vpcbr0ea1551f9f9a.
2019-05-09T18:41:43Z [INFO] Container cb1c44c1c15ad37abb2af782c9f5c86b9fca10bed796b1c457eaed9d93882089 shares netns of container fc66311b306448e4a82e5fae0a736174e6db47a9ae07ccc284ba84bb952fab2d
2019-05-09T18:41:43Z [INFO] Found existing HNS endpoint cid-fc66311b306448e4a82e5fae0a736174e6db47a9ae07ccc284ba84bb952fab2d.
2019-05-09T18:41:43Z [INFO] Attaching HNS endpoint CE26F3A2-76C9-4739-8E1F-180409F04401 to container cb1c44c1c15ad37abb2af782c9f5c86b9fca10bed796b1c457eaed9d93882089.
2019-05-09T18:41:43Z [INFO] Writing CNI result to stdout: Interfaces:[{Name:eth0 Mac:00:15:5d:fb:12:67 Sandbox:container:fc66311b306448e4a82e5fae0a736174e6db47a9ae07ccc284ba84bb952fab2d}], IP:[{Version:4 Interface:0xc0002df170 Address:{IP:10.0.2.137 Mask:ffffff00} Gateway:10.0.2.1}
], DNS:{Nameservers:[] Domain: Search:[] Options:[]}
2019-05-09T18:41:43Z [INFO] Plugin vpc-shared-eni version executing CNI command.
2019-05-09T18:41:43Z [INFO] Executing ADD with netconfig: &{NetConf:{CNIVersion:0.3.1 Name:vpc Type:vpc-shared-eni Capabilities:map[] IPAM:{Type:} DNS:{Nameservers:[172.20.0.10] Domain: Search:[] Options:[]}} ENIName: ENIMACAddress:0e:a1:55:1f:9f:9a ENIIPAddress:10.0.2.237/24 VPCCI
DRs:[{IP:10.0.0.0 Mask:ffff0000}] BridgeNetNSPath: IPAddress:10.0.2.137/24 GatewayIPAddress:10.0.2.1 InterfaceType: TapUserID:0 Kubernetes:{Namespace:default PodName:enginebackends-xvirus-58b9f76dcf-ffvjf PodInfraContainerID:fc66311b306448e4a82e5fae0a736174e6db47a9ae07ccc284ba84bb9
52fab2d ServiceCIDR:172.20.0.0/16}} ContainerID:fc66311b306448e4a82e5fae0a736174e6db47a9ae07ccc284ba84bb952fab2d Netns:none IfName:eth0 Args:IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NAME=enginebackends-xvirus-58b9f76dcf-ffvjf;K8S_POD_INFRA_CONTAINER_ID=fc66311b306448e4a82e
5fae0a736174e6db47a9ae07ccc284ba84bb952fab2d.
2019-05-09T18:41:43Z [INFO] Running on HNS version: {Major:9 Minor:1}
2019-05-09T18:41:43Z [INFO] Found existing HNS network vpcbr0ea1551f9f9a.
2019-05-09T18:41:43Z [INFO] Found existing HNS endpoint cid-fc66311b306448e4a82e5fae0a736174e6db47a9ae07ccc284ba84bb952fab2d.
EKS Logs:
173495 May 09 18:41 Information kubelet 0 E0509 18:41:53.861345 3284 eviction_manager.go:243] eviction manager: failed to get get summary stats: failed to list pod stats: failed to list all container stats: rpc error: code = Unknown desc = Error re...
173494 May 09 18:41 Information kubelet 0 E0509 18:41:53.860696 3284 remote_runtime.go:473] ListContainerStats with filter &ContainerStatsFilter{Id:,PodSandboxId:,LabelSelector:map[string]string{},} from runtime service failed: rpc error: code = Un...
173493 May 09 18:41 Information kube-proxy 0 E0509 18:41:44.259346 3176 proxier.go:1034] Policy creation failed: hnsCall failed in Win32: The provided policy configuration is invalid or missing parameters. (0x803b000d)
173492 May 09 18:41 Information kube-proxy 0 I0509 18:41:44.240828 3176 proxier.go:117] Hns Endpoint resource, {"ID":"839379E0-CD60-4655-8D09-4AB586758A07","Name":"Ethernet","VirtualNetwork":"29A3E292-5591-4E7A-AE21-C7F2239149E5","VirtualNetworkName":...
173491 May 09 18:41 Information kube-proxy 0 E0509 18:41:44.239064 3176 proxier.go:1034] Policy creation failed: hnsCall failed in Win32: The provided policy configuration is invalid or missing parameters. (0x803b000d)
173490 May 09 18:41 Information kube-proxy 0 I0509 18:41:44.219397 3176 proxier.go:117] Hns Endpoint resource, {"ID":"467867A6-BFFE-4192-80BF-754DD80A1F85","Name":"Ethernet","VirtualNetwork":"29A3E292-5591-4E7A-AE21-C7F2239149E5","VirtualNetworkName":...
173489 May 09 18:41 Information kube-proxy 0 E0509 18:41:44.217624 3176 proxier.go:1034] Policy creation failed: hnsCall failed in Win32: The provided policy configuration is invalid or missing parameters. (0x803b000d)
173488 May 09 18:41 Information kube-proxy 0 I0509 18:41:44.196198 3176 proxier.go:117] Hns Endpoint resource, {"ID":"1D604E29-0183-4504-BE73-F1D458C19718","Name":"Ethernet","VirtualNetwork":"29A3E292-5591-4E7A-AE21-C7F2239149E5","VirtualNetworkName":...
173487 May 09 18:41 Information kube-proxy 0 E0509 18:41:44.193920 3176 proxier.go:1018] Endpoint information not available for service default/windows-server-iis-service:. Not applying any policy
173486 May 09 18:41 Information kube-proxy 0 E0509 18:41:44.193920 3176 proxier.go:1034] Policy creation failed: hnsCall failed in Win32: The provided policy configuration is invalid or missing parameters. (0x803b000d)
173485 May 09 18:41 Information kube-proxy 0 I0509 18:41:44.175817 3176 proxier.go:117] Hns Endpoint resource, {"ID":"B8FACC93-B15A-4FE8-92A6-4E4D25106B69","Name":"Ethernet","VirtualNetwork":"29A3E292-5591-4E7A-AE21-C7F2239149E5","VirtualNetworkName":...
173484 May 09 18:41 Information kube-proxy 0 E0509 18:41:44.173553 3176 proxier.go:1034] Policy creation failed: hnsCall failed in Win32: The provided policy configuration is invalid or missing parameters. (0x803b000d)
173483 May 09 18:41 Information kube-proxy 0 I0509 18:41:44.154329 3176 proxier.go:117] Hns Endpoint resource, {"ID":"39D2AF1A-FD6D-4DB3-9CFF-7F48750725BD","Name":"Ethernet","VirtualNetwork":"29A3E292-5591-4E7A-AE21-C7F2239149E5","VirtualNetworkName":...
173482 May 09 18:41 Information kube-proxy 0 I0509 18:41:44.152073 3176 proxier.go:117] Hns Endpoint resource, {"ID":"83146751-A52B-40BD-92DB-627D781527C3","Name":"Ethernet","VirtualNetwork":"29A3E292-5591-4E7A-AE21-C7F2239149E5","VirtualNetworkName":...
173481 May 09 18:41 Information kube-proxy 0 I0509 18:41:44.150284 3176 proxier.go:117] Hns Endpoint resource, {"ID":"B994CB85-C1B0-40E9-9981-21814900C1D7","Name":"Ethernet","VirtualNetwork":"29A3E292-5591-4E7A-AE21-C7F2239149E5","VirtualNetworkName":...
173480 May 09 18:41 Information kube-proxy 0 E0509 18:41:44.148036 3176 proxier.go:1034] Policy creation failed: hnsCall failed in Win32: The provided policy configuration is invalid or missing parameters. (0x803b000d)
173479 May 09 18:41 Information kube-proxy 0 I0509 18:41:44.127691 3176 proxier.go:117] Hns Endpoint resource, {"ID":"D6AF99F5-9524-4B35-8AF6-8F89E54F6D69","Name":"Ethernet","VirtualNetwork":"29A3E292-5591-4E7A-AE21-C7F2239149E5","VirtualNetworkName":...
173478 May 09 18:41 Information kube-proxy 0 E0509 18:41:44.124785 3176 proxier.go:1034] Policy creation failed: hnsCall failed in Win32: The provided policy configuration is invalid or missing parameters. (0x803b000d)
173477 May 09 18:41 Information kube-proxy 0 I0509 18:41:44.105141 3176 proxier.go:117] Hns Endpoint resource, {"ID":"6BB32B45-751F-4084-829B-D02B294F5A76","Name":"Ethernet","VirtualNetwork":"29A3E292-5591-4E7A-AE21-C7F2239149E5","VirtualNetworkName":...
173476 May 09 18:41 Information kube-proxy 0 E0509 18:41:44.103363 3176 proxier.go:1034] Policy creation failed: hnsCall failed in Win32: The provided policy configuration is invalid or missing parameters. (0x803b000d)
173475 May 09 18:41 Information kube-proxy 0 I0509 18:41:44.083657 3176 proxier.go:117] Hns Endpoint resource, {"ID":"2E2C33CB-EF48-48C0-BD3B-F1FEC06B913E","Name":"Ethernet","VirtualNetwork":"29A3E292-5591-4E7A-AE21-C7F2239149E5","VirtualNetworkName":...
173474 May 09 18:41 Information kube-proxy 0 E0509 18:41:44.081884 3176 proxier.go:1034] Policy creation failed: hnsCall failed in Win32: The provided policy configuration is invalid or missing parameters. (0x803b000d)
173473 May 09 18:41 Information kube-proxy 0 I0509 18:41:44.063336 3176 proxier.go:117] Hns Endpoint resource, {"ID":"C87770E4-58FD-4AAF-8D8B-009685939A3A","Name":"Ethernet","VirtualNetwork":"29A3E292-5591-4E7A-AE21-C7F2239149E5","VirtualNetworkName":...
173472 May 09 18:41 Information kube-proxy 0 I0509 18:41:44.061080 3176 proxier.go:117] Hns Endpoint resource, {"ID":"764B40FC-34A3-4F47-9774-E646E7246026","Name":"Ethernet","VirtualNetwork":"29A3E292-5591-4E7A-AE21-C7F2239149E5","VirtualNetworkName":...
173471 May 09 18:41 Information kube-proxy 0 I0509 18:41:44.059302 3176 proxier.go:117] Hns Endpoint resource, {"ID":"883906FE-7896-44F0-BF73-DD540014ED6C","Name":"Ethernet","VirtualNetwork":"29A3E292-5591-4E7A-AE21-C7F2239149E5","VirtualNetworkName":...
173470 May 09 18:41 Information kube-proxy 0 E0509 18:41:44.057526 3176 proxier.go:1034] Policy creation failed: hnsCall failed in Win32: The provided policy configuration is invalid or missing parameters. (0x803b000d)
173469 May 09 18:41 Information kube-proxy 0 I0509 18:41:44.036078 3176 proxier.go:117] Hns Endpoint resource, {"ID":"594B8EC9-521D-4287-A8BC-1F82C42749DF","Name":"Ethernet","VirtualNetwork":"29A3E292-5591-4E7A-AE21-C7F2239149E5","VirtualNetworkName":...
173468 May 09 18:41 Information kube-proxy 0 E0509 18:41:44.033817 3176 proxier.go:1034] Policy creation failed: hnsCall failed in Win32: The provided policy configuration is invalid or missing parameters. (0x803b000d)
173467 May 09 18:41 Information kube-proxy 0 I0509 18:41:44.015281 3176 proxier.go:117] Hns Endpoint resource, {"ID":"467867A6-BFFE-4192-80BF-754DD80A1F85","Name":"Ethernet","VirtualNetwork":"29A3E292-5591-4E7A-AE21-C7F2239149E5","VirtualNetworkName":...
173466 May 09 18:41 Information kube-proxy 0 E0509 18:41:44.013510 3176 proxier.go:1034] Policy creation failed: hnsCall failed in Win32: The provided policy configuration is invalid or missing parameters. (0x803b000d)
173465 May 09 18:41 Information kube-proxy 0 I0509 18:41:43.990925 3176 proxier.go:117] Hns Endpoint resource, {"ID":"D432DD32-AB9A-4F3D-B7CC-52B4467F3E33","Name":"Ethernet","VirtualNetwork":"29A3E292-5591-4E7A-AE21-C7F2239149E5","VirtualNetworkName":...
173464 May 09 18:41 Information kube-proxy 0 E0509 18:41:43.987996 3176 proxier.go:1034] Policy creation failed: hnsCall failed in Win32: The provided policy configuration is invalid or missing parameters. (0x803b000d)
173463 May 09 18:41 Information kube-proxy 0 I0509 18:41:43.965402 3176 proxier.go:117] Hns Endpoint resource, {"ID":"880E24B3-EE27-41EC-9FAD-399A37C44550","Name":"Ethernet","VirtualNetwork":"29A3E292-5591-4E7A-AE21-C7F2239149E5","VirtualNetworkName":...
173462 May 09 18:41 Information kube-proxy 0 E0509 18:41:43.962481 3176 proxier.go:1018] Endpoint information not available for service kube-system/metrics-server:. Not applying any policy
173461 May 09 18:41 Information kube-proxy 0 E0509 18:41:43.962481 3176 proxier.go:1034] Policy creation failed: hnsCall failed in Win32: The provided policy configuration is invalid or missing parameters. (0x803b000d)
173460 May 09 18:41 Information kube-proxy 0 I0509 18:41:43.940283 3176 proxier.go:117] Hns Endpoint resource, {"ID":"27D1AB6B-15E0-4F9E-9F34-0AEABDE433FB","Name":"Ethernet","VirtualNetwork":"29A3E292-5591-4E7A-AE21-C7F2239149E5","VirtualNetworkName":...
173459 May 09 18:41 Information kube-proxy 0 E0509 18:41:43.937364 3176 proxier.go:1034] Policy creation failed: hnsCall failed in Win32: The provided policy configuration is invalid or missing parameters. (0x803b000d)
173458 May 09 18:41 Information kube-proxy 0 I0509 18:41:43.916844 3176 proxier.go:117] Hns Endpoint resource, {"ID":"467867A6-BFFE-4192-80BF-754DD80A1F85","Name":"Ethernet","VirtualNetwork":"29A3E292-5591-4E7A-AE21-C7F2239149E5","VirtualNetworkName":...
173457 May 09 18:41 Information kubelet 0 I0509 18:41:41.608297 3284 reconciler.go:301] Volume detached for volume "default-token-6jxv2" (UniqueName: "kubernetes.io/secret/967c44d6-7240-11e9-872e-0a1a43ca23d2-default-token-6jxv2") on node "ip-10-0-...
173456 May 09 18:41 Information kubelet 0 I0509 18:41:41.550567 3284 fake_cpu_manager.go:40] [fake cpumanager] AddContainer (pod: enginebackends-xvirus-58b9f76dcf-ffvjf, container: xvirus, container id: cb1c44c1c15ad37abb2af782c9f5c86b9fca10bed796b...
173455 May 09 18:41 Information kubelet 0 I0509 18:41:41.509756 3284 operation_generator.go:688] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/967c44d6-7240-11e9-872e-0a1a43ca23d2-default-token-6jxv2" (OuterVolumeSpecName: "defa...
173454 May 09 18:41 Information kubelet 0 I0509 18:41:41.507966 3284 reconciler.go:181] operationExecutor.UnmountVolume started for volume "default-token-6jxv2" (UniqueName: "kubernetes.io/secret/967c44d6-7240-11e9-872e-0a1a43ca23d2-default-token-6...
173453 May 09 18:41 Information kubelet 0 W0509 18:41:40.390523 3284 pod_container_deletor.go:75] Container "fc66311b306448e4a82e5fae0a736174e6db47a9ae07ccc284ba84bb952fab2d" not found in pod's containers
173452 May 09 18:41 Information kubelet 0 E0509 18:41:40.389873 3284 docker_sandbox.go:683] ResolvConfPath is empty.
173451 May 09 18:41 Information kubelet 0 I0509 18:41:40.353636 3284 fake_cpu_manager.go:45] [fake cpumanager] RemoveContainer (container id: e19f978e4b01a6623edefdd298a89cc458e36c1e1fe4c1a11c5bf2784d9994cc)
173450 May 09 18:41 Information kubelet 0 W0509 18:41:40.346631 3284 docker_sandbox.go:372] failed to read pod IP from plugin/docker: NetworkPlugin cni failed on the status hook for pod "enginebackends-xvirus-58b9f76dcf-4q85t_default": failed to fi...
173449 May 09 18:41 Information kubelet 0 E0509 18:41:40.346631 3284 cni_windows.go:49] error while adding to cni network: failed to find endpoint cid-6465a0fa29446f9b9591419bf5aa52a863675c4c7567871805ce222a59495e07: Endpoint cid-6465a0fa29446f9b95...
173448 May 09 18:41 Information kubelet 0 E0509 18:41:40.346631 3284 cni.go:260] Error adding network: failed to find endpoint cid-6465a0fa29446f9b9591419bf5aa52a863675c4c7567871805ce222a59495e07: Endpoint cid-6465a0fa29446f9b9591419bf5aa52a863675c...
173447 May 09 18:41 Information kubelet 0 E0509 18:41:39.397350 3284 kuberuntime_gc.go:153] Failed to stop sandbox "c23e115935ff971dc2a2e0b186102c06e9ee630e51a561c8a7001def11f85d4d" before removing: rpc error: code = Unknown desc = NetworkPlugin cn...
173446 May 09 18:41 Information kubelet 0 E0509 18:41:39.397350 3284 remote_runtime.go:115] StopPodSandbox "c23e115935ff971dc2a2e0b186102c06e9ee630e51a561c8a7001def11f85d4d" from runtime service failed: rpc error: code = Unknown desc = NetworkPlugi...
173445 May 09 18:41 Information kubelet 0 E0509 18:41:39.396701 3284 cni.go:280] Error deleting network: failed to parse Kubernetes args: failed to get pod enginebackends-qihoo-8bfc9c5fd-9rfwx: pods "enginebackends-qihoo-8bfc9c5fd-9rfwx" not found
173444 May 09 18:41 Information kubelet 0 E0509 18:41:39.253896 3284 kuberuntime_gc.go:153] Failed to stop sandbox "a98967580e663cd969cab97c6a243ecceec3934ce7e2ecc36b75541de86f81cf" before removing: rpc error: code = Unknown desc = NetworkPlugin cn...
173443 May 09 18:41 Information kubelet 0 E0509 18:41:39.253896 3284 remote_runtime.go:115] StopPodSandbox "a98967580e663cd969cab97c6a243ecceec3934ce7e2ecc36b75541de86f81cf" from runtime service failed: rpc error: code = Unknown desc = NetworkPlugi...
173442 May 09 18:41 Information kubelet 0 E0509 18:41:39.253236 3284 cni.go:280] Error deleting network: failed to parse Kubernetes args: failed to get pod enginebackends-tachyon-76d66b9cdf-nrwk2: pods "enginebackends-tachyon-76d66b9cdf-nrwk2" not ...
173441 May 09 18:41 Information kubelet 0 I0509 18:41:39.207276 3284 operation_generator.go:558] MountVolume.SetUp succeeded for volume "default-token-6jxv2" (UniqueName: "kubernetes.io/secret/14a8c72a-728a-11e9-872e-0a1a43ca23d2-default-token-6jxv...
173440 May 09 18:41 Information kubelet 0 I0509 18:41:39.194449 3284 reconciler.go:252] operationExecutor.MountVolume started for volume "default-token-6jxv2" (UniqueName: "kubernetes.io/secret/14a8c72a-728a-11e9-872e-0a1a43ca23d2-default-token-6jx...
173439 May 09 18:41 Information kubelet 0 E0509 18:41:39.111425 3284 kuberuntime_gc.go:153] Failed to stop sandbox "c6f2471a90f397f8bd4cae22d5db3412ad3e04fe96fbba4cfefcca9baa1ee229" before removing: rpc error: code = Unknown desc = NetworkPlugin cn...
173438 May 09 18:41 Information kubelet 0 E0509 18:41:39.111425 3284 remote_runtime.go:115] StopPodSandbox "c6f2471a90f397f8bd4cae22d5db3412ad3e04fe96fbba4cfefcca9baa1ee229" from runtime service failed: rpc error: code = Unknown desc = NetworkPlugi...
173437 May 09 18:41 Information kubelet 0 E0509 18:41:39.111425 3284 cni.go:280] Error deleting network: failed to parse Kubernetes args: failed to get pod enginebackends-alibaba-56ddf976cd-8wqtb: pods "enginebackends-alibaba-56ddf976cd-8wqtb" not ...
173436 May 09 18:41 Information kubelet 0 I0509 18:41:39.094047 3284 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-6jxv2" (UniqueName: "kubernetes.io/secret/14a8c72a-728a-11e9-872e-0a1a43ca23d...
173435 May 09 18:41 Information kubelet 0 E0509 18:41:38.960097 3284 kuberuntime_gc.go:153] Failed to stop sandbox "dffa7a74bb72886492feb544e015d2684fd08a2ff89ae25e6af0d4825acaa33c" before removing: rpc error: code = Unknown desc = NetworkPlugin cn...
173434 May 09 18:41 Information kubelet 0 E0509 18:41:38.960097 3284 remote_runtime.go:115] StopPodSandbox "dffa7a74bb72886492feb544e015d2684fd08a2ff89ae25e6af0d4825acaa33c" from runtime service failed: rpc error: code = Unknown desc = NetworkPlugi...
173433 May 09 18:41 Information kubelet 0 E0509 18:41:38.960097 3284 cni.go:280] Error deleting network: failed to parse Kubernetes args: failed to get pod enginebackends-alibaba-56ddf976cd-pqznd: pods "enginebackends-alibaba-56ddf976cd-pqznd" not ...
173432 May 09 18:41 Information kubelet 0 I0509 18:41:38.885920 3284 fake_cpu_manager.go:45] [fake cpumanager] RemoveContainer (container id: e19f978e4b01a6623edefdd298a89cc458e36c1e1fe4c1a11c5bf2784d9994cc)
173431 May 09 18:41 Information kubelet 0 E0509 18:41:38.818291 3284 kuberuntime_gc.go:153] Failed to stop sandbox "dd8f3aa94396cc3a5e3bebc814ddedc5bbe9ad7f8b3cf235bbe2f884a7bc6faf" before removing: rpc error: code = Unknown desc = NetworkPlugin cn...
173430 May 09 18:41 Information kubelet 0 E0509 18:41:38.818291 3284 remote_runtime.go:115] StopPodSandbox "dd8f3aa94396cc3a5e3bebc814ddedc5bbe9ad7f8b3cf235bbe2f884a7bc6faf" from runtime service failed: rpc error: code = Unknown desc = NetworkPlugi...
173429 May 09 18:41 Information kubelet 0 E0509 18:41:38.817647 3284 cni.go:280] Error deleting network: failed to parse Kubernetes args: failed to get pod enginebackends-alibaba-56ddf976cd-ntwkv: pods "enginebackends-alibaba-56ddf976cd-ntwkv" not ...
173428 May 09 18:41 Information kubelet 0 E0509 18:41:38.674979 3284 kuberuntime_gc.go:153] Failed to stop sandbox "f7f3d5ce1f11e66390b26b9c72e00b969e7fd1e943afab8b610784dc76aa483d" before removing: rpc error: code = Unknown desc = NetworkPlugin cn...
173427 May 09 18:41 Information kubelet 0 E0509 18:41:38.674979 3284 remote_runtime.go:115] StopPodSandbox "f7f3d5ce1f11e66390b26b9c72e00b969e7fd1e943afab8b610784dc76aa483d" from runtime service failed: rpc error: code = Unknown desc = NetworkPlugi...
173426 May 09 18:41 Information kubelet 0 E0509 18:41:38.674979 3284 cni.go:280] Error deleting network: failed to parse Kubernetes args: failed to get pod enginebackends-qihoo-8bfc9c5fd-fhbkt: pods "enginebackends-qihoo-8bfc9c5fd-fhbkt" not found
173425 May 09 18:41 Information kubelet 0 W0509 18:41:27.699696 3284 helpers.go:808] eviction manager: no observation found for eviction signal nodefs.inodesFree
173424 May 09 18:41 Information kubelet 0 E0509 18:41:27.699696 3284 helpers.go:735] eviction manager: failed to construct signal: "allocatableMemory.available" error: system container "pods" not found in metrics
173423 May 09 18:41 Information kubelet 0 E0509 18:41:21.226664 3284 kubelet_network.go:102] Failed to ensure that nat chain KUBE-MARK-DROP exists: error creating chain "KUBE-MARK-DROP": executable file not found in %PATH%:
173422 May 09 18:41 Information kube-proxy 0 E0509 18:41:13.914383 3176 proxier.go:1034] Policy creation failed: hnsCall failed in Win32: The provided policy configuration is invalid or missing parameters. (0x803b000d)
173421 May 09 18:41 Information kube-proxy 0 I0509 18:41:13.895847 3176 proxier.go:117] Hns Endpoint resource, {"ID":"C87770E4-58FD-4AAF-8D8B-009685939A3A","Name":"Ethernet","VirtualNetwork":"29A3E292-5591-4E7A-AE21-C7F2239149E5","VirtualNetworkName":...
173420 May 09 18:41 Information kube-proxy 0 I0509 18:41:13.894060 3176 proxier.go:117] Hns Endpoint resource, {"ID":"764B40FC-34A3-4F47-9774-E646E7246026","Name":"Ethernet","VirtualNetwork":"29A3E292-5591-4E7A-AE21-C7F2239149E5","VirtualNetworkName":...
173419 May 09 18:41 Information kube-proxy 0 I0509 18:41:13.891139 3176 proxier.go:117] Hns Endpoint resource, {"ID":"883906FE-7896-44F0-BF73-DD540014ED6C","Name":"Ethernet","VirtualNetwork":"29A3E292-5591-4E7A-AE21-C7F2239149E5","VirtualNetworkName":...
173418 May 09 18:41 Information kube-proxy 0 E0509 18:41:13.888886 3176 proxier.go:1034] Policy creation failed: hnsCall failed in Win32: The provided policy configuration is invalid or missing parameters. (0x803b000d)
173417 May 09 18:41 Information kube-proxy 0 I0509 18:41:13.870332 3176 proxier.go:117] Hns Endpoint resource, {"ID":"594B8EC9-521D-4287-A8BC-1F82C42749DF","Name":"Ethernet","VirtualNetwork":"29A3E292-5591-4E7A-AE21-C7F2239149E5","VirtualNetworkName":...
173416 May 09 18:41 Information kube-proxy 0 E0509 18:41:13.867445 3176 proxier.go:1034] Policy creation failed: hnsCall failed in Win32: The provided policy configuration is invalid or missing parameters. (0x803b000d)
173415 May 09 18:41 Information kube-proxy 0 I0509 18:41:13.848821 3176 proxier.go:117] Hns Endpoint resource, {"ID":"467867A6-BFFE-4192-80BF-754DD80A1F85","Name":"Ethernet","VirtualNetwork":"29A3E292-5591-4E7A-AE21-C7F2239149E5","VirtualNetworkName":...
173414 May 09 18:41 Information kube-proxy 0 E0509 18:41:13.847060 3176 proxier.go:1034] Policy creation failed: hnsCall failed in Win32: The provided policy configuration is invalid or missing parameters. (0x803b000d)
173413 May 09 18:41 Information kube-proxy 0 I0509 18:41:13.828479 3176 proxier.go:117] Hns Endpoint resource, {"ID":"D432DD32-AB9A-4F3D-B7CC-52B4467F3E33","Name":"Ethernet","VirtualNetwork":"29A3E292-5591-4E7A-AE21-C7F2239149E5","VirtualNetworkName":...
173412 May 09 18:41 Information kube-proxy 0 E0509 18:41:13.825571 3176 proxier.go:1034] Policy creation failed: hnsCall failed in Win32: The provided policy configuration is invalid or missing parameters. (0x803b000d)
173411 May 09 18:41 Information kube-proxy 0 I0509 18:41:13.806967 3176 proxier.go:117] Hns Endpoint resource, {"ID":"880E24B3-EE27-41EC-9FAD-399A37C44550","Name":"Ethernet","VirtualNetwork":"29A3E292-5591-4E7A-AE21-C7F2239149E5","VirtualNetworkName":...
173410 May 09 18:41 Information kube-proxy 0 E0509 18:41:13.804062 3176 proxier.go:1034] Policy creation failed: hnsCall failed in Win32: The provided policy configuration is invalid or missing parameters. (0x803b000d)
173409 May 09 18:41 Information kube-proxy 0 I0509 18:41:13.783683 3176 proxier.go:117] Hns Endpoint resource, {"ID":"27D1AB6B-15E0-4F9E-9F34-0AEABDE433FB","Name":"Ethernet","VirtualNetwork":"29A3E292-5591-4E7A-AE21-C7F2239149E5","VirtualNetworkName":...
173408 May 09 18:41 Information kube-proxy 0 E0509 18:41:13.780771 3176 proxier.go:1018] Endpoint information not available for service kube-system/metrics-server:. Not applying any policy
173407 May 09 18:41 Information kube-proxy 0 E0509 18:41:13.780771 3176 proxier.go:1034] Policy creation failed: hnsCall failed in Win32: The provided policy configuration is invalid or missing parameters. (0x803b000d)
173406 May 09 18:41 Information kube-proxy 0 I0509 18:41:13.763328 3176 proxier.go:117] Hns Endpoint resource, {"ID":"467867A6-BFFE-4192-80BF-754DD80A1F85","Name":"Ethernet","VirtualNetwork":"29A3E292-5591-4E7A-AE21-C7F2239149E5","VirtualNetworkName":...
173405 May 09 18:41 Information kube-proxy 0 E0509 18:41:13.761070 3176 proxier.go:1034] Policy creation failed: hnsCall failed in Win32: The provided policy configuration is invalid or missing parameters. (0x803b000d)
173404 May 09 18:41 Information kube-proxy 0 I0509 18:41:13.739593 3176 proxier.go:117] Hns Endpoint resource, {"ID":"839379E0-CD60-4655-8D09-4AB586758A07","Name":"Ethernet","VirtualNetwork":"29A3E292-5591-4E7A-AE21-C7F2239149E5","VirtualNetworkName":...
173403 May 09 18:41 Information kube-proxy 0 E0509 18:41:13.737803 3176 proxier.go:1034] Policy creation failed: hnsCall failed in Win32: The provided policy configuration is invalid or missing parameters. (0x803b000d)
173402 May 09 18:41 Information kube-proxy 0 I0509 18:41:13.718089 3176 proxier.go:117] Hns Endpoint resource, {"ID":"1D604E29-0183-4504-BE73-F1D458C19718","Name":"Ethernet","VirtualNetwork":"29A3E292-5591-4E7A-AE21-C7F2239149E5","VirtualNetworkName":...
173401 May 09 18:41 Information kube-proxy 0 E0509 18:41:13.716311 3176 proxier.go:1034] Policy creation failed: hnsCall failed in Win32: The provided policy configuration is invalid or missing parameters. (0x803b000d)
173400 May 09 18:41 Information kube-proxy 0 I0509 18:41:13.696612 3176 proxier.go:117] Hns Endpoint resource, {"ID":"467867A6-BFFE-4192-80BF-754DD80A1F85","Name":"Ethernet","VirtualNetwork":"29A3E292-5591-4E7A-AE21-C7F2239149E5","VirtualNetworkName":...
173399 May 09 18:41 Information kube-proxy 0 E0509 18:41:13.694826 3176 proxier.go:1018] Endpoint information not available for service default/windows-server-iis-service:. Not applying any policy
173398 May 09 18:41 Information kube-proxy 0 E0509 18:41:13.694826 3176 proxier.go:1034] Policy creation failed: hnsCall failed in Win32: The provided policy configuration is invalid or missing parameters. (0x803b000d)
173397 May 09 18:41 Information kube-proxy 0 I0509 18:41:13.675124 3176 proxier.go:117] Hns Endpoint resource, {"ID":"39D2AF1A-FD6D-4DB3-9CFF-7F48750725BD","Name":"Ethernet","VirtualNetwork":"29A3E292-5591-4E7A-AE21-C7F2239149E5","VirtualNetworkName":...
173396 May 09 18:41 Information kube-proxy 0 I0509 18:41:13.673349 3176 proxier.go:117] Hns Endpoint resource, {"ID":"83146751-A52B-40BD-92DB-627D781527C3","Name":"Ethernet","VirtualNetwork":"29A3E292-5591-4E7A-AE21-C7F2239149E5","VirtualNetworkName":...
173395 May 09 18:41 Information kube-proxy 0 I0509 18:41:13.671566 3176 proxier.go:117] Hns Endpoint resource, {"ID":"B994CB85-C1B0-40E9-9981-21814900C1D7","Name":"Ethernet","VirtualNetwork":"29A3E292-5591-4E7A-AE21-C7F2239149E5","VirtualNetworkName":...
173394 May 09 18:41 Information kube-proxy 0 E0509 18:41:13.668184 3176 proxier.go:1034] Policy creation failed: hnsCall failed in Win32: The provided policy configuration is invalid or missing parameters. (0x803b000d)
173393 May 09 18:41 Information kube-proxy 0 I0509 18:41:13.648946 3176 proxier.go:117] Hns Endpoint resource, {"ID":"B8FACC93-B15A-4FE8-92A6-4E4D25106B69","Name":"Ethernet","VirtualNetwork":"29A3E292-5591-4E7A-AE21-C7F2239149E5","VirtualNetworkName":...
173392 May 09 18:41 Information kube-proxy 0 E0509 18:41:13.646698 3176 proxier.go:1034] Policy creation failed: hnsCall failed in Win32: The provided policy configuration is invalid or missing parameters. (0x803b000d)
173391 May 09 18:41 Information kube-proxy 0 I0509 18:41:13.629308 3176 proxier.go:117] Hns Endpoint resource, {"ID":"D6AF99F5-9524-4B35-8AF6-8F89E54F6D69","Name":"Ethernet","VirtualNetwork":"29A3E292-5591-4E7A-AE21-C7F2239149E5","VirtualNetworkName":...
173390 May 09 18:41 Information kube-proxy 0 E0509 18:41:13.627543 3176 proxier.go:1034] Policy creation failed: hnsCall failed in Win32: The provided policy configuration is invalid or missing parameters. (0x803b000d)
173389 May 09 18:41 Information kube-proxy 0 I0509 18:41:13.609680 3176 proxier.go:117] Hns Endpoint resource, {"ID":"6BB32B45-751F-4084-829B-D02B294F5A76","Name":"Ethernet","VirtualNetwork":"29A3E292-5591-4E7A-AE21-C7F2239149E5","VirtualNetworkName":...
173388 May 09 18:41 Information kube-proxy 0 E0509 18:41:13.607904 3176 proxier.go:1034] Policy creation failed: hnsCall failed in Win32: The provided policy configuration is invalid or missing parameters. (0x803b000d)
173387 May 09 18:41 Information kube-proxy 0 I0509 18:41:13.587585 3176 proxier.go:117] Hns Endpoint resource, {"ID":"2E2C33CB-EF48-48C0-BD3B-F1FEC06B913E","Name":"Ethernet","VirtualNetwork":"29A3E292-5591-4E7A-AE21-C7F2239149E5","VirtualNetworkName":...
173386 May 09 18:41 Information kubelet 0 W0509 18:41:00.478804 3284 helpers.go:808] eviction manager: no observation found for eviction signal nodefs.inodesFree
173385 May 09 18:41 Information kubelet 0 E0509 18:41:00.478804 3284 helpers.go:735] eviction manager: failed to construct signal: "allocatableMemory.available" error: system container "pods" not found in metrics
173384 May 09 18:40 Information kube-proxy 0 E0509 18:40:43.584820 3176 proxier.go:1034] Policy creation failed: hnsCall failed in Win32: The provided policy configuration is invalid or missing parameters. (0x803b000d)
VPC-RESOURCE-CONTROLLER Logs:
I0509 18:41:38.929015 1 watcher.go:194] Pod watcher processing pod enginebackends-xvirus-58b9f76dcf-ffvjf on node ip-10-0-2-237.ec2.internal.
I0509 18:41:38.929065 1 manager.go:234] Node manager allocated resource vpc.amazonaws.com/PrivateIPv4Address 10.0.2.137 for pod enginebackends-xvirus-58b9f76dcf-ffvjf on node ip-10-0-2-237.ec2.internal.
I0509 18:41:38.929081 1 watcher.go:295] Pod watcher annotating pod enginebackends-xvirus-58b9f76dcf-ffvjf with resource vpc.amazonaws.com/PrivateIPv4Address 10.0.2.137.
I0509 18:41:38.929269 1 reconciler.go:102] Reconciler worker 1 starting processing node ip-10-0-2-237.ec2.internal.
I0509 18:41:38.929292 1 reconciler.go:123] Reconciler checking resource vpc.amazonaws.com/ENI warmpool size 0 desired 0 on node ip-10-0-2-237.ec2.internal.
I0509 18:41:38.929303 1 reconciler.go:123] Reconciler checking resource vpc.amazonaws.com/PrivateIPv4Address warmpool size 1 desired 3 on node ip-10-0-2-237.ec2.internal.
I0509 18:41:38.929311 1 reconciler.go:150] Reconciler creating resource vpc.amazonaws.com/PrivateIPv4Address on node ip-10-0-2-237.ec2.internal.
I0509 18:41:38.949664 1 watcher.go:236] Pod watcher completed processing pod enginebackends-xvirus-58b9f76dcf-ffvjf.
I0509 18:41:39.047710 1 eniwrapper.go:186] Assigning 2 private ipv4 addresses on network interface: eni-0df50dc0026b34dfe.
I0509 18:41:39.749089 1 ipaddress.go:177] IPAddressProvider allocated IP addresses [10.0.2.131 10.0.2.136] on eni-0df50dc0026b34dfe.
I0509 18:41:39.749116 1 reconciler.go:106] Reconciler worker 1 completed processing node ip-10-0-2-237.ec2.internal.
I0509 18:41:44.008896 1 watcher.go:247] Pod watcher processing deleted pod enginebackends-xvirus-58b9f76dcf-4q85t on node ip-10-0-2-237.ec2.internal.
I0509 18:41:44.008948 1 manager.go:279] Node manager freed resource vpc.amazonaws.com/PrivateIPv4Address 10.0.2.247 on node ip-10-0-2-237.ec2.internal for pod enginebackends-xvirus-58b9f76dcf-4q85t, pool &{Capacity:14 InUse:map[10.0.2.240:enginebackends-nanoav-6448ccbfc8-794qb 10.0.2.237:node 10.0.2.191:enginebackends-qihoo-8bfc9c5fd-l5jfq 10.0.2.88:enginebackends-tachyon-76d66b9cdf-c96ww 10.0.2.38:enginebackends-k7-86cffc75c6-45wrx 10.0.2.55:enginebackends-lionic-5dbc9d5b88-dj6xx 10.0.2.239:enginebackends-alibaba-56ddf976cd-2mv47 10.0.2.23:enginebackends-jiangmin-5f5f87b647-jhcbd 10.0.2.137:enginebackends-xvirus-58b9f76dcf-ffvjf] Warm:[10.0.2.182 10.0.2.131 10.0.2.136 10.0.2.247] Pending:0}.
I0509 18:41:44.009008 1 reconciler.go:102] Reconciler worker 1 starting processing node ip-10-0-2-237.ec2.internal.
I0509 18:41:44.009027 1 reconciler.go:123] Reconciler checking resource vpc.amazonaws.com/ENI warmpool size 0 desired 0 on node ip-10-0-2-237.ec2.internal.
I0509 18:41:44.009036 1 reconciler.go:123] Reconciler checking resource vpc.amazonaws.com/PrivateIPv4Address warmpool size 4 desired 3 on node ip-10-0-2-237.ec2.internal.
I0509 18:41:44.009044 1 reconciler.go:106] Reconciler worker 1 completed processing node ip-10-0-2-237.ec2.internal.
POD Logs:
{"level": "INFO", "name": "worker", "timestamp": "2019-05-09T18:41:44.910718Z", "message": "Logging in JSON format."}
{"level": "INFO", "name": "microengine", "timestamp": "2019-05-09T18:41:44.911844Z", "message": "Logging in JSON format."}
{"level": "INFO", "name": "polyswarm_xvirus", "timestamp": "2019-05-09T18:41:44.911844Z", "message": "Logging in JSON format."}
{"level": "INFO", "name": "worker.__main__", "timestamp": "2019-05-09T18:41:44.911844Z", "message": "Running worker with 8 tasks"}
{"level": "ERROR", "name": "worker", "timestamp": "2019-05-09T18:41:56.944583Z", "message": "Unhandled exception at top level", "exc_info": "Traceback (most recent call last):\n File \"c:\\python36\\lib\\site-packages\\worker\\__init__.py\", line 53, in run\n for i in range(self.task_count)]))\n File \"c:\\python36\\lib\\asyncio\\base_events.py\", line 473, in run_until_complete\n return future.result()\n File \"c:\\python36\\lib\\site-packages\\worker\\__init__.py\", line 79, in run_task\n redis = await aioredis.create_redis_pool(self.redis_uri)\n File \"c:\\python36\\lib\\site-packages\\aioredis\\commands\\__init__.py\", line 201, in create_redis_pool\n loop=loop)\n File \"c:\\python36\\lib\\site-packages\\aioredis\\pool.py\", line 56, in create_pool\n await pool._fill_free(override_min=False)\n File \"c:\\python36\\lib\\site-packages\\aioredis\\pool.py\", line 388, in _fill_free\n conn = await self._create_new_connection(self._address)\n File \"c:\\python36\\lib\\site-packages\\aioredis\\connection.py\", line 108, in create_connection\n timeout, loop=loop)\n File \"c:\\python36\\lib\\asyncio\\tasks.py\", line 339, in wait_for\n return (yield from fut)\n File \"c:\\python36\\lib\\site-packages\\aioredis\\stream.py\", line 19, in open_connection\n lambda: protocol, host, port, **kwds)\n File \"c:\\python36\\lib\\asyncio\\base_events.py\", line 739, in create_connection\n infos = f1.result()\n File \"c:\\python36\\lib\\concurrent\\futures\\thread.py\", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File \"c:\\python36\\lib\\socket.py\", line 745, in getaddrinfo\n for res in _socket.getaddrinfo(host, port, family, type, proto, flags):\nsocket.gaierror: [Errno 11001] getaddrinfo failed"}
scsich
commented
5 years ago So a bit more on this issue:
Windows EKS host has the following in vpc-shared-eni log.
2019-05-14T14:28:34Z [ERROR] Failed to parse netconfig from args: failed to parse Kubernetes args: pod does not have label vpc.amazonaws.com/PrivateIPv4Address.
2019-05-14T14:28:34Z [ERROR] CNI command failed: failed to parse Kubernetes args: pod does not have label vpc.amazonaws.com/PrivateIPv4Address
2019-05-14T14:28:47Z [INFO] Plugin vpc-shared-eni version executing CNI command.
2019-05-14T14:28:47Z [INFO] Waiting for pod label %s.vpc.amazonaws.com/PrivateIPv4Address
<snip repeats 20x>
2019-05-14T14:28:49Z [INFO] Waiting for pod label %s.vpc.amazonaws.com/PrivateIPv4Address
2019-05-14T14:28:49Z [ERROR] Failed to parse netconfig from args: failed to parse Kubernetes args: pod does not have label vpc.amazonaws.com/PrivateIPv4Address.
2019-05-14T14:28:49Z [ERROR] CNI command failed: failed to parse Kubernetes args: pod does not have label vpc.amazonaws.com/PrivateIPv4Address
2019-05-14T14:29:01Z [INFO] Plugin vpc-shared-eni version executing CNI command.
2019-05-14T14:29:01Z [INFO] Waiting for pod label %s.vpc.amazonaws.com/PrivateIPv4Address
<snip repeats alot>
2019-05-14T14:29:03Z [INFO] Waiting for pod label %s.vpc.amazonaws.com/PrivateIPv4Address
2019-05-14T14:29:03Z [ERROR] Failed to parse netconfig from args: failed to parse Kubernetes args: pod does not have label vpc.amazonaws.com/PrivateIPv4Address.
2019-05-14T14:29:03Z [ERROR] CNI command failed: failed to parse Kubernetes args: pod does not have label vpc.amazonaws.com/PrivateIPv4AddressExecing into pod:
PS C:\microengine> nslookup.exe google.com 8.8.8.8
Server: google-public-dns-a.google.com
Address: 8.8.8.8
Non-authoritative answer:
Name: google.com
Addresses: 2607:f8b0:4004:811::200e
172.217.7.174
PS C:\microengine> ipconfig /all
Windows IP Configuration
Host Name . . . . . . . . . . . . : enginebackends-tachyon-69dcccc6f7-qsvlg
Primary Dns Suffix . . . . . . . :
Node Type . . . . . . . . . . . . : Hybrid
IP Routing Enabled. . . . . . . . : No
WINS Proxy Enabled. . . . . . . . : No
Ethernet adapter vEthernet (cid-f5926b00d7541497b6a43c9173d0b8c7b8ef5cf227ffd82e5661eb81009610f2):
Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : Hyper-V Virtual Ethernet Adapter #4
Physical Address. . . . . . . . . : 00-15-5D-4F-93-55
DHCP Enabled. . . . . . . . . . . : Yes
Autoconfiguration Enabled . . . . : Yes
Link-local IPv6 Address . . . . . : fe80::c8dc:1d24:ca61:f24e%29(Preferred)
IPv4 Address. . . . . . . . . . . : 10.0.2.77(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 10.0.2.1
DNS Servers . . . . . . . . . . . : 172.20.0.10
NetBIOS over Tcpip. . . . . . . . : Disabled
PS C:\microengine> nslookup.exe google.com
DNS request timed out.
timeout was 2 seconds.
Server: UnKnown
Address: 172.20.0.10
DNS request timed out.
timeout was 2 seconds.
DNS request timed out.
timeout was 2 seconds.
DNS request timed out.
timeout was 2 seconds.
DNS request timed out.
timeout was 2 seconds.
*** Request to UnKnown timed-out
PS C:\microengine> ping 10.0.2.1
Pinging 10.0.2.1 with 32 bytes of data:
Reply from 10.0.2.1: bytes=32 time<1ms TTL=128
Reply from 10.0.2.1: bytes=32 time<1ms TTL=128
Reply from 10.0.2.1: bytes=32 time<1ms TTL=128
Reply from 10.0.2.1: bytes=32 time<1ms TTL=128
Ping statistics for 10.0.2.1:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms
PS C:\microengine>
Describing pod, it def has IPv4 Labels:
Name: enginebackends-tachyon-69dcccc6f7-qsvlg
Namespace: default
Priority: 0
PriorityClassName: <none>
Node: ip-10-0-2-90.ec2.internal/10.0.2.90
Start Time: Tue, 14 May 2019 10:23:06 -0400
Labels: app.kubernetes.io/instance=enginebackends
app.kubernetes.io/name=tachyon
pod-template-hash=2587777293
vpc.amazonaws.com/PrivateIPv4Address=10.0.2.77
Annotations: vpc.amazonaws.com/AdmissionWebhookStatus: injected
vpc.amazonaws.com/AdmissionWebhookVersion: beta
vpc.amazonaws.com/PrivateIPv4Address: 10.0.2.77
vpc.amazonaws.com/PrivateIPv4AddressRequestCount: 1
Status: Running
IP: 10.0.2.77
Controlled By: ReplicaSet/enginebackends-tachyon-69dcccc6f7
Ensured that:
@somujay this is a drilldown on what @aarongooch is experiencing in our org.
Thanks!
somujay
commented
5 years ago So a bit more on this issue:
- Some pods, after deletion, get a VPC IP address (10.0.x.x)
- These pods can't talk to kube dns at 172.20.0.10
- DNS is set as far as pod is concerned
- Can hit 10. gateway and external DNS (e.g. 8.8.8.8)
Windows EKS host has the following in vpc-shared-eni log.
2019-05-14T14:28:34Z [ERROR] Failed to parse netconfig from args: failed to parse Kubernetes args: pod does not have label vpc.amazonaws.com/PrivateIPv4Address. 2019-05-14T14:28:34Z [ERROR] CNI command failed: failed to parse Kubernetes args: pod does not have label vpc.amazonaws.com/PrivateIPv4Address 2019-05-14T14:28:47Z [INFO] Plugin vpc-shared-eni version executing CNI command. 2019-05-14T14:28:47Z [INFO] Waiting for pod label %s.vpc.amazonaws.com/PrivateIPv4Address <snip repeats 20x> 2019-05-14T14:28:49Z [INFO] Waiting for pod label %s.vpc.amazonaws.com/PrivateIPv4Address 2019-05-14T14:28:49Z [ERROR] Failed to parse netconfig from args: failed to parse Kubernetes args: pod does not have label vpc.amazonaws.com/PrivateIPv4Address. 2019-05-14T14:28:49Z [ERROR] CNI command failed: failed to parse Kubernetes args: pod does not have label vpc.amazonaws.com/PrivateIPv4Address 2019-05-14T14:29:01Z [INFO] Plugin vpc-shared-eni version executing CNI command. 2019-05-14T14:29:01Z [INFO] Waiting for pod label %s.vpc.amazonaws.com/PrivateIPv4Address <snip repeats alot> 2019-05-14T14:29:03Z [INFO] Waiting for pod label %s.vpc.amazonaws.com/PrivateIPv4Address 2019-05-14T14:29:03Z [ERROR] Failed to parse netconfig from args: failed to parse Kubernetes args: pod does not have label vpc.amazonaws.com/PrivateIPv4Address. 2019-05-14T14:29:03Z [ERROR] CNI command failed: failed to parse Kubernetes args: pod does not have label vpc.amazonaws.com/PrivateIPv4AddressExecing into pod:
PS C:\microengine> nslookup.exe google.com 8.8.8.8 Server: google-public-dns-a.google.com Address: 8.8.8.8 Non-authoritative answer: Name: google.com Addresses: 2607:f8b0:4004:811::200e 172.217.7.174 PS C:\microengine> ipconfig /all Windows IP Configuration Host Name . . . . . . . . . . . . : enginebackends-tachyon-69dcccc6f7-qsvlg Primary Dns Suffix . . . . . . . : Node Type . . . . . . . . . . . . : Hybrid IP Routing Enabled. . . . . . . . : No WINS Proxy Enabled. . . . . . . . : No Ethernet adapter vEthernet (cid-f5926b00d7541497b6a43c9173d0b8c7b8ef5cf227ffd82e5661eb81009610f2): Connection-specific DNS Suffix . : Description . . . . . . . . . . . : Hyper-V Virtual Ethernet Adapter #4 Physical Address. . . . . . . . . : 00-15-5D-4F-93-55 DHCP Enabled. . . . . . . . . . . : Yes Autoconfiguration Enabled . . . . : Yes Link-local IPv6 Address . . . . . : fe80::c8dc:1d24:ca61:f24e%29(Preferred) IPv4 Address. . . . . . . . . . . : 10.0.2.77(Preferred) Subnet Mask . . . . . . . . . . . : 255.255.255.0 Default Gateway . . . . . . . . . : 10.0.2.1 DNS Servers . . . . . . . . . . . : 172.20.0.10 NetBIOS over Tcpip. . . . . . . . : Disabled PS C:\microengine> nslookup.exe google.com DNS request timed out. timeout was 2 seconds. Server: UnKnown Address: 172.20.0.10 DNS request timed out. timeout was 2 seconds. DNS request timed out. timeout was 2 seconds. DNS request timed out. timeout was 2 seconds. DNS request timed out. timeout was 2 seconds. *** Request to UnKnown timed-out PS C:\microengine> ping 10.0.2.1 Pinging 10.0.2.1 with 32 bytes of data: Reply from 10.0.2.1: bytes=32 time<1ms TTL=128 Reply from 10.0.2.1: bytes=32 time<1ms TTL=128 Reply from 10.0.2.1: bytes=32 time<1ms TTL=128 Reply from 10.0.2.1: bytes=32 time<1ms TTL=128 Ping statistics for 10.0.2.1: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 0ms, Average = 0ms PS C:\microengine>Describing pod, it def has IPv4 Labels:
Name: enginebackends-tachyon-69dcccc6f7-qsvlg Namespace: default Priority: 0 PriorityClassName: <none> Node: ip-10-0-2-90.ec2.internal/10.0.2.90 Start Time: Tue, 14 May 2019 10:23:06 -0400 Labels: app.kubernetes.io/instance=enginebackends app.kubernetes.io/name=tachyon pod-template-hash=2587777293 vpc.amazonaws.com/PrivateIPv4Address=10.0.2.77 Annotations: vpc.amazonaws.com/AdmissionWebhookStatus: injected vpc.amazonaws.com/AdmissionWebhookVersion: beta vpc.amazonaws.com/PrivateIPv4Address: 10.0.2.77 vpc.amazonaws.com/PrivateIPv4AddressRequestCount: 1 Status: Running IP: 10.0.2.77 Controlled By: ReplicaSet/enginebackends-tachyon-69dcccc6f7Ensured that:
- eks:kube-proxy-windows is in mapRoles against instance role ARN
- Policies listed in cloudformation are applied to node instance role
- resource controller and admission webhook are operating
@somujay this is a drilldown on what @aarongooch is experiencing in our org.
Thanks!
This is interesting. I assume this happens randomly?? Did you check kube-proxy is running on this host machine? "Get-Service Kube-proxy" is the command to check the status. if it's running, run this command "Get-HNSPolicyList | ConvertTo-Json -Depth 3" to see it has the policy for DNS cluster IP. If you still has the problem, try restarting kube-proxy on host machine (restart-service kube-proxy). If the problem still persists then mail us, we need to set up a call to look into this issue.
pdefreitas
commented
5 years ago In my case adding a SecurityGroupEgress allowing access from the control panel to the worker nodes security group on port 443 solved the issue.
Edit: but now I do face the issue that JasonChinsen is facing. From running N number of nodes I do only get a single node with one private address available to be used.
bcmedeiros
commented
5 years ago Not sure if this is still somehow expected to happen, but I'm having this same error after the official windows support release.
I have just created a new cluster with eksctl, created the windows worker nodes group, ran the eksctl utils install-vpc-controllers --name=cluster --approve and created the default deployment suggested on the official docs.
Pod stays as pending forever:
default windows-server-iis-66bf9745b-sjsjq 0/1 Pending 0 9s
Descrive says:
21:43 $ kubectl describe pod windows-server-iis-66bf9745b-sjsjq
Name: windows-server-iis-66bf9745b-sjsjq
Namespace: default
Priority: 0
Node: <none>
Labels: app=windows-server-iis
pod-template-hash=66bf9745b
tier=backend
track=stable
Annotations: kubernetes.io/psp: eks.privileged
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/windows-server-iis-66bf9745b
Containers:
windows-server-iis:
Image: mcr.microsoft.com/windows/servercore:1809
Port: 80/TCP
Host Port: 0/TCP
Command:
powershell.exe
-command
Add-WindowsFeature Web-Server; Invoke-WebRequest -UseBasicParsing -Uri 'https://dotnetbinaries.blob.core.windows.net/servicemonitor/2.0.1.6/ServiceMonitor.exe' -OutFile 'C:\ServiceMonitor.exe'; echo '<html><body><br/><br/><marquee><H1>Hello EKS!!!<H1><marquee></body><html>' > C:\inetpub\wwwroot\default.html; C:\ServiceMonitor.exe 'w3svc';
Limits:
vpc.amazonaws.com/PrivateIPv4Address: 1
Requests:
vpc.amazonaws.com/PrivateIPv4Address: 1
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-fwkxs (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
default-token-fwkxs:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-fwkxs
Optional: false
QoS Class: BestEffort
Node-Selectors: beta.kubernetes.io/os=windows
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 18s (x5 over 6m10s) default-scheduler 0/3 nodes are available: 2 node(s) didn't match node selector, 3 Insufficient vpc.amazonaws.com/PrivateIPv4Address.Any ideas on what's happening?
aarongooch
commented
5 years ago Your pod is not being assigned an IP address. This could be due to the kube configuration, errors in the vpc-controller pod, or a lack of IPs in the subnet the ec2 node is assigned to. You're going to have to dig in to your cluster and find the issue. Too many variables in play.
bcmedeiros
commented
5 years ago @aarongooch Thanks for the reply!
As my VPC/subnets have plenty of free addresses, I checked your other suggestion and this is what I find in the vpc-resource-controller logs:
I1028 16:09:23.456376 1 watcher.go:178] Node watcher processing node ip-10-10-92-124.ec2.internal.
I1028 16:09:23.456411 1 manager.go:109] Node manager adding node ip-10-10-92-124.ec2.internal with instanceID i-0c1544862a42d8885.
E1028 16:09:23.686677 1 manager.go:126] Node manager failed to get resource vpc.amazonaws.com/CIDRBlock pool on node ip-10-10-92-124.ec2.internal: failed to find the route table for subnet subnet-12345678
E1028 16:09:23.686702 1 watcher.go:183] Node watcher failed to add node ip-10-10-92-124.ec2.internal: failed to find the route table for subnet subnet-12345678
E1028 16:09:23.686716 1 watcher.go:266] failed to find the route table for subnet subnet-12345678
W1028 16:09:23.686882 1 watcher.go:267] Node watcher dropping key "ip-10-10-92-124.ec2.internal" out of queue: failed to find the route table for subnet subnet-12345678I check my subnet in the AWS web console and the route table is there, so it must be some access control issue.
Anyone using eksctl? It's probably an issue there, as I created everything using it, no manual configs at all.
Any help is appreciated. :)
niranjan94
commented
4 years ago I experienced the same issue when trying the deploy a windows workload. The issue in my case turned out to be the node selector.
This works:
nodeSelector:
beta.kubernetes.io/arch: amd64
beta.kubernetes.io/os: windowsBut this does not:
nodeSelector:
kubernetes.io/arch: amd64
kubernetes.io/os: windows
jglick
commented
4 years ago @niranjan94 tracked as #542.
niranjan94
commented
4 years ago @jglick thanks for the update. Had not seen that. 😄
rawahars
commented
1 year ago This is an old issue but is anyone facing this with the present EKS clusters running Windows?
VPC Resource Controller now runs in control plane and is managed for the customers.
goyalpurvi
commented
1 year ago Will be closing this issue as its been stale for a while, and it doesn't seem like it should be reoccurring. Please reopen if you continue to face this issue.
Tell us about your request I am running a mixed k8s cluster based on eks-windows-preview
I am running into an issue after turning off the windows node local firewall and restarting the windows node
netsh -r computername advfirewall set privateprofile state offWhich service(s) is this request for? EKS
Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? This is a continuation of my efforts from the issue: 236 where I am trying to get my windows application to talk to MongoDB running on the Linux side of the cluster
Are you currently working around this issue? N/A
Additional context I have noticed that pods that were already alocated to a windows node before restaring would start up again once the node came online, however any new pods deployed after the reboot would not start, show pending, and have this message:
Warning FailedScheduling 2m33s (x49 over 42m) default-scheduler 0/6 nodes are available: 3 node(s) didn't match node selector, 6 Insufficient vpc.amazonaws.com/PrivateIPv4Address.I have restarted both the vpc-admission-webhook-deployment and aws-node pods
I have also checked each windows node status (could there be a missing Annotation after reboot?)
both Windows and Linux nodes are running are on version
v1.11.5Attachments If you think you might have additional information that you'd like to include via an attachment, please do - we'll take a look. (Remember to remove any personally-identifiable information.)