weaveworks / launcher

Weave Cloud Launcher
Apache License 2.0
10 stars 13 forks source link

Panic when copying stdout from kubectl #308

Closed cadedaniel closed 4 years ago

cadedaniel commented 4 years ago

I have a CI system that ran the following command:

eksctl create cluster xyz --nodes 1 --node-type=c5.xlarge --timeout=40m --region us-east-2 --auto-kubeconfig --version 1.14 --vpc-public-subnets=subnet-05cf456f7164b5bc3,subnet-0f6ee465d5b9eab20 --vpc-private-subnets=subnet-0522bc923906d6a26,subnet-0133c270976907933

It produced the following crash:

[ℹ]  eksctl version 0.11.1
[ℹ]  using region us-east-2
[✔]  using existing VPC (vpc-074a87335e1e88bbc) and subnets (private:[subnet-0133c270976907933 subnet-0522bc923906d6a26] public:[subnet-05cf456f7164b5bc3 subnet-0f6ee465d5b9eab20])
[!]  custom VPC/subnets will be used; if resulting cluster doesn't function as expected, make sure to review the configuration of VPC/subnets
[ℹ]  nodegroup "ng-6bed9233" will use "ami-082bb518441d3954c" [AmazonLinux2/1.14]
[ℹ]  using Kubernetes version 1.14
[ℹ]  creating EKS cluster "xyz" in "us-east-2" region with un-managed nodes
[ℹ]  will create 2 separate CloudFormation stacks for cluster itself and the initial nodegroup
[ℹ]  if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=us-east-2 --cluster=xyz'
[ℹ]  CloudWatch logging will not be enabled for cluster "xyz" in "us-east-2"
[ℹ]  you can enable it with 'eksctl utils update-cluster-logging --region=us-east-2 --cluster=xyz'
[ℹ]  Kubernetes API endpoint access will use default of {publicAccess=true, privateAccess=false} for cluster "xyz" in "us-east-2"
[ℹ]  2 sequential tasks: { create cluster control plane "xyz", create nodegroup "ng-6bed9233" }
[ℹ]  building cluster stack "eksctl-xyz-cluster"
[ℹ]  deploying stack "eksctl-xyz-cluster"
[ℹ]  building nodegroup stack "eksctl-xyz-nodegroup-ng-6bed9233"
[ℹ]  --nodes-min=1 was set automatically for nodegroup ng-6bed9233
[ℹ]  --nodes-max=1 was set automatically for nodegroup ng-6bed9233
[ℹ]  deploying stack "eksctl-xyz-nodegroup-ng-6bed9233"
[✔]  all EKS cluster resources for "xyz" have been created
[✔]  saved kubeconfig as "/root/.kube/eksctl/clusters/xyz"
[ℹ]  adding identity "arn:aws:iam::123456789012:role/eksctl-xyz ConfigMap
[ℹ]  nodegroup "ng-6bed9233" has 0 node(s)
[ℹ]  waiting for at least 1 node(s) to become ready in "ng-6bed9233"
[ℹ]  nodegroup "ng-6bed9233" has 1 node(s)
[ℹ]  node "ip-10-0-3-191.us-east-2.compute.internal" is ready
panic: runtime error: slice bounds out of range
goroutine 877 [running]:
bytes.(*Buffer).grow(0xc00093ad20, 0x101, 0xc0000a66d8)
    /usr/local/go/src/bytes/buffer.go:151 +0x2c9
bytes.(*Buffer).Write(0xc00093ad20, 0xc000460000, 0x101, 0x8000, 0x0, 0x0, 0x101)
    /usr/local/go/src/bytes/buffer.go:175 +0xe6
io.(*multiWriter).Write(0xc000804700, 0xc000460000, 0x101, 0x8000, 0x101, 0x0, 0x0)
    /usr/local/go/src/io/multi.go:60 +0x87
io.copyBuffer(0x3643d00, 0xc000804700, 0x3644980, 0xc00000e308, 0xc000460000, 0x8000, 0x8000, 0x31ac890, 0x0, 0x57bd460)
    /usr/local/go/src/io/io.go:404 +0x1fb
io.Copy(...)
    /usr/local/go/src/io/io.go:364
github.com/weaveworks/launcher/pkg/kubectl.outputMatrix.func1(0x3643d00, 0xc000804700, 0x3644980, 0xc00000e308)
    /go/pkg/mod/github.com/weaveworks/launcher@v0.0.0-20180711153254-f1b2830d4f2d/pkg/kubectl/local.go:61 +0x84
created by github.com/weaveworks/launcher/pkg/kubectl.outputMatrix
    /go/pkg/mod/github.com/weaveworks/launcher@v0.0.0-20180711153254-f1b2830d4f2d/pkg/kubectl/local.go:67 +0x4f6

That line in this repository is here:

        _, _ = io.Copy(dst, src)

I don't know how exactly this line was invoked, but I suspect the issue is on line 52. The docs for cmd.StdoutPipe() say it can return an error, which is ignored:

    stdoutPipe, _ := cmd.StdoutPipe()
errordeveloper commented 4 years ago

cc @martina-if