danbudris
commented
2 years ago Hi @itaytalmi have you tried this identical configuration with the 0.7.0 binary? If so, what were the results?
Closed itaytalmi closed 1 year ago
danbudris
commented
2 years ago Hi @itaytalmi have you tried this identical configuration with the 0.7.0 binary? If so, what were the results?
itaytalmi
commented
2 years ago Hi @danbudris, using the 0.7.0 binary with identical configuration, the OVA template is successfully imported to vSphere. Unfortunately using 0.7.0 I've been experiencing other issues (which I've raised here on the community as well) but the OVA part succeeds and the deployment starts.
danbudris
commented
2 years ago @itaytalmi can you re-run this with high verbosity on (-v 9 CLI arg) and let us know the results?
itaytalmi
commented
2 years ago Sure, here is the output:
$ eksctl anywhere create cluster -f eksa-mgmt-cluster.yaml -v 9
2022-02-25T19:10:07.136Z V4 Logger init completed {"vlevel": 9}
2022-02-25T19:10:07.136Z V6 Executing command {"cmd": "/usr/bin/docker version --format {{.Client.Version}}"}
2022-02-25T19:10:07.188Z V6 Executing command {"cmd": "/usr/bin/docker info --format '{{json .MemTotal}}'"}
2022-02-25T19:10:07.293Z V4 Reading releases manifest {"url": "https://anywhere-assets.eks.amazonaws.com/releases/eks-a/manifest.yaml"}
2022-02-25T19:10:07.564Z V4 Reading bundles manifest {"url": "https://anywhere-assets.eks.amazonaws.com/releases/bundles/7/manifest.yaml"}
2022-02-25T19:10:07.994Z V2 Pulling docker image {"image": "public.ecr.aws/eks-anywhere/cli-tools:v0.1.0-eks-a-7"}
2022-02-25T19:10:07.994Z V6 Executing command {"cmd": "/usr/bin/docker pull public.ecr.aws/eks-anywhere/cli-tools:v0.1.0-eks-a-7"}
2022-02-25T19:10:10.218Z V3 Initializing long running container {"name": "eksa_1645816207994777826", "image": "public.ecr.aws/eks-anywhere/cli-tools:v0.1.0-eks-a-7"}
2022-02-25T19:10:10.218Z V6 Executing command {"cmd": "/usr/bin/docker run -d --name eksa_1645816207994777826 --network host -w /home/k8s/eksa-playground/071 -v /var/run/docker.sock:/var/run/docker.sock -v /home/k8s/eksa-playground/071:/home/k8s/eksa-playground/071 --entrypoint sleep public.ecr.aws/eks-anywhere/cli-tools:v0.1.0-eks-a-7 infinity"}
2022-02-25T19:10:10.430Z V4 Task start {"task_name": "setup-validate"}
2022-02-25T19:10:10.431Z V0 Performing setup and validations
2022-02-25T19:10:10.431Z V4 Relative network path specified, using path /Demo-Datacenter/network/demo-itay-k8s-nodes
2022-02-25T19:10:10.431Z V0 Warning: VSphereDatacenterConfig configured in insecure mode
2022-02-25T19:10:10.451Z V0 ✅ Connected to server
2022-02-25T19:10:10.451Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_PASSWORD=***** -e GOVC_URL=demo-vc-01.terasky.demo -e GOVC_INSECURE=true -e GOVC_USERNAME=***** eksa_1645816207994777826 govc about -k"}
2022-02-25T19:10:10.615Z V5 Retry execution successful {"retries": 1, "duration": "164.47266ms"}
2022-02-25T19:10:10.615Z V0 ✅ Authenticated to vSphere
2022-02-25T19:10:10.615Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_PASSWORD=***** -e GOVC_URL=demo-vc-01.terasky.demo -e GOVC_INSECURE=true -e GOVC_USERNAME=***** eksa_1645816207994777826 govc datacenter.info Demo-Datacenter"}
2022-02-25T19:10:10.815Z V5 Retry execution successful {"retries": 1, "duration": "199.764044ms"}
2022-02-25T19:10:10.815Z V0 ✅ Datacenter validated
2022-02-25T19:10:10.815Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_PASSWORD=***** -e GOVC_URL=demo-vc-01.terasky.demo -e GOVC_INSECURE=true -e GOVC_USERNAME=***** eksa_1645816207994777826 govc find -maxdepth=1 /Demo-Datacenter/network -type n -name demo-itay-k8s-nodes"}
2022-02-25T19:10:10.973Z V5 Retry execution successful {"retries": 1, "duration": "157.295353ms"}
2022-02-25T19:10:10.973Z V0 ✅ Network validated
2022-02-25T19:10:10.973Z V1 Control plane VSphereMachineConfig template is not set. Using default template.
2022-02-25T19:10:10.973Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=demo-vc-01.terasky.demo -e GOVC_INSECURE=true eksa_1645816207994777826 govc find -json /Demo-Datacenter -type VirtualMachine -name bottlerocket-v1.21.5-kubernetes-1-21-eks-9-amd64-3501d2b"}
2022-02-25T19:10:11.122Z V2 Template not found: /Demo-Datacenter/vm/Templates/bottlerocket-v1.21.5-kubernetes-1-21-eks-9-amd64-3501d2b
2022-02-25T19:10:11.123Z V2 Template not available. Creating {"template": "/Demo-Datacenter/vm/Templates/bottlerocket-v1.21.5-kubernetes-1-21-eks-9-amd64-3501d2b"}
2022-02-25T19:10:11.123Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=demo-vc-01.terasky.demo -e GOVC_INSECURE=true eksa_1645816207994777826 govc library.ls eks-a-templates"}
2022-02-25T19:10:11.642Z V0 Creating template. This might take a while.
2022-02-25T19:10:11.643Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=demo-vc-01.terasky.demo -e GOVC_INSECURE=true eksa_1645816207994777826 govc library.info -json eks-a-templates/bottlerocket-v1.21.5-kubernetes-1-21-eks-9-amd64-3501d2b"}
2022-02-25T19:10:11.890Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_INSECURE=true -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=demo-vc-01.terasky.demo eksa_1645816207994777826 govc library.rm eks-a-templates/bottlerocket-v1.21.5-kubernetes-1-21-eks-9-amd64-3501d2b"}
2022-02-25T19:10:12.766Z V2 Importing template from ova url {"ova": "https://anywhere-assets.eks.amazonaws.com/releases/bundles/7/artifacts/ova/1-21/bottlerocket-v1.21.5-eks-d-1-21-9-eks-a-7-amd64.ova"}
2022-02-25T19:10:12.767Z V4 Importing template {"ova": "https://anywhere-assets.eks.amazonaws.com/releases/bundles/7/artifacts/ova/1-21/bottlerocket-v1.21.5-eks-d-1-21-9-eks-a-7-amd64.ova", "templateName": "bottlerocket-v1.21.5-kubernetes-1-21-eks-9-amd64-3501d2b"}
2022-02-25T19:10:12.767Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=demo-vc-01.terasky.demo -e GOVC_INSECURE=true eksa_1645816207994777826 govc library.import -k -pull -n bottlerocket-v1.21.5-kubernetes-1-21-eks-9-amd64-3501d2b eks-a-templates https://anywhere-assets.eks.amazonaws.com/releases/bundles/7/artifacts/ova/1-21/bottlerocket-v1.21.5-eks-d-1-21-9-eks-a-7-amd64.ova"}
govc: The import of library item 5aa58e48-235c-4a33-92f5-eed8d8415e00 has failed. Reason: Error transferring file bottlerocket-v1.21.5-eks-d-1-21-9-eks-a-7-amd64.ova to ds:///vmfs/volumes/vsan:52de39076e99c733-3a9ee62808d57b6a//contentlib-d50c09f9-d417-42d3-bd07-29e2015ee19d/5aa58e48-235c-4a33-92f5-eed8d8415e00/bottlerocket-v1.21.5-eks-d-1-21-9-eks-a-7-amd64_bf5f1151-3770-4162-bc43-e8981ebf3642.ova?serverId=fa304e5a-0db4-433e-9fd6-3241ac7a10d3. Reason: Error during transfer of ds:///vmfs/volumes/vsan:52de39076e99c733-3a9ee62808d57b6a//contentlib-d50c09f9-d417-42d3-bd07-29e2015ee19d/5aa58e48-235c-4a33-92f5-eed8d8415e00/bottlerocket-v1.21.5-eks-d-1-21-9-eks-a-7-amd64_bf5f1151-3770-4162-bc43-e8981ebf3642.ova?serverId=fa304e5a-0db4-433e-9fd6-3241ac7a10d3: IO error during transfer of ds:/vmfs/volumes/vsan:52de39076e99c733-3a9ee62808d57b6a/contentlib-d50c09f9-d417-42d3-bd07-29e2015ee19d/5aa58e48-235c-4a33-92f5-eed8d8415e00/bottlerocket-vmware-k8s-1.21-x86_64-1.6.0-9749af99_bf5f1151-3770-4162-bc43-e8981ebf3642.vmdk: Pipe closed.
2022-02-25T19:10:36.252Z V0 ❌ Validation failed {"validation": "vsphere Provider setup is valid", "error": "failed setting default values for vsphere machine configs: failed importing template into library: error importing template: exit status 1", "remediation": ""}
2022-02-25T19:10:36.252Z V0 ✅
2022-02-25T19:10:36.252Z V0 ✅ Create preflight validations pass
2022-02-25T19:10:36.252Z V4 Task finished {"task_name": "setup-validate", "duration": "25.820985977s"}
2022-02-25T19:10:36.252Z V4 ----------------------------------
2022-02-25T19:10:36.252Z V4 Tasks completed {"duration": "25.822620047s"}
2022-02-25T19:10:36.252Z V3 Logging out from current govc session
2022-02-25T19:10:36.252Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_INSECURE=true -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=demo-vc-01.terasky.demo eksa_1645816207994777826 govc session.logout"}
2022-02-25T19:10:36.390Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=demo-vc-01.terasky.demo -e GOVC_INSECURE=true eksa_1645816207994777826 govc session.logout -k"}
2022-02-25T19:10:36.565Z V3 Cleaning up long running container {"name": "eksa_1645816207994777826"}
2022-02-25T19:10:36.566Z V6 Executing command {"cmd": "/usr/bin/docker rm -f -v eksa_1645816207994777826"}
Error: failed to create cluster: validations failedJust to test/narrow it down, I've changed osFamily: bottlerocket to osFamily: ubuntu. The Ubuntu template is successfully imported to vSphere and the deployment runs... But I'm not going to use Ubuntu as we've standardized on Bottlerocket.
I think this Bottlerocket OVA image is corrupted or invalid.
Output using Ubuntu (as you can see, the OVA import phase succeeds):
$ eksctl anywhere create cluster -f eksa-mgmt-cluster.yaml -v 9
2022-02-25T19:11:53.740Z V4 Logger init completed {"vlevel": 9}
2022-02-25T19:11:53.740Z V6 Executing command {"cmd": "/usr/bin/docker version --format {{.Client.Version}}"}
2022-02-25T19:11:53.785Z V6 Executing command {"cmd": "/usr/bin/docker info --format '{{json .MemTotal}}'"}
2022-02-25T19:11:53.878Z V4 Reading releases manifest {"url": "https://anywhere-assets.eks.amazonaws.com/releases/eks-a/manifest.yaml"}
2022-02-25T19:11:54.167Z V4 Reading bundles manifest {"url": "https://anywhere-assets.eks.amazonaws.com/releases/bundles/7/manifest.yaml"}
2022-02-25T19:11:54.597Z V2 Pulling docker image {"image": "public.ecr.aws/eks-anywhere/cli-tools:v0.1.0-eks-a-7"}
2022-02-25T19:11:54.597Z V6 Executing command {"cmd": "/usr/bin/docker pull public.ecr.aws/eks-anywhere/cli-tools:v0.1.0-eks-a-7"}
2022-02-25T19:11:56.801Z V3 Initializing long running container {"name": "eksa_1645816314597739236", "image": "public.ecr.aws/eks-anywhere/cli-tools:v0.1.0-eks-a-7"}
2022-02-25T19:11:56.802Z V6 Executing command {"cmd": "/usr/bin/docker run -d --name eksa_1645816314597739236 --network host -w /home/k8s/eksa-playground/071 -v /var/run/docker.sock:/var/run/docker.sock -v /home/k8s/eksa-playground/071:/home/k8s/eksa-playground/071 --entrypoint sleep public.ecr.aws/eks-anywhere/cli-tools:v0.1.0-eks-a-7 infinity"}
2022-02-25T19:11:57.054Z V4 Task start {"task_name": "setup-validate"}
2022-02-25T19:11:57.054Z V0 Performing setup and validations
2022-02-25T19:11:57.054Z V4 Relative network path specified, using path /Demo-Datacenter/network/demo-itay-k8s-nodes
2022-02-25T19:11:57.054Z V0 Warning: VSphereDatacenterConfig configured in insecure mode
2022-02-25T19:11:57.078Z V0 ✅ Connected to server
2022-02-25T19:11:57.078Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=demo-vc-01.terasky.demo -e GOVC_INSECURE=true eksa_1645816314597739236 govc about -k"}
2022-02-25T19:11:57.251Z V5 Retry execution successful {"retries": 1, "duration": "173.045146ms"}
2022-02-25T19:11:57.251Z V0 ✅ Authenticated to vSphere
2022-02-25T19:11:57.251Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_URL=demo-vc-01.terasky.demo -e GOVC_INSECURE=true -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** eksa_1645816314597739236 govc datacenter.info Demo-Datacenter"}
2022-02-25T19:11:57.472Z V5 Retry execution successful {"retries": 1, "duration": "220.851863ms"}
2022-02-25T19:11:57.472Z V0 ✅ Datacenter validated
2022-02-25T19:11:57.472Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=demo-vc-01.terasky.demo -e GOVC_INSECURE=true eksa_1645816314597739236 govc find -maxdepth=1 /Demo-Datacenter/network -type n -name demo-itay-k8s-nodes"}
2022-02-25T19:11:57.630Z V5 Retry execution successful {"retries": 1, "duration": "157.415086ms"}
2022-02-25T19:11:57.630Z V0 ✅ Network validated
2022-02-25T19:11:57.630Z V1 Control plane VSphereMachineConfig template is not set. Using default template.
2022-02-25T19:11:57.630Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=demo-vc-01.terasky.demo -e GOVC_INSECURE=true eksa_1645816314597739236 govc find -json /Demo-Datacenter -type VirtualMachine -name ubuntu-v1.21.5-kubernetes-1-21-eks-9-amd64-b223da4"}
2022-02-25T19:11:57.789Z V2 Template not found: /Demo-Datacenter/vm/Templates/ubuntu-v1.21.5-kubernetes-1-21-eks-9-amd64-b223da4
2022-02-25T19:11:57.789Z V2 Template not available. Creating {"template": "/Demo-Datacenter/vm/Templates/ubuntu-v1.21.5-kubernetes-1-21-eks-9-amd64-b223da4"}
2022-02-25T19:11:57.789Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=demo-vc-01.terasky.demo -e GOVC_INSECURE=true eksa_1645816314597739236 govc library.ls eks-a-templates"}
2022-02-25T19:11:58.306Z V0 Creating template. This might take a while.
2022-02-25T19:11:58.307Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_INSECURE=true -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=demo-vc-01.terasky.demo eksa_1645816314597739236 govc library.info -json eks-a-templates/ubuntu-v1.21.5-kubernetes-1-21-eks-9-amd64-b223da4"}
2022-02-25T19:11:58.511Z V2 Importing template from ova url {"ova": "https://anywhere-assets.eks.amazonaws.com/releases/bundles/7/artifacts/ova/1-21/ubuntu-v1.21.5-eks-d-1-21-9-eks-a-7-amd64.ova"}
2022-02-25T19:11:58.511Z V4 Importing template {"ova": "https://anywhere-assets.eks.amazonaws.com/releases/bundles/7/artifacts/ova/1-21/ubuntu-v1.21.5-eks-d-1-21-9-eks-a-7-amd64.ova", "templateName": "ubuntu-v1.21.5-kubernetes-1-21-eks-9-amd64-b223da4"}
2022-02-25T19:11:58.511Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_INSECURE=true -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=demo-vc-01.terasky.demo eksa_1645816314597739236 govc library.import -k -pull -n ubuntu-v1.21.5-kubernetes-1-21-eks-9-amd64-b223da4 eks-a-templates https://anywhere-assets.eks.amazonaws.com/releases/bundles/7/artifacts/ova/1-21/ubuntu-v1.21.5-eks-d-1-21-9-eks-a-7-amd64.ova"}
2022-02-25T19:14:57.404Z V4 Deploying template {"dir": "/Demo-Datacenter/vm/Templates", "templateName": "ubuntu-v1.21.5-kubernetes-1-21-eks-9-amd64-b223da4"}
2022-02-25T19:14:57.405Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=demo-vc-01.terasky.demo -e GOVC_INSECURE=true eksa_1645816314597739236 govc folder.info /Demo-Datacenter/vm/Templates"}
2022-02-25T19:14:57.577Z V5 Retry execution successful {"retries": 1, "duration": "172.610711ms"}
2022-02-25T19:14:57.577Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=demo-vc-01.terasky.demo -e GOVC_INSECURE=true eksa_1645816314597739236 govc library.deploy -dc Demo-Datacenter -ds vsanDatastore -pool /Demo-Datacenter/host/Demo-Cluster/Resources -folder /Demo-Datacenter/vm/Templates -options it-eksa-mgmt/deploy-opts.json /eks-a-templates/ubuntu-v1.21.5-kubernetes-1-21-eks-9-amd64-b223da4 ubuntu-v1.21.5-kubernetes-1-21-eks-9-amd64-b223da4"}
2022-02-25T19:15:55.513Z V4 Taking template snapshot {"templateName": "/Demo-Datacenter/vm/Templates/ubuntu-v1.21.5-kubernetes-1-21-eks-9-amd64-b223da4"}
2022-02-25T19:15:55.513Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=demo-vc-01.terasky.demo -e GOVC_INSECURE=true eksa_1645816314597739236 govc snapshot.create -dc Demo-Datacenter -m=false -vm /Demo-Datacenter/vm/Templates/ubuntu-v1.21.5-kubernetes-1-21-eks-9-amd64-b223da4 root"}
2022-02-25T19:15:56.238Z V4 Marking vm as template {"templateName": "/Demo-Datacenter/vm/Templates/ubuntu-v1.21.5-kubernetes-1-21-eks-9-amd64-b223da4"}
2022-02-25T19:15:56.238Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_URL=demo-vc-01.terasky.demo -e GOVC_INSECURE=true -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** eksa_1645816314597739236 govc vm.markastemplate -dc Demo-Datacenter /Demo-Datacenter/vm/Templates/ubuntu-v1.21.5-kubernetes-1-21-eks-9-amd64-b223da4"}
2022-02-25T19:15:56.578Z V2 Tagging template {"template": "/Demo-Datacenter/vm/Templates/ubuntu-v1.21.5-kubernetes-1-21-eks-9-amd64-b223da4"}
2022-02-25T19:15:56.578Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=demo-vc-01.terasky.demo -e GOVC_INSECURE=true eksa_1645816314597739236 govc tags.category.ls -json"}
2022-02-25T19:15:56.845Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_URL=demo-vc-01.terasky.demo -e GOVC_INSECURE=true -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** eksa_1645816314597739236 govc tags.ls -json"}
2022-02-25T19:15:57.066Z V3 Adding tag to template {"tag": "os:ubuntu", "template": "/Demo-Datacenter/vm/Templates/ubuntu-v1.21.5-kubernetes-1-21-eks-9-amd64-b223da4"}
2022-02-25T19:15:57.066Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=demo-vc-01.terasky.demo -e GOVC_INSECURE=true eksa_1645816314597739236 govc tags.attach os:ubuntu /Demo-Datacenter/vm/Templates/ubuntu-v1.21.5-kubernetes-1-21-eks-9-amd64-b223da4"}
2022-02-25T19:15:57.333Z V3 Creating tag {"tag": "eksdRelease:kubernetes-1-21-eks-9", "category": "eksdRelease"}
2022-02-25T19:15:57.333Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=demo-vc-01.terasky.demo -e GOVC_INSECURE=true eksa_1645816314597739236 govc tags.create -c eksdRelease eksdRelease:kubernetes-1-21-eks-9"}
2022-02-25T19:15:57.511Z V3 Adding tag to template {"tag": "eksdRelease:kubernetes-1-21-eks-9", "template": "/Demo-Datacenter/vm/Templates/ubuntu-v1.21.5-kubernetes-1-21-eks-9-amd64-b223da4"}
2022-02-25T19:15:57.511Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=demo-vc-01.terasky.demo -e GOVC_INSECURE=true eksa_1645816314597739236 govc tags.attach eksdRelease:kubernetes-1-21-eks-9 /Demo-Datacenter/vm/Templates/ubuntu-v1.21.5-kubernetes-1-21-eks-9-amd64-b223da4"}
2022-02-25T19:15:57.769Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_INSECURE=true -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=demo-vc-01.terasky.demo eksa_1645816314597739236 govc find -json /Demo-Datacenter -type VirtualMachine -name ubuntu-v1.21.5-kubernetes-1-21-eks-9-amd64-b223da4"}
2022-02-25T19:15:57.935Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_PASSWORD=***** -e GOVC_URL=demo-vc-01.terasky.demo -e GOVC_INSECURE=true -e GOVC_USERNAME=***** eksa_1645816314597739236 govc snapshot.tree -vm /Demo-Datacenter/vm/Templates/ubuntu-v1.21.5-kubernetes-1-21-eks-9-amd64-b223da4"}
2022-02-25T19:15:58.087Z V1 Control plane VSphereMachineConfig template is not set. Using default template.
2022-02-25T19:15:58.087Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=demo-vc-01.terasky.demo -e GOVC_INSECURE=true eksa_1645816314597739236 govc find -json /Demo-Datacenter -type VirtualMachine -name ubuntu-v1.21.5-kubernetes-1-21-eks-9-amd64-b223da4"}
2022-02-25T19:15:58.251Z V2 Template already exists. Skipping creation {"template": "/Demo-Datacenter/vm/Templates/ubuntu-v1.21.5-kubernetes-1-21-eks-9-amd64-b223da4"}
2022-02-25T19:15:58.251Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=demo-vc-01.terasky.demo -e GOVC_INSECURE=true eksa_1645816314597739236 govc find -json /Demo-Datacenter -type VirtualMachine -name ubuntu-v1.21.5-kubernetes-1-21-eks-9-amd64-b223da4"}
2022-02-25T19:15:58.414Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=demo-vc-01.terasky.demo -e GOVC_INSECURE=true eksa_1645816314597739236 govc snapshot.tree -vm /Demo-Datacenter/vm/Templates/ubuntu-v1.21.5-kubernetes-1-21-eks-9-amd64-b223da4"}
2022-02-25T19:15:58.576Z V1 Control plane VSphereMachineConfig template is not set. Using default template.
2022-02-25T19:15:58.576Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=demo-vc-01.terasky.demo -e GOVC_INSECURE=true eksa_1645816314597739236 govc find -json /Demo-Datacenter -type VirtualMachine -name ubuntu-v1.21.5-kubernetes-1-21-eks-9-amd64-b223da4"}
2022-02-25T19:15:58.742Z V2 Template already exists. Skipping creation {"template": "/Demo-Datacenter/vm/Templates/ubuntu-v1.21.5-kubernetes-1-21-eks-9-amd64-b223da4"}
2022-02-25T19:15:58.742Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=demo-vc-01.terasky.demo -e GOVC_INSECURE=true eksa_1645816314597739236 govc find -json /Demo-Datacenter -type VirtualMachine -name ubuntu-v1.21.5-kubernetes-1-21-eks-9-amd64-b223da4"}
2022-02-25T19:15:58.908Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_URL=demo-vc-01.terasky.demo -e GOVC_INSECURE=true -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** eksa_1645816314597739236 govc snapshot.tree -vm /Demo-Datacenter/vm/Templates/ubuntu-v1.21.5-kubernetes-1-21-eks-9-amd64-b223da4"}
2022-02-25T19:15:59.056Z V4 Relative datastore path specified, using path /Demo-Datacenter/datastore/vsanDatastore
2022-02-25T19:15:59.056Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=demo-vc-01.terasky.demo -e GOVC_INSECURE=true eksa_1645816314597739236 govc datastore.info /Demo-Datacenter/datastore/vsanDatastore"}
2022-02-25T19:15:59.202Z V5 Retry execution successful {"retries": 1, "duration": "145.477287ms"}
2022-02-25T19:15:59.202Z V0 ✅ Datastore validated
2022-02-25T19:15:59.202Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_INSECURE=true -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=demo-vc-01.terasky.demo eksa_1645816314597739236 govc folder.info /Demo-Datacenter/vm/LABS/Itay/EKSA"}
2022-02-25T19:15:59.355Z V5 Retry execution successful {"retries": 1, "duration": "153.205111ms"}
2022-02-25T19:15:59.355Z V0 ✅ Folder validated
2022-02-25T19:15:59.355Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=demo-vc-01.terasky.demo -e GOVC_INSECURE=true eksa_1645816314597739236 govc find -json /Demo-Datacenter -type p -name Resources"}
2022-02-25T19:15:59.510Z V5 Retry execution successful {"retries": 1, "duration": "155.130256ms"}
2022-02-25T19:15:59.510Z V0 ✅ Resource pool validated
2022-02-25T19:15:59.510Z V4 Relative datastore path specified, using path /Demo-Datacenter/datastore/vsanDatastore
2022-02-25T19:15:59.510Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_INSECURE=true -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=demo-vc-01.terasky.demo eksa_1645816314597739236 govc datastore.info /Demo-Datacenter/datastore/vsanDatastore"}
2022-02-25T19:15:59.659Z V5 Retry execution successful {"retries": 1, "duration": "148.932482ms"}
2022-02-25T19:15:59.659Z V0 ✅ Datastore validated
2022-02-25T19:15:59.659Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=demo-vc-01.terasky.demo -e GOVC_INSECURE=true eksa_1645816314597739236 govc folder.info /Demo-Datacenter/vm/LABS/Itay/EKSA"}
2022-02-25T19:15:59.823Z V5 Retry execution successful {"retries": 1, "duration": "164.242579ms"}
2022-02-25T19:15:59.823Z V0 ✅ Folder validated
2022-02-25T19:15:59.824Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=demo-vc-01.terasky.demo -e GOVC_INSECURE=true eksa_1645816314597739236 govc find -json /Demo-Datacenter -type p -name Resources"}
2022-02-25T19:15:59.984Z V5 Retry execution successful {"retries": 1, "duration": "160.682044ms"}
2022-02-25T19:15:59.984Z V0 ✅ Resource pool validated
2022-02-25T19:15:59.984Z V4 Relative datastore path specified, using path /Demo-Datacenter/datastore/vsanDatastore
2022-02-25T19:15:59.984Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=demo-vc-01.terasky.demo -e GOVC_INSECURE=true eksa_1645816314597739236 govc datastore.info /Demo-Datacenter/datastore/vsanDatastore"}
2022-02-25T19:16:00.141Z V5 Retry execution successful {"retries": 1, "duration": "156.984836ms"}
2022-02-25T19:16:00.141Z V0 ✅ Datastore validated
2022-02-25T19:16:00.141Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=demo-vc-01.terasky.demo -e GOVC_INSECURE=true eksa_1645816314597739236 govc folder.info /Demo-Datacenter/vm/LABS/Itay/EKSA"}
2022-02-25T19:16:00.313Z V5 Retry execution successful {"retries": 1, "duration": "171.644668ms"}
2022-02-25T19:16:00.313Z V0 ✅ Folder validated
2022-02-25T19:16:00.313Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_URL=demo-vc-01.terasky.demo -e GOVC_INSECURE=true -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** eksa_1645816314597739236 govc find -json /Demo-Datacenter -type p -name Resources"}
2022-02-25T19:16:00.481Z V5 Retry execution successful {"retries": 1, "duration": "167.980785ms"}
2022-02-25T19:16:00.481Z V0 ✅ Resource pool validated
2022-02-25T19:16:00.481Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_PASSWORD=***** -e GOVC_URL=demo-vc-01.terasky.demo -e GOVC_INSECURE=true -e GOVC_USERNAME=***** eksa_1645816314597739236 govc find -json /Demo-Datacenter -type VirtualMachine -name ubuntu-v1.21.5-kubernetes-1-21-eks-9-amd64-b223da4"}
2022-02-25T19:16:00.644Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=demo-vc-01.terasky.demo -e GOVC_INSECURE=true eksa_1645816314597739236 govc tags.attached.ls -json -r /Demo-Datacenter/vm/Templates/ubuntu-v1.21.5-kubernetes-1-21-eks-9-amd64-b223da4"}
2022-02-25T19:16:00.828Z V0 ✅ Control plane and Workload templates validated
2022-02-25T19:16:00.828Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=demo-vc-01.terasky.demo -e GOVC_INSECURE=true eksa_1645816314597739236 govc datastore.info -json=true /Demo-Datacenter/datastore/vsanDatastore"}
2022-02-25T19:16:00.976Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=demo-vc-01.terasky.demo -e GOVC_INSECURE=true eksa_1645816314597739236 govc datastore.info -json=true /Demo-Datacenter/datastore/vsanDatastore"}
2022-02-25T19:16:01.146Z V6 Executing command {"cmd": "/usr/bin/docker exec -i -e GOVC_USERNAME=***** -e GOVC_PASSWORD=***** -e GOVC_URL=demo-vc-01.terasky.demo -e GOVC_INSECURE=true eksa_1645816314597739236 govc datastore.info -json=true /Demo-Datacenter/datastore/vsanDatastore"}
2022-02-25T19:16:05.359Z V0 ✅ Vsphere Provider setup is valid
2022-02-25T19:16:05.359Z V0 ✅
2022-02-25T19:16:05.359Z V0 ✅ Create preflight validations pass
2022-02-25T19:16:05.359Z V4 Task finished {"task_name": "setup-validate", "duration": "4m8.304786342s"}
2022-02-25T19:16:05.359Z V4 ----------------------------------
2022-02-25T19:16:05.359Z V4 Task start {"task_name": "bootstrap-cluster-init"}
2022-02-25T19:16:05.359Z V0 Creating new bootstrap cluster
2022-02-25T19:16:05.360Z V4 Creating kind cluster {"name": "it-eksa-mgmt-eks-a-cluster", "kubeconfig": "it-eksa-mgmt/generated/it-eksa-mgmt.kind.kubeconfig"}
2022-02-25T19:16:05.360Z V6 Executing command {"cmd": "/usr/bin/docker exec -i eksa_1645816314597739236 kind create cluster --name it-eksa-mgmt-eks-a-cluster --kubeconfig it-eksa-mgmt/generated/it-eksa-mgmt.kind.kubeconfig --image public.ecr.aws/eks-anywhere/kubernetes-sigs/kind/node:v1.21.5-eks-d-1-21-9-eks-a-7 --config it-eksa-mgmt/generated/kind_tmp.yaml"}
Creating cluster "it-eksa-mgmt-eks-a-cluster" ...
• Ensuring node image (public.ecr.aws/eks-anywhere/kubernetes-sigs/kind/node:v1.21.5-eks-d-1-21-9-eks-a-7) 🖼 ...
✓ Ensuring node image (public.ecr.aws/eks-anywhere/kubernetes-sigs/kind/node:v1.21.5-eks-d-1-21-9-eks-a-7) 🖼
• Preparing nodes 📦 ...
✓ Preparing nodes 📦
• Writing configuration 📜 ...
✓ Writing configuration 📜
• Starting control-plane 🕹️ ...
abhay-krishna
commented
2 years ago @itaytalmi Could you try using the bundles override flag as follows to use a custom bundle manifest with the older BR OVA?
--bundles-override flag, this will make it use the downloaded and modified local bundle manifest.The following commands should be useful
export KUBEVERSION=1.21
# Download the latest release bundle manifest
wget https://anywhere-assets.eks.amazonaws.com/releases/bundles/7/manifest.yaml -O bundle-release-7.yaml
# Get the BR OVA URI for 1.21 from the previous bundle release
PREVIOUS_BR_OVA=$(curl -fsSL https://anywhere-assets.eks.amazonaws.com/releases/bundles/6/manifest.yaml | yq e ".spec.versionsBundles[] | select(.kubeVersion==\"$KUBEVERSION\").eksD.ova.bottlerocket.uri" -)
# Replace the BR OVA for 1.21 in the downloaded bundle manifest with the OVA from the previous bundle
yq e -i ".spec.versionsBundles[] | select(.kubeVersion==\"$KUBEVERSION\").eksD.ova.bottlerocket.uri = \"$PREVIOUS_BR_OVA\"" bundle-release-7.yaml
# Run EKS-A cluster creation with the `--bundles-override` flag.
eksctl anywhere create cluster -f eksa-mgmt-cluster.yaml -v 9 --bundles-override bundle-release-7.yamlI have manually downloaded the OVA template using:
wget https://anywhere-assets.eks.amazonaws.com/releases/bundles/7/artifacts/ova/1-21/bottlerocket-v1.21.5-eks-d-1-21-9-eks-a-7-amd64.ovaThen manually imported to vSphere and assigned the tags:
os:bottlerocket
eksdRelease:kubernetes-1-21-eks-9Added the template field to the cluster config. e.g. template: "/Demo-Datacenter/vm/LABS/Itay/EKSA/bottlerocket-vmware-k8s-1.21-x86_64-1.6.0-9749af99"
Ran the deployment, it completed successfully.
It doesn't work only when the EKS Anywhere binary tries to import it when the template field is not specified.
@abhay-krishna I could try that workaround if it helps, but I guess it would mean I'd have to use an outdated Bottlerocket image?
Performing setup and validations
Warning: VSphereDatacenterConfig configured in insecure mode
✅ Connected to server
✅ Authenticated to vSphere
✅ Datacenter validated
✅ Network validated
Warning: Your VM template has no snapshots. Defaulting to FullClone mode. VM provisioning might take longer.
Warning: Your VM template has no snapshots. Defaulting to FullClone mode. VM provisioning might take longer.
Warning: Your VM template has no snapshots. Defaulting to FullClone mode. VM provisioning might take longer.
✅ Datastore validated
✅ Folder validated
✅ Resource pool validated
✅ Datastore validated
✅ Folder validated
✅ Resource pool validated
✅ Datastore validated
✅ Folder validated
✅ Resource pool validated
✅ Control plane and Workload templates validated
✅ Vsphere Provider setup is valid
✅
✅ Create preflight validations pass
Creating new bootstrap cluster
Installing cluster-api providers on bootstrap cluster
Provider specific setup
Creating new workload cluster
Installing networking on workload cluster
Installing storage class on workload cluster
Installing cluster-api providers on workload cluster
Installing EKS-A secrets on workload cluster
Moving cluster management from bootstrap to workload cluster
Installing EKS-A custom components (CRD and controller) on workload cluster
Creating EKS-A CRDs instances on workload cluster
Installing AddonManager and GitOps Toolkit on workload cluster
GitOps field not specified, bootstrap flux skipped
Writing cluster config file
Deleting bootstrap cluster
🎉 Cluster created!@itaytalmi It's technically the OVA from the previous BR release, this is the OS image you would have used when running the create workflow with the v0.7.0 EKS-A binary.
itaytalmi
commented
2 years ago I think it might have been a temporary issue with the CDN. It works now... Thank you
itaytalmi
commented
2 years ago Reopening the issue as it is now occurring again - after deleting the imported image and content library from vSphere and running another deployment...
itaytalmi
commented
2 years ago @abhay-krishna The suggested workaround returns an error.
$ wget https://anywhere-assets.eks.amazonaws.com/releases/bundles/7/manifest.yaml -O bundle-release-7.yaml
--2022-02-25 22:01:57-- https://anywhere-assets.eks.amazonaws.com/releases/bundles/7/manifest.yaml
Resolving anywhere-assets.eks.amazonaws.com (anywhere-assets.eks.amazonaws.com)... 52.85.96.105, 52.85.96.107, 52.85.96.35, ...
Connecting to anywhere-assets.eks.amazonaws.com (anywhere-assets.eks.amazonaws.com)|52.85.96.105|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 41114 (40K) [binary/octet-stream]
Saving to: ‘bundle-release-7.yaml’
bundle-release-7.yaml 100%[======================================================================================================>] 40.15K --.-KB/s in 0.05s
2022-02-25 22:01:58 (811 KB/s) - ‘bundle-release-7.yaml’ saved [41114/41114]
$ PREVIOUS_BR_OVA=$(curl -fsSL https://anywhere-assets.eks.amazonaws.com/releases/bundles/6/manifest.yaml | yq e ".spec.versionsBundles[] | select(.kubeVersion==\"$KUBEVERSION\").eksD.ova.bottlerocket.uri" -)
$ yq e -i ".spec.versionsBundles[] | select(.kubeVersion==\"$KUBEVERSION\").eksD.ova.bottlerocket.uri = \"$PREVIOUS_BR_OVA\"" bundle-release-7.yaml
$ ls -l
total 56
drwxrwxr-x 2 k8s k8s 4096 Feb 25 22:02 ./
drwxr-xr-x 10 k8s k8s 4096 Feb 25 22:00 ../
-rw-rw-r-- 1 k8s k8s 37638 Feb 25 22:02 bundle-release-7.yaml
-rw-rw-r-- 1 k8s k8s 4345 Feb 25 22:00 eksa-mgmt-cluster.yaml
$ eksctl anywhere create cluster -f eksa-mgmt-cluster.yaml -v 9 --bundles-override bundle-release-7.yaml
2022-02-25T22:02:23.338Z V4 Logger init completed {"vlevel": 9}
2022-02-25T22:02:23.338Z V6 Executing command {"cmd": "/usr/bin/docker version --format {{.Client.Version}}"}
2022-02-25T22:02:23.393Z V6 Executing command {"cmd": "/usr/bin/docker info --format '{{json .MemTotal}}'"}
2022-02-25T22:02:23.501Z V4 Reading bundles manifest {"url": "bundle-release-7.yaml"}
Error: failed to create cluster: unable to get cluster config from file: kubernetes version 1.21 is not supported by bundles manifest 0
jaxesn
commented
2 years ago @itaytalmi could you send us the steps you took to import the ova manually?
We have seen something like this before, we have a couple vcenter clusters internally and previous versions of the BR ova would import fine on one, but not the other. We havent been able to track down the root cause and did not run into it with the latest 1.6.0 BR ova. The import flow through the CLI uses the Library to avoid having to download the file locally and then upload to vcenter. We have seen weirdness in the past around this feature in vcenter so its possible we are running into another vcenter bug/issue here that seems to affect different versions/clusters differently.
itaytalmi
commented
2 years ago Hi @jaxesn,
wget https://anywhere-assets.eks.amazonaws.com/releases/bundles/7/artifacts/ova/1-21/bottlerocket-v1.21.5-eks-d-1-21-9-eks-a-7-amd64.ovaFrom vSphere UI, I deployed the OVA directly to some VM folder where all our templates reside (using the Deploy OVF Template option).
Created and assigned the tags:
os:bottlerocket
eksdRelease:kubernetes-1-21-eks-9Those are the exact steps I took.
itaytalmi
commented
2 years ago I've uploaded the vSphere Content Library logs (/var/log/vmware/content-library/cls.log):
There are some interesting entries containing Bottlerocket information. Such as:
2022-02-25T22:33:17.173Z | DEBUG | aa92e7dd-275e-407a-9c09-cabbad462b62-82-60 | cls-simple-activity-5 | UpdateSessionServiceImpl | Sub-file/TransferStatus/BytesTransferred = bottlerocket-vmware-k8s-1.21-x86_64-1.6.0-9749af99.vmdk/TRANSFERRING/223092808
2022-02-25T22:33:17.173Z | DEBUG | aa92e7dd-275e-407a-9c09-cabbad462b62-82-60 | cls-simple-activity-5 | UpdateSessionServiceImpl | received content for update session 14af649c-ca56-43f7-aa64-1a7204e51275 (transfer session urn:transfer:c2cc7f78-8c5d-4c8b-ad3d-8e8b2bb36a5a): 1 file(s) | source [https://anywhere-assets.eks.amazonaws.com/releases/bundles/7/artifacts/ova/1-21/bottlerocket-v1.21.5-eks-d-1-21-9-eks-a-7-amd64.ova]: 223116360b/223395840b (212 MB/213 MB : 99%) - w:0 t:1 v:0 d:0 e:0 | target [ds:///vmfs/volumes/vsan:52de39076e99c733-3a9ee62808d57b6a//contentlib-8b15fc7f-ea62-4a46-aede-9cf19d3dee61/ae8fc4a1-78f8-4cd3-8f2e-8c41f6fdf4cc/bottlerocket-v1.21.5-eks-d-1-21-9-eks-a-7-amd64_14af649c-ca56-43f7-aa64-1a7204e51275.ova?serverId=fa304e5a-0db4-433e-9fd6-3241ac7a10d3]: 0b/?b (0 bytes/? : ?%) - w:1 t:0 v:0 d:0 e:0
2022-02-25T22:33:17.173Z | DEBUG | aa92e7dd-275e-407a-9c09-cabbad462b62-82-60 | cls-simple-activity-5 | VcTaskService | Setting progress for task 14af649c-ca56-43f7-aa64-1a7204e51275 (ManagedObjectReference: type = Task, value = task-200765, serverGuid = fa304e5a-0db4-433e-9fd6-3241ac7a10d3) from 90 to 94
2022-02-25T22:33:17.177Z | DEBUG | aa92e7dd-275e-407a-9c09-cabbad462b62-82-60 | cls-simple-activity-5 | AsyncCommitActivity | Transferring files [bottlerocket-vmware-k8s-1.21-x86_64-1.6.0-9749af99.mf, bottlerocket-vmware-k8s-1.21-x86_64-1.6.0-9749af99.ovf, bottlerocket-vmware-k8s-1.21-x86_64-1.6.0-9749af99.vmdk], 94% complete
2022-02-25T22:33:17.177Z | DEBUG | aa92e7dd-275e-407a-9c09-cabbad462b62-82-60 | cls-simple-activity-5 | AsyncCommitActivity | file bottlerocket-v1.21.5-eks-d-1-21-9-eks-a-7-amd64.ova not transferred fully, checking state again in 1000 ms
2022-02-25T22:33:17.794Z | ERROR | 99042739-ebab-49b1-82fd-9a24d4a69f0b-1b | transferService-pool-6-thread-271 | TransferEndpointImpl | Session c2cc7f78-8c5d-4c8b-ad3d-8e8b2bb36a5a, Item bottlerocket-v1.21.5-eks-d-1-21-9-eks-a-7-amd64.ova, Endpoint ds:/vmfs/volumes/vsan:52de39076e99c733-3a9ee62808d57b6a/contentlib-8b15fc7f-ea62-4a46-aede-9cf19d3dee61/ae8fc4a1-78f8-4cd3-8f2e-8c41f6fdf4cc/bottlerocket-vmware-k8s-1.21-x86_64-1.6.0-9749af99_14af649c-ca56-43f7-aa64-1a7204e51275.vmdk: IO error during transfer of ds:/vmfs/volumes/vsan:52de39076e99c733-3a9ee62808d57b6a/contentlib-8b15fc7f-ea62-4a46-aede-9cf19d3dee61/ae8fc4a1-78f8-4cd3-8f2e-8c41f6fdf4cc/bottlerocket-vmware-k8s-1.21-x86_64-1.6.0-9749af99_14af649c-ca56-43f7-aa64-1a7204e51275.vmdk: Pipe closed
java.io.IOException: Pipe closed
at java.io.PipedInputStream.checkStateForReceive(PipedInputStream.java:260)
at java.io.PipedInputStream.receive(PipedInputStream.java:226)
at java.io.PipedOutputStream.write(PipedOutputStream.java:149)
at com.vmware.transfer.impl.TransferOutputStream.write(TransferOutputStream.java:55)
at com.vmware.transfer.impl.TransferItemImpl.transferTarEntry(TransferItemImpl.java:1227)
at com.vmware.transfer.impl.TransferItemImpl.initiateTargetSequentialStreaming(TransferItemImpl.java:1079)
at com.vmware.transfer.impl.TransferItemImpl.run(TransferItemImpl.java:606)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
What happened:
Attempting to deploy a cluster using v0.7.1, I am experiencing the following issue:
In vCenter, I can see the same error:
This occurs in two different environments, two different VMware vCenters.
What you expected to happen:
As the Bottlerocket OVA template doesn't exist in vSphere, EKS Anywhere is supposed to import it, then use it to deploy the cluster.
How to reproduce it (as minimally and precisely as possible):
Just deploy a cluster using the standard
eksctl anywhere create cluster -f eksa-mgmt-cluster.yamlcommand, using a standard cluster config (without explicitly specifying the template or importing the OVA to vSphere, so EKS Anywhere imports it during the deployment). Here is the config I'm using:Anything else we need to know?:
Environment: