search

vmware / container-service-extension

Container Service for VMware vCloud Director
https://vmware.github.io/container-service-extension
Other
78 stars 52 forks source link

cse upgrade failed with VM customization error. #843

Open yaju-singh opened 4 years ago

yaju-singh commented 4 years ago

Hi, While performing cse upgrade the installation failed with the VM customization error. The install logs have the following entries.

Get:1 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [8993 B] Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [41.7 kB] Err:3 https://download.docker.com/linux/ubuntu xenial InRelease Connection timed out after 30000 milliseconds Err:4 http://security.ubuntu.com/ubuntu xenial-security InRelease Temporary failure resolving 'security.ubuntu.com' Err:5 http://archive.ubuntu.com/ubuntu xenial InRelease Could not connect to archive.ubuntu.com:80 (91.189.88.152), connection timed out [IP: 91.189.88.152 80] Err:6 http://archive.ubuntu.com/ubuntu xenial-updates InRelease Unable to connect to archive.ubuntu.com:http: [IP: 91.189.88.152 80] Err:7 http://archive.ubuntu.com/ubuntu xenial-backports InRelease Unable to connect to archive.ubuntu.com:http: [IP: 91.189.88.152 80] Fetched 50.7 kB in 1min 40s (506 B/s) Reading package lists... Reading package lists... Building dependency tree... Reading state information...

20-12-02 19:46:52 | template_builder:355 - _customize_vm | ERROR :: Failed VM customization Result start=== [100, <Response [200]>, <Response [200]>] ===Result end NoneType: None 20-12-02 19:46:52 | configure_cse:1445 - upgrade_cse | ERROR :: CSE Installation Error Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/container_service_extension/configure_cse.py", line 1422, in upgrade_cse _upgrade_to_35( File "/usr/local/lib/python3.8/site-packages/container_service_extension/configure_cse.py", line 1600, in _upgrade_to_35 _install_all_templates( File "/usr/local/lib/python3.8/site-packages/container_service_extension/configure_cse.py", line 1048, in _install_all_templates _install_single_template( File "/usr/local/lib/python3.8/site-packages/container_service_extension/configure_cse.py", line 1263, in _install_single_template builder.build(force_recreate=force_update, File "/usr/local/lib/python3.8/site-packages/container_service_extension/template_builder.py", line 465, in build self._customize_vm(vapp, self.temp_vm_name) File "/usr/local/lib/python3.8/site-packages/container_service_extension/template_builder.py", line 359, in _customize_vm raise Exception(f"{msg}; Result: {result}") Exception: Failed VM customization; Result: [100, <Response [200]>, <Response [200]>]

Please guide where could the issue have been

ChintanShahVMware commented 3 years ago

Hi Yaju, Let me look into it. Can you also attach CSE server debug logs to the ticket?

miguelhv81 commented 3 years ago

Hi ,

I am facing the same issue , i attach the server debug logs , could you help me?

[root@esal-vdcm-cse03 .cse-logs]# cat cse-install_2021-07-28_08-35-22.log 21-07-28 08:35:23 | remote_template_manager:104 - get_remote_template_cookbook | DEBUG :: Downloaded remote template cookbook from https://raw.githubusercontent.com/andrew-ni/container-service-extension-templates/tkg/template.yaml 21-07-28 08:35:23 | configure_cse:1351 - upgrade_cse | INFO :: Upgrading CSE on vCloud Director using config file './config.yaml' 21-07-28 08:35:23 | configure_cse:1374 - upgrade_cse | INFO :: Connected to vCD as system administrator: iaas3.vdc-p1.telefonica.com:443 21-07-28 08:35:23 | configure_cse:1398 - upgrade_cse | INFO :: Found CSE api extension registered with vCD, but couldn't determine version of CSE and/or vCD api used previously. 21-07-28 08:35:23 | configure_cse:975 - _setup_placement_policies | INFO :: Setting up placement policies for cluster types 21-07-28 08:35:23 | configure_cse:987 - _setup_placement_policies | INFO :: Skipping creation of global PVDC compute policy. Policy already exists 21-07-28 08:35:23 | configure_cse:1009 - _setup_placement_policies | INFO :: Skipping creation of VDC placement policy 'native'. Policy already exists 21-07-28 08:35:23 | configure_cse:1009 - _setup_placement_policies | INFO :: Skipping creation of VDC placement policy 'tkgplus'. Policy already exists 21-07-28 08:35:23 | configure_cse:803 - _register_def_schema | INFO :: Registering defined entity schema 21-07-28 08:35:23 | configure_cse:861 - _register_def_schema | INFO :: Skipping creation of Defined Entity Type. Defined Entity Type already exists. 21-07-28 08:35:24 | remote_template_manager:104 - get_remote_template_cookbook | DEBUG :: Downloaded remote template cookbook from https://raw.githubusercontent.com/andrew-ni/container-service-extension-templates/tkg/template.yaml 21-07-28 08:35:24 | utils:339 - download_file | INFO :: Downloading file from 'https://raw.githubusercontent.com/andrew-ni/container-service-extension-templates/tkg/scripts/ubuntu-16.04_tkgplus-1.19_weave-2.6.5_rev3/cust.sh' to '/root/.cse_scripts/ubuntu-16.04_tkgplus-1.19_weave-2.6.5_rev3/cust.sh'... 21-07-28 08:35:24 | utils:348 - download_file | INFO :: Download complete 21-07-28 08:35:24 | utils:339 - download_file | INFO :: Downloading file from 'https://raw.githubusercontent.com/andrew-ni/container-service-extension-templates/tkg/scripts/ubuntu-16.04_tkgplus-1.19_weave-2.6.5_rev3/init.sh' to '/root/.cse_scripts/ubuntu-16.04_tkgplus-1.19_weave-2.6.5_rev3/init.sh'... 21-07-28 08:35:24 | utils:348 - download_file | INFO :: Download complete 21-07-28 08:35:24 | utils:339 - download_file | INFO :: Downloading file from 'https://raw.githubusercontent.com/andrew-ni/container-service-extension-templates/tkg/scripts/ubuntu-16.04_tkgplus-1.19_weave-2.6.5_rev3/nfsd.sh' to '/root/.cse_scripts/ubuntu-16.04_tkgplus-1.19_weave-2.6.5_rev3/nfsd.sh'... 21-07-28 08:35:24 | utils:348 - download_file | INFO :: Download complete 21-07-28 08:35:24 | utils:339 - download_file | INFO :: Downloading file from 'https://raw.githubusercontent.com/andrew-ni/container-service-extension-templates/tkg/scripts/ubuntu-16.04_tkgplus-1.19_weave-2.6.5_rev3/mstr.sh' to '/root/.cse_scripts/ubuntu-16.04_tkgplus-1.19_weave-2.6.5_rev3/mstr.sh'... 21-07-28 08:35:25 | utils:348 - download_file | INFO :: Download complete 21-07-28 08:35:25 | utils:339 - download_file | INFO :: Downloading file from 'https://raw.githubusercontent.com/andrew-ni/container-service-extension-templates/tkg/scripts/ubuntu-16.04_tkgplus-1.19_weave-2.6.5_rev3/node.sh' to '/root/.cse_scripts/ubuntu-16.04_tkgplus-1.19_weave-2.6.5_rev3/node.sh'... 21-07-28 08:35:25 | utils:348 - download_file | INFO :: Download complete 21-07-28 08:35:25 | utils:339 - download_file | INFO :: Downloading file from 'https://raw.githubusercontent.com/andrew-ni/container-service-extension-templates/tkg/scripts/ubuntu-16.04_tkgplus-1.19_weave-2.6.5_rev3/cluster-upgrade/docker-upgrade.sh' to '/root/.cse_scripts/ubuntu-16.04_tkgplus-1.19_weave-2.6.5_rev3/cluster-upgrade/docker-upgrade.sh'... 21-07-28 08:35:25 | utils:348 - download_file | INFO :: Download complete 21-07-28 08:35:25 | utils:339 - download_file | INFO :: Downloading file from 'https://raw.githubusercontent.com/andrew-ni/container-service-extension-templates/tkg/scripts/ubuntu-16.04_tkgplus-1.19_weave-2.6.5_rev3/cluster-upgrade/master-cni-apply.sh' to '/root/.cse_scripts/ubuntu-16.04_tkgplus-1.19_weave-2.6.5_rev3/cluster-upgrade/master-cni-apply.sh'... 21-07-28 08:35:25 | utils:348 - download_file | INFO :: Download complete 21-07-28 08:35:25 | utils:339 - download_file | INFO :: Downloading file from 'https://raw.githubusercontent.com/andrew-ni/container-service-extension-templates/tkg/scripts/ubuntu-16.04_tkgplus-1.19_weave-2.6.5_rev3/cluster-upgrade/master-k8s-upgrade.sh' to '/root/.cse_scripts/ubuntu-16.04_tkgplus-1.19_weave-2.6.5_rev3/cluster-upgrade/master-k8s-upgrade.sh'... 21-07-28 08:35:25 | utils:348 - download_file | INFO :: Download complete 21-07-28 08:35:25 | utils:339 - download_file | INFO :: Downloading file from 'https://raw.githubusercontent.com/andrew-ni/container-service-extension-templates/tkg/scripts/ubuntu-16.04_tkgplus-1.19_weave-2.6.5_rev3/cluster-upgrade/worker-k8s-upgrade.sh' to '/root/.cse_scripts/ubuntu-16.04_tkgplus-1.19_weave-2.6.5_rev3/cluster-upgrade/worker-k8s-upgrade.sh'... 21-07-28 08:35:26 | utils:348 - download_file | INFO :: Download complete 21-07-28 08:35:26 | template_builder:155 - _cleanup_old_artifacts | INFO :: If K8 template, source ova file, and temporary vApp exists, they will be deleted 21-07-28 08:35:39 | template_builder:189 - _delete_temp_vapp | INFO :: Deleting temporary vApp 'ubuntu-16.04_tkgplus-1.19_weave-2.6.5_temp' 21-07-28 08:35:55 | template_builder:197 - _delete_temp_vapp | INFO :: Deleted temporary vApp 'ubuntu-16.04_tkgplus-1.19_weave-2.6.5_temp' 21-07-28 08:35:56 | utils:333 - download_file | INFO :: Skipping download to 'cse_cache/ubuntu-16.04-server-cloudimg-amd64.ova' (file already exists) 21-07-28 08:35:56 | pyvcloud_utils:313 - upload_ova_to_catalog | INFO :: Uploading 'ubuntu-16.04-server-cloudimg-amd64.ova' to catalog 'cse' 21-07-28 08:36:43 | pyvcloud_utils:321 - upload_ova_to_catalog | INFO :: Uploaded 'ubuntu-16.04-server-cloudimg-amd64.ova' to catalog 'cse' 21-07-28 08:36:43 | template_builder:189 - _delete_temp_vapp | INFO :: Deleting temporary vApp 'ubuntu-16.04_tkgplus-1.19_weave-2.6.5_temp' 21-07-28 08:36:43 | template_builder:246 - _create_temp_vapp | INFO :: Creating vApp 'ubuntu-16.04_tkgplus-1.19_weave-2.6.5_temp' 21-07-28 08:36:43 | utils:379 - read_data_file | DEBUG :: Found data file: /root/.cse_scripts/ubuntu-16.04_tkgplus-1.19_weave-2.6.5_rev3/init.sh 21-07-28 08:37:23 | template_builder:274 - _create_temp_vapp | INFO :: Created vApp 'ubuntu-16.04_tkgplus-1.19_weave-2.6.5_temp' 21-07-28 08:37:23 | template_builder:289 - _customize_vm | INFO :: Customizing vApp 'ubuntu-16.04_tkgplus-1.19_weave-2.6.5_temp', vm 'ubuntu-1604-k8s1193-vmware1-weave265-vm' 21-07-28 08:37:23 | utils:379 - read_data_file | DEBUG :: Found data file: /root/.cse_scripts/ubuntu-16.04_tkgplus-1.19_weave-2.6.5_rev3/cust.sh 21-07-28 08:37:23 | vsphere_utils:71 - get_vsphere | DEBUG :: VM ID: urn:vcloud:vm:d16d9956-2ea5-4ca0-83f7-78824ac99f20, Hostname: esal-vdct-vcs03 21-07-28 08:37:23 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:37:28 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:37:33 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:37:38 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:37:43 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:37:48 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:37:53 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:37:58 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:38:03 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:38:08 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:38:13 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:38:18 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:38:23 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:38:28 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:38:33 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:38:38 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:38:43 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:38:48 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:38:53 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:38:58 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:39:03 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:39:08 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:39:13 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:39:18 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:39:23 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:39:28 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:39:33 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:39:38 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:39:43 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:39:48 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:39:54 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:39:59 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:40:04 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:40:09 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:40:14 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:40:19 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:40:24 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:40:29 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:40:34 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:40:39 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:40:44 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:40:49 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:40:54 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:40:59 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:41:04 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:41:09 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:41:14 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:41:19 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:41:24 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:41:29 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:41:34 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:41:39 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:41:44 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:41:49 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:41:54 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:41:59 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:42:04 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:42:09 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:42:14 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:42:19 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:42:24 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:42:29 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:42:34 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:42:39 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:42:44 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:42:49 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:42:54 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:42:59 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:43:04 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:43:09 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:43:14 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:43:19 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:43:24 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:43:29 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:43:34 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:43:39 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:43:44 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:43:49 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:43:54 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:43:59 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:44:04 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:44:09 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:44:14 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:44:19 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:44:24 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:44:29 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:44:34 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:44:39 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:44:44 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsNotRunning 21-07-28 08:44:49 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsExecutingScripts 21-07-28 08:44:54 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsExecutingScripts 21-07-28 08:44:59 | vsphere_utils:94 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-2985', status=guestToolsRunning 21-07-28 08:45:01 | vsphere_utils:94 - callback | INFO :: waiting for process 1615 on vm 'vim.VirtualMachine:vm-2985' to finish (1) 21-07-28 08:45:11 | vsphere_utils:94 - callback | INFO :: waiting for process 1615 on vm 'vim.VirtualMachine:vm-2985' to finish (2) 21-07-28 08:45:21 | vsphere_utils:94 - callback | INFO :: waiting for process 1615 on vm 'vim.VirtualMachine:vm-2985' to finish (3) 21-07-28 08:45:31 | vsphere_utils:94 - callback | INFO :: waiting for process 1615 on vm 'vim.VirtualMachine:vm-2985' to finish (4) 21-07-28 08:45:41 | vsphere_utils:94 - callback | INFO :: waiting for process 1615 on vm 'vim.VirtualMachine:vm-2985' to finish (5) 21-07-28 08:45:51 | vsphere_utils:94 - callback | INFO :: waiting for process 1615 on vm 'vim.VirtualMachine:vm-2985' to finish (6) 21-07-28 08:46:01 | vsphere_utils:94 - callback | INFO :: waiting for process 1615 on vm 'vim.VirtualMachine:vm-2985' to finish (7) 21-07-28 08:46:11 | vsphere_utils:94 - callback | INFO :: waiting for process 1615 on vm 'vim.VirtualMachine:vm-2985' to finish (8) 21-07-28 08:46:21 | vsphere_utils:94 - callback | INFO :: waiting for process 1615 on vm 'vim.VirtualMachine:vm-2985' to finish (9) 21-07-28 08:46:32 | vsphere_utils:94 - callback | INFO :: process [100, <Response [200]>, <Response [200]>] on vm 'vim.VirtualMachine:vm-2985' finished, exit code: 100 21-07-28 08:46:32 | template_builder:333 - _customize_vm | DEBUG :: Result: [100, <Response [200]>, <Response [200]>] 21-07-28 08:46:32 | template_builder:340 - _customize_vm | DEBUG :: stderr: 21-07-28 08:46:32 | template_builder:343 - _customize_vm | DEBUG :: resize2fs 1.42.13 (17-May-2015) The filesystem is already 2593019 (4k) blocks long. Nothing to do!

W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/xenial/InRelease Could not connect to archive.ubuntu.com:80 (91.189.88.152), connection timed out [IP: 91.189.88.152 80] W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/xenial-updates/InRelease Unable to connect to archive.ubuntu.com:http: [IP: 91.189.88.152 80] W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/xenial-backports/InRelease Unable to connect to archive.ubuntu.com:http: [IP: 91.189.88.152 80] W: Some index files failed to download. They have been ignored, or old ones used instead. W: Failed to fetch https://download.docker.com/linux/ubuntu/dists/xenial/InRelease Resolving timed out after 20530 milliseconds W: Some index files failed to download. They have been ignored, or old ones used instead. E: Version '5:19.03.12~3-0~ubuntu-xenial' for 'docker-ce' was not found

21-07-28 08:46:32 | template_builder:347 - _customize_vm | DEBUG :: stdout: 21-07-28 08:46:32 | template_builder:350 - _customize_vm | DEBUG :: net.ipv6.conf.all.disable_ipv6 = 1 net.ipv6.conf.default.disable_ipv6 = 1 net.ipv6.conf.lo.disable_ipv6 = 1 NOCHANGE: partition 1 is size 20744159. it cannot be grown installing docker Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [109 kB] Get:2 http://security.ubuntu.com/ubuntu xenial-security/main Sources [252 kB] Get:3 http://security.ubuntu.com/ubuntu xenial-security/restricted Sources [2976 B] Get:4 http://security.ubuntu.com/ubuntu xenial-security/universe Sources [211 kB] Get:5 http://security.ubuntu.com/ubuntu xenial-security/multiverse Sources [4636 B] Get:6 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [1648 kB] Get:7 http://security.ubuntu.com/ubuntu xenial-security/main Translation-en [380 kB] Get:8 http://security.ubuntu.com/ubuntu xenial-security/restricted amd64 Packages [9824 B] Get:9 http://security.ubuntu.com/ubuntu xenial-security/restricted Translation-en [2152 B] Get:10 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages [785 kB] Get:11 http://security.ubuntu.com/ubuntu xenial-security/universe Translation-en [225 kB] Get:12 http://security.ubuntu.com/ubuntu xenial-security/multiverse amd64 Packages [7864 B] Get:13 http://security.ubuntu.com/ubuntu xenial-security/multiverse Translation-en [2672 B] Err:14 http://archive.ubuntu.com/ubuntu xenial InRelease Could not connect to archive.ubuntu.com:80 (91.189.88.152), connection timed out [IP: 91.189.88.152 80] Err:15 http://archive.ubuntu.com/ubuntu xenial-updates InRelease Unable to connect to archive.ubuntu.com:http: [IP: 91.189.88.152 80] Err:16 http://archive.ubuntu.com/ubuntu xenial-backports InRelease Unable to connect to archive.ubuntu.com:http: [IP: 91.189.88.152 80] Fetched 3640 kB in 213503982334601d 5h 2min 34s (0 B/s) Reading package lists... Reading package lists... Building dependency tree... Reading state information... The following package was automatically installed and is no longer required: grub-pc-bin Use 'apt autoremove' to remove it. The following additional packages will be installed: libcurl3-gnutls python3-software-properties The following packages will be upgraded: apt-transport-https ca-certificates curl libcurl3-gnutls python3-software-properties software-properties-common 6 upgraded, 0 newly installed, 0 to remove and 149 not upgraded. Need to get 532 kB of archives. After this operation, 31.7 kB disk space will be freed. Get:1 http://security.ubuntu.com/ubuntu xenial-security/main amd64 curl amd64 7.47.0-1ubuntu2.19 [139 kB] Get:2 http://security.ubuntu.com/ubuntu xenial-security/main amd64 libcurl3-gnutls amd64 7.47.0-1ubuntu2.19 [189 kB] Get:3 http://security.ubuntu.com/ubuntu xenial-security/main amd64 apt-transport-https amd64 1.2.32ubuntu0.2 [26.6 kB] Get:4 http://security.ubuntu.com/ubuntu xenial-security/main amd64 ca-certificates all 20210119~16.04.1 [148 kB] Get:5 http://security.ubuntu.com/ubuntu xenial-security/main amd64 software-properties-common all 0.96.20.10 [9504 B] Get:6 http://security.ubuntu.com/ubuntu xenial-security/main amd64 python3-software-properties all 0.96.20.10 [20.2 kB] Preconfiguring packages ... Fetched 532 kB in 1s (458 kB/s) (Reading database ... 54032 files and directories currently installed.) Preparing to unpack .../curl_7.47.0-1ubuntu2.19_amd64.deb ... Unpacking curl (7.47.0-1ubuntu2.19) over (7.47.0-1ubuntu2.7) ... Preparing to unpack .../libcurl3-gnutls_7.47.0-1ubuntu2.19_amd64.deb ... Unpacking libcurl3-gnutls:amd64 (7.47.0-1ubuntu2.19) over (7.47.0-1ubuntu2.7) ... Preparing to unpack .../apt-transport-https_1.2.32ubuntu0.2_amd64.deb ... Unpacking apt-transport-https (1.2.32ubuntu0.2) over (1.2.26) ... Preparing to unpack .../ca-certificates_20210119~16.04.1_all.deb ... Unpacking ca-certificates (20210119~16.04.1) over (20170717~16.04.1) ... Preparing to unpack .../software-properties-common_0.96.20.10_all.deb ... Unpacking software-properties-common (0.96.20.10) over (0.96.20.7) ... Preparing to unpack .../python3-software-properties_0.96.20.10_all.deb ... Unpacking python3-software-properties (0.96.20.10) over (0.96.20.7) ... Processing triggers for man-db (2.7.5-1) ... Processing triggers for libc-bin (2.23-0ubuntu10) ... Processing triggers for dbus (1.10.6-1ubuntu3.3) ... Setting up libcurl3-gnutls:amd64 (7.47.0-1ubuntu2.19) ... Setting up curl (7.47.0-1ubuntu2.19) ... Setting up apt-transport-https (1.2.32ubuntu0.2) ... Setting up ca-certificates (20210119~16.04.1) ... Setting up python3-software-properties (0.96.20.10) ... Setting up software-properties-common (0.96.20.10) ... Processing triggers for libc-bin (2.23-0ubuntu10) ... Processing triggers for ca-certificates (20210119~16.04.1) ... Updating certificates in /etc/ssl/certs... 31 added, 50 removed; done. Running hooks in /etc/ca-certificates/update.d... done. OK Hit:1 http://archive.ubuntu.com/ubuntu xenial InRelease Get:2 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [109 kB] Get:3 http://archive.ubuntu.com/ubuntu xenial-backports InRelease [107 kB] Get:4 http://archive.ubuntu.com/ubuntu xenial/main Sources [868 kB] Get:5 http://archive.ubuntu.com/ubuntu xenial/restricted Sources [4808 B] Get:6 http://archive.ubuntu.com/ubuntu xenial/universe Sources [7728 kB] Get:7 http://archive.ubuntu.com/ubuntu xenial/multiverse Sources [179 kB] Get:8 http://archive.ubuntu.com/ubuntu xenial/universe amd64 Packages [7532 kB] Get:9 http://archive.ubuntu.com/ubuntu xenial/universe Translation-en [4354 kB] Get:10 http://archive.ubuntu.com/ubuntu xenial/multiverse amd64 Packages [144 kB] Get:11 http://archive.ubuntu.com/ubuntu xenial/multiverse Translation-en [106 kB] Get:12 http://archive.ubuntu.com/ubuntu xenial-updates/main Sources [537 kB] Get:13 http://archive.ubuntu.com/ubuntu xenial-updates/restricted Sources [3608 B] Get:14 http://archive.ubuntu.com/ubuntu xenial-updates/universe Sources [444 kB] Get:15 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse Sources [12.3 kB] Get:16 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [2048 kB] Get:17 http://archive.ubuntu.com/ubuntu xenial-updates/main Translation-en [482 kB] Get:18 http://archive.ubuntu.com/ubuntu xenial-updates/restricted amd64 Packages [10.2 kB] Get:19 http://archive.ubuntu.com/ubuntu xenial-updates/restricted Translation-en [2272 B] Get:20 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [1219 kB] Get:21 http://archive.ubuntu.com/ubuntu xenial-updates/universe Translation-en [358 kB] Get:22 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse amd64 Packages [22.6 kB] Get:23 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse Translation-en [8476 B] Get:24 http://archive.ubuntu.com/ubuntu xenial-backports/main Sources [6720 B] Get:25 http://archive.ubuntu.com/ubuntu xenial-backports/universe Sources [11.0 kB] Get:26 http://archive.ubuntu.com/ubuntu xenial-backports/main amd64 Packages [9812 B] Get:27 http://archive.ubuntu.com/ubuntu xenial-backports/main Translation-en [4456 B] Get:28 http://archive.ubuntu.com/ubuntu xenial-backports/universe amd64 Packages [11.3 kB] Get:29 http://archive.ubuntu.com/ubuntu xenial-backports/universe Translation-en [4476 B] Hit:30 http://security.ubuntu.com/ubuntu xenial-security InRelease Err:31 https://download.docker.com/linux/ubuntu xenial InRelease Resolving timed out after 20530 milliseconds Fetched 26.3 MB in 20s (1282 kB/s) Reading package lists... Reading package lists... Building dependency tree... Reading state information...

21-07-28 08:46:32 | template_builder:357 - _customize_vm | ERROR :: Failed VM customization Result start=== [100, <Response [200]>, <Response [200]>] ===Result end NoneType: None 21-07-28 08:46:32 | configure_cse:1498 - upgrade_cse | ERROR :: CSE Installation Error Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/container_service_extension/configure_cse.py", line 1484, in upgrade_cse log_wire=log_wire) File "/usr/local/lib/python3.7/site-packages/container_service_extension/configure_cse.py", line 1660, in _upgrade_to_35 msg_update_callback=msg_update_callback) File "/usr/local/lib/python3.7/site-packages/container_service_extension/configure_cse.py", line 1115, in _install_all_templates msg_update_callback=msg_update_callback) File "/usr/local/lib/python3.7/site-packages/container_service_extension/configure_cse.py", line 1317, in _install_single_template retain_temp_vapp=retain_temp_vapp) File "/usr/local/lib/python3.7/site-packages/container_service_extension/template_builder.py", line 465, in build self._customize_vm(vapp, self.temp_vm_name) File "/usr/local/lib/python3.7/site-packages/container_service_extension/template_builder.py", line 359, in _customize_vm raise Exception(f"{msg}; Result: {result}") Exception: Failed VM customization; Result: [100, <Response [200]>, <Response [200]>]

sakthisunda commented 3 years ago

Hi, Noticed underlying cause Unable to connect to archive.ubuntu.com might cause the customization failure. If there is intermittent network issue in connecting to ubuntu.com, or ubuntu.com server is dropping connection, we usually notice this error at our end as well. We retry the upgrade command.

Can you please provide what CSE you are upgrading from say CSE source version (3.0.x) --> CSE target version(3.1), what VCD version and if possible configuration file with your secured data removed/obscured. It would be helpful to analyze or reproduce at our end.

Also, I noticed your upgrade process will also install all the supported templates, if -t option is omitted at the time of upgrade.

If your intention is to install specific template, you may want to use cse template install <template-name> <template-revision> -c <absoulte path of unencrypted configfile> -s or cse template install <template-name> <template-revision> -c <absoulte path of encrypted configfile>, then you may try upgrade. This way, upgrade process won't get stopped just for template creation failure.

Thanks