search

vmware / container-service-extension

Container Service for VMware vCloud Director
https://vmware.github.io/container-service-extension
Other
77 stars 52 forks source link

TGK 1.6.0 not deployable - guestinfo.postcustomization.networkconfiguration.status is None #1403

Open benedikt-haug opened 1 year ago

benedikt-haug commented 1 year ago

Describe the bug

Dear developers,

TKG 1.6.0 deployments with kubernetes 1.23 don't work due to "Post guest customization phase guestinfo.postcustomization.networkconfiguration.status is None" timeout. The issue appears in both ubuntu and photon releases, same log output. I can't connect to the machine via SSH, given SSH is configured directly after this step in the setup. 1.5.4 deploys fine, but kubernetes 1.22 reached end of life on the 28 Oct 2022, so isn't really an option

Could it be that its currently not yet supported, given the documentation only states 1.5.4?

CSE version 3.1.4

If there is additional info you could want, please tell me <3

Deployed config (grabbed from the webinterfaces POST call and redacted orgName, virtualDataCenterName and site):

{"apiVersion":"cse.vmware.com/v2.0","kind":"TKGm","metadata":{"additionalProperties":true,"orgName":"test","virtualDataCenterName":"test-dc","name":"version160test","site":"https://myemployer.xyz"},"spec":{"additionalProperties":true,"topology":{"controlPlane":{"count":1,"sizingClass":"System Default","storageProfile":"mass_FTT-2","cpu":null,"memory":null},"workers":{"count":2,"sizingClass":"System Default","storageProfile":"mass_FTT-2","cpu":null,"memory":null},"nfs":{"count":0,"sizingClass":null,"storageProfile":null}},"settings":{"ovdcNetwork":"vmwareKannKeinEscaping","sshKey":"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOULTplVav8t/WR5ZVvZQWkTRBeCR9OQQ0WnCr+TTHXD ubuntu@cse1","rollbackOnFailure":true,"network":{"expose":true,"pods":{"cidrBlocks":["100.96.0.0/11"]},"services":{"cidrBlocks":["100.64.0.0/13"]}}},"distribution":{"templateName":"ubuntu-2004-kube-v1.23.8+vmware.2-tkg.1-85a434f93857371fccb566a414462981","templateRevision":1}}}

How this looks ~/.cse-logs/cse-server-debug.log:

22-10-31 09:35:34 | pyvcloud_utils:761 - wait_for_completion_of_post_customization_procedure | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | INFO :: Post guest customization phase guestinfo.postcustomization.networkconfiguration.status is None
22-10-31 09:35:39 | pyvcloud_utils:761 - wait_for_completion_of_post_customization_procedure | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | INFO :: Post guest customization phase guestinfo.postcustomization.networkconfiguration.status is None
22-10-31 09:35:44 | pyvcloud_utils:761 - wait_for_completion_of_post_customization_procedure | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | INFO :: Post guest customization phase guestinfo.postcustomization.networkconfiguration.status is None
22-10-31 09:35:49 | pyvcloud_utils:761 - wait_for_completion_of_post_customization_procedure | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | INFO :: Post guest customization phase guestinfo.postcustomization.networkconfiguration.status is None
22-10-31 09:35:54 | pyvcloud_utils:761 - wait_for_completion_of_post_customization_procedure | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | INFO :: Post guest customization phase guestinfo.postcustomization.networkconfiguration.status is None
22-10-31 09:36:00 | pyvcloud_utils:761 - wait_for_completion_of_post_customization_procedure | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | INFO :: Post guest customization phase guestinfo.postcustomization.networkconfiguration.status is None
22-10-31 09:36:05 | pyvcloud_utils:761 - wait_for_completion_of_post_customization_procedure | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | INFO :: Post guest customization phase guestinfo.postcustomization.networkconfiguration.status is None
22-10-31 09:36:10 | pyvcloud_utils:761 - wait_for_completion_of_post_customization_procedure | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | INFO :: Post guest customization phase guestinfo.postcustomization.networkconfiguration.status is None
22-10-31 09:36:15 | pyvcloud_utils:761 - wait_for_completion_of_post_customization_procedure | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | INFO :: Post guest customization phase guestinfo.postcustomization.networkconfiguration.status is None
22-10-31 09:36:20 | pyvcloud_utils:761 - wait_for_completion_of_post_customization_procedure | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | INFO :: Post guest customization phase guestinfo.postcustomization.networkconfiguration.status is None
22-10-31 09:36:25 | pyvcloud_utils:761 - wait_for_completion_of_post_customization_procedure | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | INFO :: Post guest customization phase guestinfo.postcustomization.networkconfiguration.status is None
22-10-31 09:36:30 | pyvcloud_utils:761 - wait_for_completion_of_post_customization_procedure | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | INFO :: Post guest customization phase guestinfo.postcustomization.networkconfiguration.status is None
22-10-31 09:36:35 | pyvcloud_utils:761 - wait_for_completion_of_post_customization_procedure | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | INFO :: Post guest customization phase guestinfo.postcustomization.networkconfiguration.status is None
22-10-31 09:36:40 | pyvcloud_utils:761 - wait_for_completion_of_post_customization_procedure | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | INFO :: Post guest customization phase guestinfo.postcustomization.networkconfiguration.status is None
22-10-31 09:36:46 | pyvcloud_utils:761 - wait_for_completion_of_post_customization_procedure | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | INFO :: Post guest customization phase guestinfo.postcustomization.networkconfiguration.status
 is None
22-10-31 09:36:51 | pyvcloud_utils:761 - wait_for_completion_of_post_customization_procedure | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | INFO :: Post guest customization phase guestinfo.postcustomization.networkconfiguration.status is None
22-10-31 09:36:56 | pyvcloud_utils:761 - wait_for_completion_of_post_customization_procedure | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | INFO :: Post guest customization phase guestinfo.postcustomization.networkconfiguration.status is None
22-10-31 09:37:01 | pyvcloud_utils:761 - wait_for_completion_of_post_customization_procedure | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | INFO :: Post guest customization phase guestinfo.postcustomization.networkconfiguration.status is None
22-10-31 09:37:06 | pyvcloud_utils:761 - wait_for_completion_of_post_customization_procedure | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | INFO :: Post guest customization phase guestinfo.postcustomization.networkconfiguration.status is None
22-10-31 09:37:06 | pyvcloud_utils:776 - wait_for_completion_of_post_customization_procedure | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | ERROR :: VM Post guest customization failed due to timeout(600 sec)
22-10-31 09:37:06 | cluster_service_2_x_tkgm:2516 - _add_control_plane_nodes | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | ERROR ::
Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.10/site-packages/container_service_extension/rde/backend/cluster_service_2_x_tkgm.py", line 2499, in _add_control_plane_nodes
    vcd_utils.wait_for_completion_of_post_customization_procedure(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/container_service_extension/common/utils/pyvcloud_utils.py", line 777, in wait_for_completion_of_post_customization_procedure
    raise exceptions.PostCustomizationTimeoutError
container_service_extension.exception.exceptions.PostCustomizationTimeoutError
22-10-31 09:37:06 | cluster_service_2_x_tkgm:922 - _create_cluster_async | Request Id: 061596d2-7b69-4cac-989b-63910c566b41 | ERROR :: failure on creating nodes ['mstr-32t7']
Error:
Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.10/site-packages/container_service_extension/rde/backend/cluster_service_2_x_tkgm.py", line 2499, in _add_control_plane_nodes
    vcd_utils.wait_for_completion_of_post_customization_procedure(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/container_service_extension/common/utils/pyvcloud_utils.py", line 777, in wait_for_completion_of_post_customization_procedure
    raise exceptions.PostCustomizationTimeoutError
container_service_extension.exception.exceptions.PostCustomizationTimeoutError

During handling of the above exception, another exception occurred:

Reproduction steps

  1. Start a ubuntu-2004-kube-v1.23.8+vmware.2-tkg.1-85a434f93857371fccb566a414462981 (TKG 1.6.0) deployment. Same issue appears with the v1.23 photon image.
  2. It gets deleted after a few minutes without succeeding
  3. Check logs ...

Expected behavior

It should finish successfully and provide a useable cluster.

Additional context

No response

kutukcub commented 1 year ago

I have an same issue in CSE 3.1.3 environment with VCD 10.3.3. While i was creating TKGm cluster, I was facing an issue the cluster creation failed in a minutes and deleting immediately itself. Also I can create Native cluster and there is no issue. I checked the FW there is no drop or disable rule, actually I can not see any action in FW while TKGm cluster was creating.

Error : Error adding control plane node: failure on creating nodes [‘mstr-d69s’] Error: /root/control_plane.sh: wget -nv github.com/vmware-tanzu/carvel-imgpkg/releases/download/v0.24.0/imgpkg-linux-amd64 -O $imgpkg_path

benedikt-haug commented 1 year ago

It would be nice if one could check if compatibility is intended: grafik