search

vmware / open-vm-tools

Official repository of VMware open-vm-tools project
http://sourceforge.net/projects/open-vm-tools/
2.23k stars 425 forks source link

failed to exec /usr/bin/cloud-init, cloud-init is not installed #596

Open vangarg opened 2 years ago

vangarg commented 2 years ago

Describe the bug

I am trying to clone a VM on vSphere and customize it using cloud-init. I am using OVF datasource for it. I have created an ISO file which I attact to my VM. Then I power on but I do not see any event on vSphere logs. I should have seen CustomizationStarted event and Customization Succeeded event in the logs.

Reproduction steps

1. Add disable_vmware_customisation: false in /etc/cloud/cloud.cfg
2. Create an Ubuntu 20.04 VM from the above template
3. Create and ISO image using genisoimage
4. Mount it on the VM
5. Power On the VM
...

Expected behavior

Events CustomizationStarted and CustomizationSucceeded/Customization failed should appear on vCenter events.

Additional context

No response

chengcheng-chcheng commented 2 years ago

Hi vangarg, From the "Reproduction steps 1. Add disable_vmware_customisation:false in /etc/cloud/cloud.cfg" I didn't find a space before false. Could you try add the following entry in /etc/cloud/cloud.cfg, notice there is a space before false disable_vmware_customization: false

Also, could you try "cloud-init -v" to see if the cloud-init is already installed?

chengcheng-chcheng commented 2 years ago

If the issue still occurs after confirming that cloud-init has been installed and the "disable_vmware_customization: false" has set, pls share the /var/log/vmware-imc/toolsDeployPkg.log and /var/log/cloud-init.log to further investigating.

vangarg commented 2 years ago

Hi chengcheng-chcheng, I have added the line correctly in/etc/cloud/cloud.cfg with a space. Also cloud-init is installed and ran successfully. But the problem that I am facing is my /var/log/vmware-imc/toolsDeployPkg.log file is not getting created. I have attached my /var/log/cloud-init.log file here. https://microsoftapc-my.sharepoint.com/:u:/g/personal/t-vangarg_microsoft_com/ETp0DgqGoR5PjUy3fDSCGigBsdoiEeIAIIERo-BL79Up-g?e=bQjues My guest OS is ubuntu 20.04 and I have enabling this also, but nothing is working: sudo vmware-toolbox-cmd config set deployPkg enable-custom-scripts true

chengcheng-chcheng commented 2 years ago

From the log in the cloud-init.log, it seems the cust.cfg file was not copied to the path. May I ask what is the version of VMTools?(the version can be checked by command "vmtoolsd -v") 2022-06-10 05:34:40,186 - DataSourceOVF.py[DEBUG]: Did not find VMware Customization Config File

chengcheng-chcheng commented 2 years ago

Hi vangarg, Just found that in the 'Reproduction steps', there is no step to create a customization specification and customize the vm by using the specification. So you seems did not use the vmware customization method to customize the vm, but used the ISO method.

chengcheng-chcheng commented 2 years ago

You can create the VM Customization Specifications from the 'Policies and Profiles' in the vCenter. And then choose the vm, right click the VM, from Guest OS->Customize Guest OS to customize with the Specification just created.

vangarg commented 2 years ago

Hi, The version of Open VM Tools that I am using is image

vangarg commented 2 years ago

https://github.com/vmware/open-vm-tools/issues/596#issuecomment-1152306473 I also tried as you mentioned in this comment. The expected behaviour should be two events in VMware logs like this. image Instead, what I am getting is image

vangarg commented 2 years ago

I am attaching both the log files that got creating in this case.[Here, I in addition to following your comment https://github.com/vmware/open-vm-tools/issues/596#issuecomment-1152306473, also mounted ISO on the VM that I mentioned in reproduction step and powered-on the VM.]. toolsDeployPkg.log cloud-init.log

chengcheng-chcheng commented 2 years ago

From the toolsDeployPkg.log it showed as below which means as cloud-init is disabled, it executed traditional GOSC workflow, not the expected cloud-init customization workflow. So it need to check why cloud-init is disabled. You can check whether the file /etc/cloud/cloud-init.disabled exists, if it exists pls remove it, or if in /proc/cmdline contains cloud-init=disabled, if contains pls remove this line.

logs: [2022-06-10T12:59:14.304Z] [ info] cloud-init is installed. [2022-06-10T12:59:14.304Z] [ info] Checking if cloud.cfg exists and if cloud-init is enabled. [2022-06-10T12:59:14.305Z] [ info] UseCloudInitWorkflow return: 4 [2022-06-10T12:59:14.305Z] [ info] Executing traditional GOSC workflow. [2022-06-10T12:59:14.305Z] [ debug] Command to exec : '/usr/bin/perl'.

vangarg commented 2 years ago

This is my proc/cmdline file. image It does not have cloud-init=disabled line. Also, I do not have cloud-init.disabled file in /etc/cloud or /etc/cloud/cloud.cfg*. Is there anything else I can do to check how is this cloud-init disabled?

vangarg commented 2 years ago

Also, this is my /run/cloud-init/cloud-init-generator.log file. image According to this, it looks like cloud-init is enabled.

chengcheng-chcheng commented 2 years ago

Hi vangarg, Could you double check whether "disable_vmware_customization: false" exists in the /etc/cloud/cloud.cfg after the customization operation?

chengcheng-chcheng commented 2 years ago

For the cabinet file, I can extract it and got 4 files. You can use decompression software like 7-zip to extract it.

vangarg commented 2 years ago

Thanks a lot, this https://github.com/vmware/open-vm-tools/issues/596#issuecomment-1153432937 worked. Also, what was the file format of this cabinet file and how can I recompress it?

chengcheng-chcheng commented 2 years ago

The cab file was created when you triggered the customization guest from the vCenter, the cab was created by the libCreatePkg.so in the vCenter. The cab will be extracted into the guest by vmware Tools, if you want to check the content of the cab, as above mentioned you can extract it throw the decompression software, but it can not be recompress by decompression softwares.

May I ask whether the cloud-init customization work? Did you double check whether "disable_vmware_customization: false" exists in the /etc/cloud/cloud.cfg after the customization operation?

vangarg commented 2 years ago

Hi, Thanks, cloud-init customization worked. My file /etc/cloud/cloud.cfg file had "disable_vmware_customisation: false" instead of "disable_vmware_customization: false". As per you last comment, is there a way to manually create this cab file using libCreatePkg.so in the vCenter?

chengcheng-chcheng commented 2 years ago

/usr/lib/vmware-vpx/imgcust/linux/bin64/deployPkg in the vcenter can help to create cab file. It needs to run like this ./deployPkg -script $SCRIPT_PATH -config $CONFIG_PATH -create lintest.cab -os linux

If you only want to get a .cab file, maybe revert the snapshot and trigger customization again and do not power on the vm, you can get a new .cab file again.

vangarg commented 2 years ago

Hi, I am trying to create VM's on vcenter and then customize them using cloud-init. This is my main goal. Now so far, I am successfully able to customize VMs using cloud-init, but the problem that I am facing is that I found no way to track whether or not cloud-init ran successfully from vcenter events in vcenter version 6.7. Enabling cloud-init customization by adding line "disable_vmware_customization: false" in /etc/cloud/cloud.cfg helps me somewhat here as I am able to get CustomizationStarted and CustomizationSucceeded event in VMware events but they does not give me CustomizationFailed event whenever it fails due to some reason. And, for testing in my code, I also need event like CustomizationFailed. The reasons might be cloud-init is not installed in the guest, or VMware tools are not installed, some modules in the user-data script didn't run successfully, or something else, basically anything that might give a customization failure. Is there some way I can do that? I searched web, looked so many blogs but nothing has helped so far.

chengcheng-chcheng commented 2 years ago

Hi vangarg, To simulate the error to get CustomizationFailed event based on cloud-init customization. ./deployPkg is not a good choice, it depends on many constraints. Maybe you can directly change the cloud-init code on your environment to send the CustomizationFailed event by yourself. The cloud-init code can be found in the path in the vm: /usr/lib/python3/dist-packages/cloudinit/sources/DataSourceOVF.py you can add "GuestCustEvent.GUESTCUST_EVENT_CUSTOMIZE_FAILED" in the code where it is needed to send the event. Details pls also refer to https://github.com/canonical/cloud-init/blob/main/cloudinit/sources/DataSourceOVF.py

vangarg commented 2 years ago

Hi @chengcheng-chcheng, Thanks for the suggestion. Just wanted to confirm that is there no other way out to know that some cloud-init module failed to run from outside the guest other than changing Cloud-init's code to send the CustomizationFailed event? Like when I trigger Customization that will be executing the cloud-init workflow from the vCenter, till what stage(of cloud-init) can I be sure of that cloud-init ran successfully by looking at the CustomizationSucceeded event that I am currently getting on vCenter events?