Provisioning of VM extension 'AzureDiskEncryptionForLinux' has timed out

[lawrence-c]

Similar to #383.

VM:

Linux / Ubuntu 16.04
Standard B2ms (2 vcpus, 8 GB memory)

$ waagent --version

WALinuxAgent-2.2.32.2 running on ubuntu 16.04
Python: 3.5.2
Goal state agent: 2.2.37

The VM has GitLab installed, however I've disabled it (gitlab-ctl stop) so that nothing should be running on the VM. The VM also has an external disk which is acting as an NFS share with another VM (not sure if this is relevant), and is mounted in /etc/fstab as so: UUID=<redacted> /media/<name> ext4 defaults 0 0

I attempt to encrypt the VM with the following command:

az vm encryption enable \
--resource-group "[resource-group]" \
--name [vm-name] \
--disk-encryption-keyvault /subscriptions/[subscription-id-guid]/resourceGroups/[resource-group-name]/providers/Microsoft.KeyVault/vaults/[keyvault-name] \
--key-encryption-key https://[keyvault-name].vault.azure.net/keys/[kekname]/[kek-unique-id] \
--volume-type all

and get the following error an hour and a half later:

Provisioning of VM extension 'AzureDiskEncryptionForLinux' has timed out. Extension installation may be taking too long, or extension status could not be obtained.

I've attempted installing the extension separately with:

az vm extension set \
--name AzureDiskEncryptionForLinux \
--publisher Microsoft.Azure.Security \
--vm-name [vm-name] \
--resource-group [resource-group]

and this also fails after an hour and a half with the error:

Deployment failed. Correlation ID: <id>. Provisioning of VM extension 'AzureDiskEncryptionForLinux' has timed out. Extension installation may be taking too long, or extension status could not be obtained.

I'm not sure if there's something already installed from a previous attempt of encryption which only got half removed? I've tried removing the Microsoft.Azure.Security__AzureDiskEncryptionForLinux__1.1.0.28.zip from /var/lib/waagent but to no avail.

Errors include...

VM has reported a failure when processing extension 'AzureDiskEncryptionForLinux'. Error message: "Failed to get status file [Errno 2] No such file or directory: '/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-1.1.0.25/status/4.status'".

and

VM has reported a failure when processing extension 'AzureDiskEncryptionForLinux'. Error message: "Failed to encrypt OS volume with error: [Errno 2] No such file or directory: '/mnt/azure_bek_disk/ddd789b1-c298-4152-9231-f81c5c5c8dfb', stack trace: Traceback (most recent call last):
  File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-1.1.0.25/main/handle.py", line 1648, in daemon_encrypt
    stamp_disks_with_settings(os_items_to_stamp, encryption_config)
  File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-1.1.0.25/main/handle.py", line 207, in stamp_disks_with_settings
    settings.remove_protector_file(new_protector_name)
  File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-1.1.0.25/main/EncryptionSettingsUtil.py", line 81, in remove_protector_file
    os.remove(CommonVariables.encryption_key_mount_point + '/' + protector_name)
OSError: [Errno 2] No such file or directory: '/mnt/azure_bek_disk/ddd789b1-c298-4152-9231-f81c5c5c8dfb'
, machine state: uninitialized".

Also could be related, whenever I try to install the AzureDiskEncryptionForLinux extension, I get new mail in /var/mail/root which says:

src kt = /etc/krb5.keytab
File /etc/krb5.keytab does not exist
/opt/omi/bin/support/ktstrip: 38: exit: Illegal number: 01

Any ideas what I can do to resolve this?

[lawrence-c]

I've also attempted the following:

$ systemctl stop walinuxagent
$ rm -rf /var/lib/waagent/Microsoft.Azure.Security*
$ rm -rf /var/log/azure/Microsoft.Azure.Security.AzureDiskEncryptionForLinux/
$ systemctl start walinux agent

and I'm now getting the following error when running

az vm extension set \
--name AzureDiskEncryptionForLinux \
--publisher Microsoft.Azure.Security \
--vm-name [vm-name] \
--resource-group [resource-group]

Deployment failed. Correlation ID: <id> Handler 'Microsoft.Azure.Security.AzureDiskEncryptionForLinux' has reported failure for VM Extension 'AzureDiskEncryptionForLinux' with terminal error code '1007' and error message: '[ExtensionError] Non-zero exit code: 1, extension_shim.sh -c main/handle.py --install
[stdout]
2019/03/12 09:41:34 ERROR:[Microsoft.Azure.Security.AzureDiskEncryptionForLinux-1.0]: JSON error processing settings file:
Traceback (most recent call last):
  File "main/handle.py", line 1924, in <module>
    main()
  File "main/handle.py", line 487, in main
    install()
  File "main/handle.py", line 62, in install
    hutil.do_parse_context('Install')
  File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-1.1.0.28/main/Utils/HandlerUtil.py", line 204, in do_parse_context
    _context = self.try_parse_context()
  File "/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-1.1.0.28/main/Utils/HandlerUtil.py", line 285, in try_parse_context
    public_settings_str = self._context._config['runtimeSettings'][0]['handlerSettings'].get('publicSettings')
TypeError: 'NoneType' object has no attribute '__getitem__'

[stderr]
'

[nathank1989]

Did you ever find a resolution to this issue?

[lawrence-c]

Nope...