Datasource vmware vmware-rpctool Permission denied

playerla commented 1 year ago

Hi,

Following a security configuration https://github.com/vmware/open-vm-tools/issues/690 (metadata, vendordata or userdata are all readable by any local user with the defautl configuration...) I learn that :

rpctool does not support secure RPCs, and we'd like to deprecate it. 'vmtoolsd --cmd' does support secure RPCs, and is the preferred way to execute RPCs.

I'm know stuck with a datasource vmware vmware-rpctool Permission denied : https://github.com/canonical/cloud-init/blob/97e09dfea2d74ef583bd5650b5c6e40aa19fbb24/cloudinit/sources/DataSourceVMware.py#L92

In what extend rpctool could be replaced with vmtoolsd ? (optional switch, impact or even deprecation?)

TheRealFalcon commented 1 year ago

@playerla , do you have documentation for how vmtoolsd is supposed to be used? If it is meant to be a replacement for rpctools, then it makes sense to update in the VMWare datasource.

cc: @sshedi and @akutz

akutz commented 1 year ago

This was raised internally as well. The tl;dr:

Context

The tool vmware-rpctool is failing due to a secure RPC feature implemented in ESXi that currently is only supported by vmtoolsd.
Why did we switch from vmtoolsd to vmware-rpctool in the first place? For that please see https://github.com/vmware-archive/cloud-init-vmware-guestinfo/pull/23
- vmtoolsd was subject to a performance issue that was patched on the ESXi side an in open-vm-tools
- It is not reasonsable for the guest to know whether the hypervisor is running with that patch, not without using the same, potentially affected command to query that information in the first place...

Short-term solution

@ravindravmw recommended that if vmware-rpctool fail, we fall back to vmtoolsd
My concern was that returning no data is a perfectly valid scenario, so failing has to be based on a specific condition, not just a non-zero exit code
We decided to fall back to vmtoolsd if vmware-rpctool returns a non-zero exit code and writes Permission denied to stderr.

Long-term plan

In the long-run, we want to move back to vmtoolsd anyway as it supports other features new in vSphere 8.0, such as datasets, a more robust data transport model than guestinfo.
However, again, Cloud-Init and the guest won't know what the host supports, so we need to develop a migration/fall-back strategy for vmware-rpctool --> vmtoolsd that:
- Does not cause performance issues on older hosts
- Allows the use of secure RPC and datasets on new hosts

Action Items

[ ] @ravindravmw Can the secure RPC feature can be added to vmware-rpctool so it is no longer subject to permission denied errors.
[ ] @ravindravmw Can vmware-rpctool be updated to support datasets?

TheRealFalcon commented 1 year ago

Thanks @akutz , that helps a lot. Given this, I agree that a fallback to vmtoolsd makes the most sense currently.

akutz commented 1 year ago

Thanks @akutz , that helps a lot. Given this, I agree that a fallback to vmtoolsd makes the most sense currently.

Agreed. And I would have honestly submitted a PR by now, but have not for the same reason I have not added support for datasets. If it was just in the DS, that would be one thing, but ds_identify has to be updated as well. I've been wracking my brain to try and figure out a way to unify the two approaches...

TheRealFalcon commented 1 year ago

Yeah, I have noticed the VMWare section of ds-identify is a little more complicated than some other datasources. Is there nothing else within the environment to identify an instance as being on VMWare/vSphere?

akutz commented 1 year ago

Perhaps, but I should also address the OvfEnv DS as well, which this affects since it also uses vmware-rpctool. I'm working on it now. I'll put something up sans tests for folks to review, and then work on, or get someone to help work on, the tests for it.

akutz commented 1 year ago

Okay, the ds-identify update is complete at least. The two commands are basically plug-and-play, with the exception of the need for the --cmd flag for vmtoolsd:

diff --git a/tools/ds-identify b/tools/ds-identify
index 16376ef05..4f9212278 100755
--- a/tools/ds-identify
+++ b/tools/ds-identify
@@ -972,14 +972,78 @@ vmware_guest_customization() {
     return 1
 }

+vmware_has_rpctool() {
+    command -v vmware-rpctool >/dev/null 2>&1
+}
+
+vmware_rpctool_guestinfo() {
+    vmware-rpctool "info-get guestinfo.${1}" 2>/dev/null | grep "[[:alnum:]]"
+}
+
+vmware_rpctool_guestinfo_err() {
+    vmware-rpctool "info-get guestinfo.${1}" 2>&1 | grep "[[:alnum:]]"
+}
+
+vmware_has_vmtoolsd() {
+    command -v vmtoolsd >/dev/null 2>&1
+}
+
+vmware_vmtoolsd_guestinfo() {
+    vmtoolsd --cmd "info-get guestinfo.${1}" 2>/dev/null | grep "[[:alnum:]]"
+}
+
+vmware_vmtoolsd_guestinfo_err() {
+    vmtoolsd --cmd "info-get guestinfo.${1}" 2>&1 | grep "[[:alnum:]]"
+}
+
+vmware_guestinfo() {
+    vmware_rpctool_guestinfo "${1}" || vmware_vmtoolsd_guestinfo "${1}"
+}
+
+vmware_guestinfo_err() {
+    vmware_rpctool_guestinfo_err "${1}" || vmware_vmtoolsd_guestinfo_err "${1}"
+}
+
+vmware_guestinfo_ovfenv() {
+    if [ "${1}" = "err" ]; then
+      vmware_guestinfo_err "ovfEnv"
+    else
+      vmware_guestinfo "ovfEnv"
+    fi
+}
+
+vmware_guestinfo_metadata() {
+    if [ "${1}" = "err" ]; then
+      vmware_guestinfo_err "metadata"
+    else
+      vmware_guestinfo "metadata"
+    fi
+}
+
+vmware_guestinfo_userdata() {
+    if [ "${1}" = "err" ]; then
+      vmware_guestinfo_err "userdata"
+    else
+      vmware_guestinfo "userdata"
+    fi
+}
+
+vmware_guestinfo_vendordata() {
+    if [ "${1}" = "err" ]; then
+      vmware_guestinfo_err "vendordata"
+    else
+      vmware_guestinfo "vendordata"
+    fi
+}
+
 ovf_vmware_transport_guestinfo() {
     [ "${DI_VIRT}" = "vmware" ] || return 1
-    command -v vmware-rpctool >/dev/null 2>&1 || return 1
+    vmware_has_rpctool || vmware_has_vmtoolsd || return 1
     local out="" ret=""
-    out=$(vmware-rpctool "info-get guestinfo.ovfEnv" 2>&1)
+    out=$(vmware_guestinfo_ovfenv "err")
     ret=$?
     if [ $ret -ne 0 ]; then
-        debug 1 "Running on vmware but rpctool query returned $ret: $out"
+        debug 1 "Running on vmware but query returned $ret: $out"
         return 1
     fi
     case "$out" in
@@ -1439,26 +1503,6 @@ vmware_has_envvar_vmx_guestinfo_vendordata() {
     [ -n "${VMX_GUESTINFO_VENDORDATA:-}" ]
 }

-vmware_has_rpctool() {
-    command -v vmware-rpctool >/dev/null 2>&1
-}
-
-vmware_rpctool_guestinfo() {
-    vmware-rpctool "info-get guestinfo.${1}" 2>/dev/null | grep "[[:alnum:]]"
-}
-
-vmware_rpctool_guestinfo_metadata() {
-    vmware_rpctool_guestinfo "metadata"
-}
-
-vmware_rpctool_guestinfo_userdata() {
-    vmware_rpctool_guestinfo "userdata"
-}
-
-vmware_rpctool_guestinfo_vendordata() {
-    vmware_rpctool_guestinfo "vendordata"
-}
-
 dscheck_VMware() {
     # Checks to see if there is valid data for the VMware datasource.
     # The data transports are checked in the following order:
@@ -1486,16 +1530,16 @@ dscheck_VMware() {
         return "${DS_NOT_FOUND}"
     fi

-    # Do not proceed if the vmware-rpctool command is not present.
-    if ! vmware_has_rpctool; then
+    # Do not proceed if neither the vmware-rpctool or vmtoolsd command exists.
+    if ! vmware_has_rpctool && ! vmware_has_vmtoolsd; then
         return "${DS_NOT_FOUND}"
     fi

     # Activate the VMware datasource only if any of the fields used
     # by the datasource are present in the guestinfo table.
-    if { vmware_rpctool_guestinfo_metadata || \
-         vmware_rpctool_guestinfo_userdata || \
-         vmware_rpctool_guestinfo_vendordata; } >/dev/null 2>&1; then
+    if { vmware_guestinfo_metadata || \
+         vmware_guestinfo_userdata || \
+         vmware_guestinfo_vendordata; } >/dev/null 2>&1; then
         return "${DS_FOUND}"
     fi

akutz commented 1 year ago

Yeah, I have noticed the VMWare section of ds-identify is a little more complicated than some other datasources. Is there nothing else within the environment to identify an instance as being on VMWare/vSphere?

@TheRealFalcon I remember the other reason it is complicated. As I am looking at the tests for ds_identify, validating the VMware logic required distinct functions for each property to mock, hence all the funcs.

cpaelzer commented 1 year ago

Yeah, I have noticed the VMWare section of ds-identify is a little more complicated than some other datasources. Is there nothing else within the environment to identify an instance as being on VMWare/vSphere?

If it is, as usual, just about identifying "on vmware" then it is about "Mechanisms to determine if software is running in a VMware virtual machine" which e.g. systemd-detect-virt already uses.

But that will only tell you where you are, not give you any config data. Furthermore you already have that as detect_virt uses this and vmware_guest_customization is first checking [ "${DI_VIRT}" = "vmware" ].

Only later in dscheck_VMware comes all the complexity as there are various combinations how it could be configured and therefore of fetching that data on that platform.

playerla commented 1 year ago

Thank you for you reactivity ! I just get the daily build but still Permission denied. I can't figure why. Open up #4475

canonical / cloud-init