Pyoxidizer musl execution on VMware ESXi

[Zawadidone]

Using the Pyoxidizer configuration (https://github.com/fox-it/acquire/pull/109) I was able to build a static musl binary, but when executing the binary on VMware ESXi 7 the execution fails while it works on the Docker image (quay.io/pypa/manylinux2014_x86_64).

The error as shown below is triggered because it cannot obtain the current path of the executable, because /self/proc/exe (https://github.com/indygreg/PyOxidizer/blob/b78b0cb75f4317c45408bbc9a569c062c482c679/pyembed/src/config.rs#L478) is not available on VMware ESXi 7.

I don't understand well enough how Pyoxidizer works to determine what causes this error and how this issue can be resolved, but I will look into this.

*VMWare ESXi 7*

vmware -v
VMware ESXi 7.0.3 build-21930508

./acquire --help
error instantiating embedded Python interpreter: could not obtain current executable

strace ./acquire
[...]
mmap(0x7221612000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7221612000
readlink("/proc/self/exe", 0x7221612000, 256) = -1 EINVAL (Invalid argument)
[...]

ls -al /proc/self/exe
ls: /proc/self/exe: No such file or directory

*quay.io/pypa/manylinux2014_x86_64*

acquire --help
[...]
If no options are given, the collection profile 'default' is used.

ls -al /proc/self/exe
lrwxrwxrwx 1 root root 0 Nov 20 16:23 /proc/self/exe -> /usr/bin/ls

*Alpine Linux 3.18*

acquire --help
[...]
If no options are given, the collection profile 'default' is used.

strace ./acquire --help
[...]
readlink("/proc/self/exe", "/tmp/acquire", 256) = 12
open("/tmp/acquire", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_PATH) = 3
readlink("/proc/self/fd/3", "/tmp/acquire", 4095) = 12
fstat(3, {st_mode=S_IFREG|0750, st_size=112942000, ...}) = 0
stat("/tmp/acquire", {st_mode=S_IFREG|0750, st_size=112942000, ...}) = 0
close(3)
[...]

ls -al /proc/self/exe
lrwxrwxrwx    1 root     root             0 Nov 20 16:37 /proc/self/exe -> /bin/busybox

[Schamper]

I haven't tested on ESXi yet, so I also didn't expect it to work out of the box. I don't have the time to work on this myself right now, so if you're willing to look into some things, that'd be appreciated!

Looks like there might be some possibility of modifying the Python interpreter configuration from the PyOxidizer configuration file: https://pyoxidizer.readthedocs.io/en/stable/pyoxidizer_config_type_python_interpreter_config.html#starlark_pyoxidizer.PythonInterpreterConfig.executable Perhaps setting it to an empty string (or just python) will progress it further.

We've already tackled this issue on our internal Python interpreter, so I'll see if I can dig up what worked there.

As a temporary workaround, you may also try to temporarily enable the proc filesystem on ESXi:

vsish -e set /config/User/intOpts/UserProcEnable 1

[Zawadidone]

Both suggestions did not resolve the issue, because the second argument given to readlink is invalid.

I am looking into the option to manually set the variable exe in the PyOxidizer configuration file:

https://pyoxidizer.readthedocs.io/en/stable/pyembed_interpreter_config.html#exe-field https://github.com/indygreg/PyOxidizer/blob/b78b0cb75f4317c45408bbc9a569c062c482c679/pyembed/src/config.rs#L82-L91

Or adjusting the code to do this dynamically with support for VMware ESXi.

[Schamper]

Looks like we set this to sys.argv[0] in our own variant.

I expect that some patching will be required in order to run PyOxidizer on ESXi. We can probably host those under https://github.com/fox-it.

[Zawadidone]

The function std::env::current_exe() used by Pyoxidizer (https://github.com/indygreg/PyOxidizer/blob/b78b0cb75f4317c45408bbc9a569c062c482c679/pyembed/src/config.rs#L477) is not supported on VMware ESXi (https://github.com/indygreg/PyOxidizer/blob/b78b0cb75f4317c45408bbc9a569c062c482c679/pyembed/src/config.rs#L477, https://github.com/rust-lang/rust/blob/cc1130732d1b246d1bb11890a063878f68ebb0f5/library/std/src/env.rs#L688, https://github.com/rust-lang/rust/blob/cc1130732d1b246d1bb11890a063878f68ebb0f5/library/std/src/sys/unix/os.rs#L413, https://github.com/rust-lang/rust/blob/master/library/std/src/fs.rs#L2160, https://github.com/rust-lang/rust/blob/master/library/std/src/sys/unix/fs.rs#L1608)

main.rs
// https://doc.rust-lang.org/std/env/fn.current_exe.html
// rustup target add x86_64-unknown-linux-musl
// rustc -g --target=x86_64-unknown-linux-musl main.rs 
use std::env::consts::OS;

fn main() {
    println!("The OS is: {}", OS);
    let _exe_path = std::env::current_exe();
    println!("Path of this executable is: {}", _exe_path.expect("REASON").display());
}

# VMware ESXi execution
RUST_BACKTRACE=full ./main 
The OS is: linux
thread 'main' panicked at main.rs:10:58:
REASON: Os { code: 22, kind: InvalidInput, message: "Invalid argument" }
[...]

strace ./main 
[...]
readlink("/proc/self/exe", 0x96d0ead840, 256) = -1 EINVAL (Invalid argument)
[...]

As far as I known this can only be resolved by adding VMware ESXi support to Rust and the underlying libraries for the functions used by Pyoxidizer to setup the Python interpreter (https://github.com/rust-lang/rust/issues/41347)

[Schamper]

I have not extensively tested this yet, but I believe this should work too: https://github.com/fox-it/PyOxidizer/commit/8a0a13ca50c24bc1dd45ec9fd23e2b89dd45a273