Postgres | Error opening executable

[dzygann]

Hi,

we're trying to run Postgres in Graphene. OS is Ubuntu 20.04.

To that end we created our own Dockerfile:

# syntax=docker/dockerfile:1
FROM ubuntu:18.04

# Install ``python-software-properties``, ``software-properties-common`` and PostgreSQL 9.3
#  There are some warnings (in red) that show up during the build. You can hide
#  them by prefixing each apt-get statement with DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y software-properties-common postgresql-10 postgresql-client-10 postgresql-contrib-10

# Note: The official Debian and Ubuntu images automatically ``apt-get clean``
# after each ``apt-get``

# Run the rest of the commands as the ``postgres`` user created by the ``postgres-9.3`` package when it was ``apt-get installed``
USER postgres

# Create a PostgreSQL role named ``docker`` with ``docker`` as the password and
# then create a database `docker` owned by the ``docker`` role.
# Note: here we use ``&&\`` to run commands one after the other - the ``\``
#       allows the RUN command to span multiple lines.
RUN    /etc/init.d/postgresql start &&\
    psql --command "CREATE USER docker WITH SUPERUSER PASSWORD 'docker';" &&\
    createdb -O docker docker

# Adjust PostgreSQL configuration so that remote connections to the
# database are possible.
RUN echo "host all  all    0.0.0.0/0  md5" >> /etc/postgresql/10/main/pg_hba.conf

# And add ``listen_addresses`` to ``/etc/postgresql/9.3/main/postgresql.conf``
RUN echo "listen_addresses='*'" >> /etc/postgresql/10/main/postgresql.conf

# Expose the PostgreSQL port
EXPOSE 5432

# Add VOLUMEs to allow backup of config, logs and databases
VOLUME  ["/etc/postgresql", "/var/log/postgresql", "/var/lib/postgresql"]

# Set the default command to run when starting the container
CMD ["/usr/lib/postgresql/10/bin/postgres", "-D", "/var/lib/postgresql/10/main", "-c", "config_file=/etc/postgresql/10/main/postgresql.conf"]

Which we then build and signed using gsc. However when starting the image we encounter the following problem:

[P9:postgres] debug: Host: Linux-SGX
[P9:postgres] debug: LibOS xsave_enabled 1, xsave_size 0x440(1088), xsave_features 0x1f
[P9:postgres] debug: Initial VMA region 0xfb2a2000-0xfb545000 (LibOS) bookkeeped
[P9:postgres] debug: Initial VMA region 0xffce6000-0x100000000 (manifest) bookkeeped
[P9:postgres] debug: ASLR top address adjusted to 0x38c53000
[P9:postgres] debug: Shim loaded at 0xfb2a2000, ready to initialize
[P9:postgres] debug: Mounting root as chroot filesystem: from file:/ to /
[P9:postgres] debug: Mounting special proc filesystem: /proc
[P9:postgres] debug: Mounting special dev filesystem: /dev
[P9:postgres] debug: Mounting terminal device /dev/tty under /dev
[P9:postgres] debug: Mounting special sys filesystem: /sys
[P9:T1:postgres] error: init_exec_handle: cannot find executable in filesystem: -2
[P9:T1:postgres] error: Error during shim_init() in init_important_handles (-2)
debug: DkProcessExit: Returning exit code 2

Remembering what dimakuv said in a previous issue, we took a step back to try again without gsc and only graphene-direct using the memcached example as a base. During the Postgres server startup we faced a similar issue:

# Start Postgres service
graphene-direct bin/pg_ctl -D data -l logfile start

[P71712::] debug: Host: Linux
[P71712::] debug: LibOS xsave_enabled 1, xsave_size 0x440(1088), xsave_features 0x1f
[P71712::] debug: Initial VMA region 0x7fe147624000-0x7fe14769b000 (LibOS) bookkeeped
[P71712::] debug: Initial VMA region 0x7ffd04aae000-0x7ffd04aaf000 (vdso) bookkeeped
[P71712::] debug: Initial VMA region 0x7ffd04aaa000-0x7ffd04aae000 (vvar) bookkeeped
[P71712::] debug: Initial VMA region 0x7fe14769b000-0x7fe14769c000 (pal_internal_me) bookkeeped
[P71712::] debug: ASLR top address adjusted to 0x54629ca88000
[P71712::] debug: host is not Linux-SGX, skipping /dev/attestation setup
[P71712::] debug: Shim loaded at 0x7fe147624000, ready to initialize
[P71712::] debug: Mounting root as chroot filesystem: from file:. to /
[P71712::] debug: Mounting special proc filesystem: /proc
[P71712::] debug: Mounting special dev filesystem: /dev
[P71712::] debug: Mounting terminal device /dev/tty under /dev
[P71712::] debug: Mounting special sys filesystem: /sys
[P71712:T1:] debug: Mounting as chroot filesystem: from file:/usr/local/lib/x86_64-linux-gnu/graphene/runtime/glibc to /lib
[P71712:T1:] debug: Mounting as chroot filesystem: from file:/etc to /etc
[P71712:T1:] debug: Mounting as chroot filesystem: from file:/lib/x86_64-linux-gnu to /lib/x86_64-linux-gnu
[P71712:T1:] debug: Mounting as chroot filesystem: from file:/usr//lib/x86_64-linux-gnu to /usr//lib/x86_64-linux-gnu
[P71712:T1:] error: init_exec_handle: error opening executable: -2
[P71712:T1:] error: Error during shim_init() in init_important_handles (-2)

Do you have any ideas to fix this issue?

Thanks in advance.

[mkow]

I think your postgres is located at /usr/lib/postgresql/10/bin/postgres, but you run bin/pg_ctl, which will be /bin/pg_ctl unless you set a non-default CWD in the manifest. At least that's what the Graphene error you see says - it can't find the executable you want it to run.

[dzygann]

Czesc @mkow

What we did is to move the Postgres DB to the Examples folder for the graphene-direct test. It works like a charm by using the following command without Graphene:

~/workspaces/graphene/Examples/pgsql$ sudo -u postgres bin/psql test
[sudo] password for sgx: 
psql (15devel)
Type "help" for help.
test=# 

Is it possible to define a user who should execute the command? E.g something like this:

graphene-direct -u postgres bin/psql test

In the dockerfile is already such a command:

# Run the rest of the commands as the ``postgres`` user created by the ``postgres`` package when it was ``apt-get installed``
USER postgres

It seems that this isn't working, because the user postgres is not used. Is there a way to figure this out?

[mkow]

Cześć :)

No, we don't emulate users and permissions inside Graphene. But the logs you pasted above show that the problem is just in a wrong path to the postgres binary. init_exec_handle: cannot find executable in filesystem: -2 means that the executable was not found inside Graphene. Please read my previous comment carefully, it explains what's exactly wrong ;)

[dzygann]

Hi,

yes, I understood your point and you are right! I was confused and used the wrong path :D When I use the correct path I get the following error:

sgx@sgx-PC:~/workspaces/graphene/Examples/pgsql/bin$ graphene-direct ./pg_ctl -D ../data -l /var/log/postgresql/logfile.log start
error: Using insecure argv source. Graphene will continue application execution, but this configuration must not be used in production!
[P141908::] debug: Host: Linux
[P141908::] debug: LibOS xsave_enabled 1, xsave_size 0x440(1088), xsave_features 0x1f
[P141908::] debug: Initial VMA region 0x7fa6b9ab0000-0x7fa6b9b27000 (LibOS) bookkeeped
[P141908::] debug: Initial VMA region 0x7ffe76d32000-0x7ffe76d33000 (vdso) bookkeeped
[P141908::] debug: Initial VMA region 0x7ffe76d2e000-0x7ffe76d32000 (vvar) bookkeeped
[P141908::] debug: Initial VMA region 0x7fa6b9b27000-0x7fa6b9b28000 (pal_internal_me) bookkeeped
[P141908::] debug: ASLR top address adjusted to 0x65eb84049000
[P141908::] debug: host is not Linux-SGX, skipping /dev/attestation setup
[P141908::] debug: Shim loaded at 0x7fa6b9ab0000, ready to initialize
[P141908::] debug: Mounting root as chroot filesystem: from file:. to /
[P141908::] debug: Mounting special proc filesystem: /proc
[P141908::] debug: Mounting special dev filesystem: /dev
[P141908::] debug: Mounting terminal device /dev/tty under /dev
[P141908::] debug: Mounting special sys filesystem: /sys
[P141908:T1:] debug: Mounting as chroot filesystem: from file:/usr/local/lib/x86_64-linux-gnu/graphene/runtime/glibc to /lib
[P141908:T1:] debug: Mounting as chroot filesystem: from file:/etc to /etc
[P141908:T1:] debug: Mounting as chroot filesystem: from file:/lib/x86_64-linux-gnu to /lib/x86_64-linux-gnu
[P141908:T1:] debug: Mounting as chroot filesystem: from file:/usr//lib/x86_64-linux-gnu to /usr//lib/x86_64-linux-gnu
[P141908:T1:pg_ctl] debug: Allocating stack at 0x0 (size = 262144)
[P141908:T1:pg_ctl] debug: loading "file:./pg_ctl"
[P141908:T1:pg_ctl] debug: searching for interpreter: /lib/ld-linux-x86-64.so.2
[P141908:T1:pg_ctl] debug: Creating pipe: pipe.srv:141908
[P141908:T1:pg_ctl] debug: Shim process initialized
[P141908:shim] debug: IPC worker started
[P141908:T1:pg_ctl] warning: Not supported flag (0x3001) passed to arch_prctl
[P141908:T1:pg_ctl] debug: glibc register library /lib/libpthread.so.0 loaded at 0x65eb83f9d000
[P141908:T1:pg_ctl] debug: glibc register library /lib/libc.so.6 loaded at 0x65eb83ddd000
could not change directory to "": No such file or directory
pg_ctl: cannot be run as root
Please log in (using, e.g., "su") as the (unprivileged) user that will
own the server process.
[P141908:T1:pg_ctl] debug: ---- shim_exit_group (returning 1)
[P141908:T1:pg_ctl] debug: clearing POSIX locks for pid 1
[P141908:T1:pg_ctl] debug: sync client shutdown: closing handles
[P141908:T1:pg_ctl] debug: sync client shutdown: waiting for confirmation
[P141908:T1:pg_ctl] debug: sync client shutdown: finished
[P141908:shim] debug: IPC worker: exiting worker thread
[P141908:T1:pg_ctl] debug: process 141908 exited with status 1

Since there is no argument to emulate the user, is there no way to get around this issue?

pg_ctl: cannot be run as root
Please log in (using, e.g., "su") as the (unprivileged) user that will
own the server process.

[mkow]

Hmm, depends how postgres checks it. You can try changing the env variable ($USER), but if it's using the user-ID-related syscalls, then you'll need to patch shim_do_getuid() (and maybe others also) in Graphene to pretend you're another user.

[dimakuv]

Maybe it's good time to implement emulation of a current user in Graphene? The current hard-coding of 0 (= root user) in Graphene is wrong, misleading and confuses everyone who encounters this.

Why not:

• Get the current host user ID on startup
• Memorize and propagate to the LibOS
• Use this user ID in all syscalls, and skip/disallow syscalls that change it

Sounds much more logical than having the root user always and asking developers to circumvent this weirdness.

[mkow]

But there's no "current user", we're running a completely different OS instance than the host, so, passing host user id is conceptually wrong. What we can do is to hardcode some other ID than 0 or add a proper uid emulation.

[boryspoplawski]

The current hard-coding of 0 (= root user) in Graphene is wrong, misleading and confuses everyone who encounters this.

It's not wrong: we do not emulate any permission checks, i.e. allow for everything, which is pretty much all capabilities (default uid 0 "permissions").

I would rather say it's the problem of postgress. Why it disallows being run as root? uid 0 does not mean anything anymore (think of capabilities and namespaces).

[dimakuv]

But there's no "current user", we're running a completely different OS instance than the host, so, passing host user id is conceptually wrong. What we can do is to hardcode some other ID than 0 or add a proper uid emulation.

Well, yes, maybe just hard-coding some ID other than 0. We can also have a manifest option for this.

I am also curious if Postgress decides whether it runs as root by looking at /etc/passwd and finding the row corresponding to user ID (group ID?) 0. In this case, supplying a specially crafted /etc/passwd file would fix this issue.

[dimakuv]

But there's no "current user", we're running a completely different OS instance than the host, so, passing host user id is conceptually wrong.

Actually, why do you say that it's "conceptually wrong"? Graphene is not really a "completely different OS instance", since Graphene re-uses the FS and the network stacks (and all the resources limitations, and all scheduling, etc.) from the host OS.

So I would say that what conceptually wrong is that Graphene virtualizes the user ID (hey, you have all permissions) but doesn't virtualize host-file permissions (hey, you have all virtual permissions but you can't open files because of insufficient real permissions). There seems to be a clear disconnect between "what Graphene virtualizes" and "what Graphene can do".

[boryspoplawski]

Actually, why do you say that it's "conceptually wrong"? Graphene is not really a "completely different OS instance", since Graphene re-uses the FS and the network stacks (and all the resources limitations, and all scheduling, etc.) from the host OS.

This is only the case for a specific implementation of specific PAL. Well, it's the case for all PALs currently, but that should not affect the overall arch design.

So I would say that what conceptually wrong is that Graphene virtualizes the user ID (hey, you have all permissions) but doesn't virtualize host-file permissions (hey, you have all virtual permissions but you can't open files because of insufficient real permissions). There seems to be a clear disconnect between "what Graphene virtualizes" and "what Graphene can do".

We definitely should visualize UIDs. Keeping Graphene file permissions as host-level file permissions is imo design flaw, but that's an orthogonal issue.

Also please read my previous message.

[dimakuv]

This is only the case for a specific implementation of specific PAL. Well, it's the case for all PALs currently, but that should not affect the overall arch design.

Ah, this is a good point. Ok, so virtualizing the user ID (instead of propagating such info from the host OS) is the correct way.

So how should Graphene behave in a situation with an application refusing to run with uid == 0 (if this is indeed the kind of check in Postgres)?

And, continuing my example, how should Graphene behave on an application trying to e.g. access some file that can be opened only by root (assuming current Linux PAL), but the current host-user ID is non-root? In this case, the application gets confused: it thinks that it has root privileges and can open any file, but opening results in -EPERM.

[boryspoplawski]

So how should Graphene behave in a situation with an application refusing to run with uid == 0 (if this is indeed the kind of check in Postgres)?

Fix your application then :) Is there no option like --insecure_run_as_root_pretty_please_with_sugar_on_top_Im_aware_of_consequences in progress? I guess we could add a manifest option for initial, virtualized uid.

how should Graphene behave on an application trying to e.g. access some file that can be opened only by root (assuming current Linux PAL), but the current host-user ID is non-root?

If the host-fs file permissions are wrong (Graphene cannot access the file): a) file was created by Graphene - bug in Graphene, needs fixing, b) file was not created by Graphene - deployment issue / malicious host.

[dimakuv]

Agreed with your logic.

I guess we could add a manifest option for initial, virtualized uid.

Yes, sounds like everyone is fine with this approach.