Fail to bind AF_UNIX socket in python3.6 with gramine-sgx

[black197]

Description of the problem

I want to run gramine-sgx in ubuntu18.04 docker container so I built gramine from master and successfully run the helloworld with gramine-sgx. And I'm tring to bind unix socket in python3.6 with gramine-sgx but it failed. It doesn't work with gramine-direct too. If I run it without gramine, the python script can work fine.

Here's my output:

root@f8dd81b201a8:/gramine/CI-Examples/fdk# gramine-sgx /gramine/CI-Examples/fdk/fdk /python/bin/fdk /function/func.py handler
Gramine is starting. Parsing TOML manifest file, this may take some time...
-----------------------------------------------------------------------------------------------------------------------
Gramine detected the following insecure configurations:

  - sgx.debug = true                           (this is a debug enclave)
  - loader.insecure__use_cmdline_argv = true   (forwarding command-line args from untrusted host to the app)
  - sgx.allowed_files = [ ... ]                (some files are passed through from untrusted host without verification)

Gramine will continue application execution, but this configuration must not be used in production!
-----------------------------------------------------------------------------------------------------------------------

Traceback (most recent call last):
  File "/python/bin/fdk", line 13, in <module>
    sys.exit(main())
  File "/python/fdk/scripts/fdk.py", line 38, in main
    fdk.handle(handler)
  File "/python/fdk/__init__.py", line 126, in handle
    start(handle_code, lsnr, loop=loop)
  File "/python/fdk/__init__.py", line 63, in start
    sock.bind(phony_socket_path)
OSError: [Errno 22] Invalid argument

Here's part of my python code:

sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
sock.bind(phony_socket_path) #  phony_socket_path = "//tmp/phonyfunc.sock"

Makefile:

ARCH_LIBDIR ?= /lib/$(shell $(CC) -dumpmachine)

SGX_SIGNER_KEY ?= ../../Pal/src/host/Linux-SGX/signer/enclave-key.pem

ifeq ($(DEBUG),1)
GRAMINE_LOG_LEVEL = debug
else
GRAMINE_LOG_LEVEL = error
endif

.PHONY: all
all: fdk.manifest
ifeq ($(SGX),1)
all: fdk.manifest.sgx fdk.sig fdk.token
endif

fdk.manifest: fdk.manifest.template
    gramine-manifest \
        -Dlog_level=$(GRAMINE_LOG_LEVEL) \
        -Darch_libdir=$(ARCH_LIBDIR) \
        -Dentrypoint=$(realpath $(shell sh -c "command -v python3")) \
        $< >$@

fdk.manifest.sgx: fdk.manifest
    @test -s $(SGX_SIGNER_KEY) || \
        { echo "SGX signer private key was not found, please specify SGX_SIGNER_KEY!"; exit 1; }
    gramine-sgx-sign \
        --key $(SGX_SIGNER_KEY) \
        --manifest $< \
        --output $@

fdk.sig: fdk.manifest.sgx

fdk.token: fdk.sig
    gramine-sgx-get-token --output $@ --sig $<

.PHONY: clean
clean:
    $(RM) *.manifest *.manifest.sgx *.token *.sig OUTPUT* *.PID TEST_STDOUT TEST_STDERR
    $(RM) -r scripts/__pycache__

.PHONY: distclean
distclean: clean

And fdk.manifest:

[loader]
preload = "file:/usr/local/lib/x86_64-linux-gnu/gramine/libsysdb.so"
entrypoint = "file:/usr/local/lib/x86_64-linux-gnu/gramine/libsysdb.so"
log_level = "error"
insecure__use_cmdline_argv = true

[libos]
entrypoint = "/usr/bin/python3.6"

[sys]
enable_sigterm_injection = true

[fs]
[[fs.mounts]]
type = "chroot"
uri = "file:/usr/local/lib/x86_64-linux-gnu/gramine/runtime/glibc"
path = "/lib"

[[fs.mounts]]
type = "chroot"
uri = "file:/lib/x86_64-linux-gnu"
path = "/lib/x86_64-linux-gnu"

[[fs.mounts]]
type = "chroot"
uri = "file:/usr"
path = "/usr"

[[fs.mounts]]
type = "chroot"
uri = "file:/usr/lib/python3.6"
path = "/usr/lib/python3.6"

[[fs.mounts]]
type = "chroot"
uri = "file:/usr/lib/python3/dist-packages"
path = "/usr/lib/python3/dist-packages"

[[fs.mounts]]
type = "chroot"
uri = "file:/tmp"
path = "/tmp"

[[fs.mounts]]
type = "chroot"
uri = "file:/etc"
path = "/etc"

[[fs.mounts]]
type = "chroot"
uri = "file:/python"
path = "/python"

[[fs.mounts]]
type = "chroot"
uri = "file:/function"
path = "/function"

[sgx]
debug = true
nonpie_binary = true
enclave_size = "512M"
thread_num = 32
allowed_files = [ "file:/etc/nsswitch.conf", "file:/etc/ethers", "file:/etc/hosts", "file:/etc/group", "file:/etc/passwd", "file:/etc/gai.conf", "file:/etc/host.conf", "file:/etc/resolv.conf", "file:/tmp",]
isvprodid = 0
isvsvn = 0
remote_attestation = false
require_avx = false
require_avx512 = false
require_mpx = false
require_pkru = false
require_amx = false
support_exinfo = false
enable_stats = false
[[sgx.trusted_files]]
uri = "file:/usr/local/lib/x86_64-linux-gnu/gramine/libsysdb.so"

[[sgx.trusted_files]]
uri = "file:/usr/bin/python3.6"

[[sgx.trusted_files]]
uri = "file:/usr/local/lib/x86_64-linux-gnu/gramine/runtime/glibc/"

[[sgx.trusted_files]]
uri = "file:/lib/x86_64-linux-gnu/"

[[sgx.trusted_files]]
uri = "file:/usr//lib/x86_64-linux-gnu/"

[[sgx.trusted_files]]
uri = "file:/usr/lib/python3.6/"

[[sgx.trusted_files]]
uri = "file:/usr/lib/python3/dist-packages/"

[[sgx.trusted_files]]
uri = "file:/function/"

[[sgx.trusted_files]]
uri = "file:/etc/mime.types"

[[sgx.trusted_files]]
uri = "file:/etc/default/apport"

[[sgx.trusted_files]]
uri = "file:/python/"

[loader.env]
LD_LIBRARY_PATH = "/usr/lib/python3.6/lib:/lib:/lib/x86_64-linux-gnu:/usr/lib:/usr//lib/x86_64-linux-gnu"

[sys.stack]
size = "2M"

[loader.env.FDK_DEBUG]
passthrough = true

[loader.env.FN_FORMAT]
passthrough = true

[loader.env.FN_LISTENER]
passthrough = true

Gramine commit hash

commit 35a6654c

[dimakuv]

@boryspoplawski is working on the Big Sockets Rewrite. Borys, does this sound familiar to you? I think we indeed had some problems with AF_UNIX sockets in Python. Is this a legit issue? Will it be fixed in your rewrite?

[boryspoplawski]

It should work even now, no idea why it does not.

Will it be fixed in your rewrite?

Hopefully.

[boryspoplawski]

@black197 could you verify if https://github.com/gramineproject/gramine/pull/579 fixes the problem? It does for me

[black197]

@black197 could you verify if #579 fixes the problem? It does for me

It works. Thank you very much.

[boryspoplawski]

@black197 That PR is still not merged, this issue will be closed automatically on merge.

[black197]

@black197 That PR is still not merged, this issue will be closed automatically on merge.

Got it. Besides, I noticed that in #579 there's no entry for UNIX socket in FS. So how am I able to connect to the socket outside gramine, like in another program? I appreciate any help.

[dimakuv]

So how am I able to connect to the socket outside gramine, like in another program?

You cannot do this with UNIX domain sockets (AF_UNIX sockets). This is a deliberate security decision in Gramine.

The reason behind this is that UNIX domain sockets are automatically encrypted by Gramine (similarly to classic pipes). So when two Gramine threads/processes communicate with each other via UNIX domain sockets, then share an encryption key and know how to transparently encrypt/decrypt packets sent on these sockets.

This cannot be done in any secure and transparent way when Gramine process would communicate with an outside program. The outside program doesn't know how to encrypt UNIX domain sockets, and doesn't have the encryption key (that is only shared between Gramine processes).

That's why the only sockets that allow communication of Gramine with other programs is AF_INET (classic TCP or UDP connections).

[black197]

@dimakuv Oh I see. Thank you very much. Then I have to try other ways.