Packaging a complete Node app with Gramine

XNinety9 commented 2 years ago

Hi.

I've been fiddling with Gramine for a few day, and I have a question.

I want to run a complete NodeJs app with gramine. To be precise: Node-Red. I use the nodejs from the example repo as a starter.

My naive approach to pack NodeRed is to git-clone it inside the nodejs example's folder, use npm install and npm run build to prepare NodeRed and then add the whole NodeRed folder to sgx.trusted_files. From here it sounds a bit...brutal... Can someone point me in the right direction?

Thanks a lot.

dimakuv commented 2 years ago

So your problem is adding all the relevant files to sgx.trusted_files? There are three simple solutions I can see:

For testing and development purposes only, you can put sgx.file_check_policy = "allow_all_but_log". This is insecure, but easy for testing. See also https://gramine.readthedocs.io/en/latest/manifest-syntax.html#file-check-policy
You can add whole directories as trusted files: sgx.trusted_files = [ "file:dir1/", "file:dir2/", ...]. This is much easier and faster than adding each required file.
You can try GSC, if you're comfortable working with Docker images: https://github.com/gramineproject/gsc/. Documentation is here: https://gramine.readthedocs.io/projects/gsc/en/latest/

XNinety9 commented 2 years ago

I'm exploring the GSC path. I had to make a custom version of NodeRed's image (to be based on ubuntu), make use of non-root PR, I finally got the GSC process to complete.

Now to the docker run phase. When running the GSC-ed image with

docker run --device=/dev/isgx -p 1880:1880 -v /var/run/aesmd/aesm.socket:/var/run/aesmd/aesm.socket gsc-b4523c183c66

I get a huge error trace:

+ set -e
++ find /gramine/meson_build_output/lib -type d -path '*/site-packages'
+ export PYTHONPATH=:/gramine/meson_build_output/lib/python3.8/site-packages
+ PYTHONPATH=:/gramine/meson_build_output/lib/python3.8/site-packages
++ find /gramine/meson_build_output/lib -type d -path '*/pkgconfig'
+ export PKG_CONFIG_PATH=:/gramine/meson_build_output/lib/x86_64-linux-gnu/pkgconfig
+ PKG_CONFIG_PATH=:/gramine/meson_build_output/lib/x86_64-linux-gnu/pkgconfig
+ '[' -z '' ']'
+ gramine-sgx-get-token --quiet --sig /gramine/app_files/entrypoint.sig --output /gramine/app_files/entrypoint.token
+ gramine-sgx /gramine/app_files/entrypoint /usr/src/node-red/node_modules/node-red/red.js ''
Gramine is starting. Parsing TOML manifest file, this may take some time...
debug: Gramine parsed TOML manifest file successfully
debug: Token file: /gramine/app_files/entrypoint.token
debug: Read token:
debug:     valid:                 0x00000001
debug:     attr.flags:            0x0000000000000006
debug:     attr.xfrm:             0x0000000000000007
debug:     mr_enclave:            2e87e9c2431362923931446817d4b4aec55897b4452e4f392a830ed9a04d1d3d
debug:     mr_signer:             6d398fed9a176aba2f7a845416ff9d9b8d6d13a40d2c6f1c5b5ccddac95cd432
debug:     LE cpu_svn:            0712ffffff0200000000000000000000
debug:     LE isv_prod_id:        20
debug:     LE isv_svn:            05
debug:     LE masked_misc_select: 0x00000000
debug:     LE attr.flags:         0x0000000000000021
debug:     LE attr.xfrm:          0x0000000000000000
debug: Enclave created:
debug:     base:           0x0000000000000000
debug:     size:           0x0000000100000000
debug:     misc_select:    0x00000000
debug:     attr.flags:     0x0000000000000007
debug:     attr.xfrm:      0x0000000000000007
debug:     ssa_frame_size: 4
debug:     isv_prod_id:    0x00000000
debug:     isv_svn:        0x00000000
debug: Adding pages to SGX enclave, this may take some time...
debug: Adding pages to enclave: 0xfee54000-0x100000000 [REG:R--] (manifest) measured
debug: Adding pages to enclave: 0xfee14000-0xfee54000 [REG:RW-] (ssa) measured
debug: Adding pages to enclave: 0xfee0c000-0xfee14000 [TCS:---] (tcs) measured
debug: Adding pages to enclave: 0xfee04000-0xfee0c000 [REG:RW-] (tls) measured
debug: Adding pages to enclave: 0xfedc4000-0xfee04000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0xfed84000-0xfedc4000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0xfed44000-0xfed84000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0xfed04000-0xfed44000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0xfecc4000-0xfed04000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0xfec84000-0xfecc4000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0xfec44000-0xfec84000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0xfec04000-0xfec44000 [REG:RW-] (stack) measured
debug: Adding pages to enclave: 0xfebf4000-0xfec04000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0xfebe4000-0xfebf4000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0xfebd4000-0xfebe4000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0xfebc4000-0xfebd4000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0xfebb4000-0xfebc4000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0xfeba4000-0xfebb4000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0xfeb94000-0xfeba4000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0xfeb84000-0xfeb94000 [REG:RW-] (sig_stack) measured
debug: Adding pages to enclave: 0xfe75a000-0xfe7ac000 [REG:R-X] (code) measured
debug: Adding pages to enclave: 0xfe7ac000-0xfe7b0000 [REG:RW-] (data) measured
debug: Adding pages to enclave: 0xfe7b0000-0xfeb84000 [REG:RW-] (bss) measured
debug: Adding pages to enclave: 0x10000-0xfe75a000 [REG:RWX] (free)
debug: Added all pages to SGX enclave
debug: Enclave initializing:
debug:     enclave id:   0x00000000fffff000
debug:     mr_enclave:   2e87e9c2431362923931446817d4b4aec55897b4452e4f392a830ed9a04d1d3d
Detected a huge manifest, preallocating 128MB of internal memory.
-----------------------------------------------------------------------------------------------------------------------
Gramine detected the following insecure configurations:

  - sgx.debug = true                           (this is a debug enclave)
  - loader.log_level = warning|debug|trace|all (verbose log level, may leak information)
  - loader.insecure__use_cmdline_argv = true   (forwarding command-line args from untrusted host to the app)

Gramine will continue application execution, but this configuration must not be used in production!
-----------------------------------------------------------------------------------------------------------------------

warning: DkVirtualMemoryProtect is unimplemented in Linux-SGX PAL
[::] debug: Gramine was built from commit: 63d8dcaa66f8ee0a4462fca58055a4057821eeb1
[::] debug: Host: Linux-SGX
[::] debug: LibOS xsave_enabled 1, xsave_size 0x340(832), xsave_features 0x7
[::] debug: Initial VMA region 0xfcce5000-0xfcdbc000 (LibOS) bookkeeped
[::] debug: Initial VMA region 0xfee54000-0x100000000 (manifest) bookkeeped
[::] debug: ASLR top address adjusted to 0x6eef7000
[::] debug: host is Linux-SGX, adding SGX-specific /dev/attestation files: report, quote, etc.
[::] debug: Shim loaded at 0xfcce5000, ready to initialize
[::] debug: mounting "file:/" (chroot) under /
[::] debug: mounting "proc" (pseudo) under /proc
[::] debug: mounting "dev" (pseudo) under /dev
[::] debug: mounting "dev:tty" (chroot) under /dev/tty
[::] debug: mounting "sys" (pseudo) under /sys
[::] debug: Creating pipe: pipe.srv:19502e9a8e93fc935cdfbc45545a7ade9abe5d012e8468a7d801362827fbeda9
[P1:T1:node] debug: Creating pipe: pipe.srv:532278f0ce8664290dc771ac714dc6fb6f12f07ae83e671221f0da4134f84494
[P1:T1:node] debug: Allocating stack at 0 (size = 262144)
[P1:T1:node] debug: loading "file://gramine/app_files/node"
[P1:T1:node] debug: append_r_debug: adding file://gramine/app_files/node at 0
[P1:T1:node] debug: find_interp: searching for interpreter: /gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/ld-linux-x86-64.so.2
[P1:T1:node] debug: loading "file://gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/ld-linux-x86-64.so.2"
[P1:T1:node] debug: append_r_debug: adding file://gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/ld-linux-x86-64.so.2 at 0x6ee7d000
[P1:T1:node] debug: Creating pipe: pipe.srv:1
[P1:T1:node] debug: Creating pipe: pipe.srv:1a1f12ae7873f410527c44e64dd3c4746b58359aa99122f64aacc0b814b79412
[P1:T1:node] debug: LibOS initialized
[P1:shim] debug: IPC worker started
[P1:T1:node] debug: append_r_debug: adding file:[vdso_libos] at 0x6ee7c000
[P1:T1:node] trace: ---- shim_brk(0) = 0x5f93000
[P1:T1:node] warning: Not supported flag (0x3001) passed to arch_prctl
[P1:T1:node] trace: ---- shim_arch_prctl(12289, 0x6eef6b60) = -38
[P1:T1:node] trace: ---- shim_uname(0x6eef6750) = 0x0
[P1:T1:node] trace: ---- shim_mmap(0, 0x2000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0x0) ...
[P1:T1:node] trace: ---- return from shim_mmap(...) = 0x6ee7a000
[P1:T1:node] trace: ---- shim_access("/etc/ld.so.preload", F_OK|X_OK) = -2
[P1:T1:node] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/glibc-hwcaps/x86-64-v3/libdl.so.2", O_RDONLY|0x80000, 0000) = -2
[P1:T1:node] trace: ---- shim_newfstatat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/glibc-hwcaps/x86-64-v3", 0x6eef5d80, 0) = -2
[P1:T1:node] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/glibc-hwcaps/x86-64-v2/libdl.so.2", O_RDONLY|0x80000, 0000) = -2
[P1:T1:node] trace: ---- shim_newfstatat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/glibc-hwcaps/x86-64-v2", 0x6eef5d80, 0) = -2
[P1:T1:node] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/tls/haswell/x86_64/libdl.so.2", O_RDONLY|0x80000, 0000) = -2
[P1:T1:node] trace: ---- shim_newfstatat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/tls/haswell/x86_64", 0x6eef5d80, 0) = -2
[P1:T1:node] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/tls/haswell/libdl.so.2", O_RDONLY|0x80000, 0000) = -2
[P1:T1:node] trace: ---- shim_newfstatat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/tls/haswell", 0x6eef5d80, 0) = -2
[P1:T1:node] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/tls/x86_64/libdl.so.2", O_RDONLY|0x80000, 0000) = -2
[P1:T1:node] trace: ---- shim_newfstatat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/tls/x86_64", 0x6eef5d80, 0) = -2
[P1:T1:node] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/tls/libdl.so.2", O_RDONLY|0x80000, 0000) = -2
[P1:T1:node] trace: ---- shim_newfstatat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/tls", 0x6eef5d80, 0) = -2
[P1:T1:node] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/haswell/x86_64/libdl.so.2", O_RDONLY|0x80000, 0000) = -2
[P1:T1:node] trace: ---- shim_newfstatat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/haswell/x86_64", 0x6eef5d80, 0) = -2
[P1:T1:node] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/haswell/libdl.so.2", O_RDONLY|0x80000, 0000) = -2
[P1:T1:node] trace: ---- shim_newfstatat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/haswell", 0x6eef5d80, 0) = -2
[P1:T1:node] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/x86_64/libdl.so.2", O_RDONLY|0x80000, 0000) = -2
[P1:T1:node] trace: ---- shim_newfstatat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/x86_64", 0x6eef5d80, 0) = -2
[P1:T1:node] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libdl.so.2", O_RDONLY|0x80000, 0000) = 0x3
[P1:T1:node] trace: ---- shim_read(3, 0x6eef5ee8, 0x340) ...
[P1:T1:node] trace: ---- return from shim_read(...) = 0x340
[P1:T1:node] trace: ---- shim_newfstatat(3, "", 0x6eef5d80, 4096) = 0x0
[P1:T1:node] trace: ---- shim_mmap(0, 0x4028, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0x0) ...
[P1:T1:node] trace: ---- return from shim_mmap(...) = 0x6ee75000
[P1:T1:node] trace: ---- shim_mmap(0x6ee76000, 0x1000, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) ...
[P1:T1:node] trace: ---- return from shim_mmap(...) = 0x6ee76000
[P1:T1:node] trace: ---- shim_mmap(0x6ee77000, 0x1000, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) ...
[P1:T1:node] trace: ---- return from shim_mmap(...) = 0x6ee77000
[P1:T1:node] trace: ---- shim_mmap(0x6ee78000, 0x2000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) ...
[P1:T1:node] trace: ---- return from shim_mmap(...) = 0x6ee78000
[P1:T1:node] trace: ---- shim_close(3) = 0x0
[P1:T1:node] debug: glibc register library /gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libdl.so.2 loaded at 0x6ee75000
[P1:T1:node] debug: append_r_debug: adding file://gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libdl.so.2 at 0x6ee75000
[P1:T1:node] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libstdc++.so.6", O_RDONLY|0x80000, 0000) = -2
[P1:T1:node] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v3/libstdc++.so.6", O_RDONLY|0x80000, 0000) = -2
[P1:T1:node] trace: ---- shim_newfstatat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v3", 0x6eef5d60, 0) = -2
[P1:T1:node] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v2/libstdc++.so.6", O_RDONLY|0x80000, 0000) = -2
[P1:T1:node] trace: ---- shim_newfstatat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v2", 0x6eef5d60, 0) = -2
[P1:T1:node] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/tls/haswell/x86_64/libstdc++.so.6", O_RDONLY|0x80000, 0000) = -2
[P1:T1:node] trace: ---- shim_newfstatat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/tls/haswell/x86_64", 0x6eef5d60, 0) = -2
[P1:T1:node] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/tls/haswell/libstdc++.so.6", O_RDONLY|0x80000, 0000) = -2
[P1:T1:node] trace: ---- shim_newfstatat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/tls/haswell", 0x6eef5d60, 0) = -2
[P1:T1:node] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/tls/x86_64/libstdc++.so.6", O_RDONLY|0x80000, 0000) = -2
[P1:T1:node] trace: ---- shim_newfstatat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/tls/x86_64", 0x6eef5d60, 0) = -2
[P1:T1:node] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/tls/libstdc++.so.6", O_RDONLY|0x80000, 0000) = -2
[P1:T1:node] trace: ---- shim_newfstatat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/tls", 0x6eef5d60, 0) = -2
[P1:T1:node] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/haswell/x86_64/libstdc++.so.6", O_RDONLY|0x80000, 0000) = -2
[P1:T1:node] trace: ---- shim_newfstatat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/haswell/x86_64", 0x6eef5d60, 0) = -2
[P1:T1:node] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/haswell/libstdc++.so.6", O_RDONLY|0x80000, 0000) = -2
[P1:T1:node] trace: ---- shim_newfstatat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/haswell", 0x6eef5d60, 0) = -2
[P1:T1:node] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/x86_64/libstdc++.so.6", O_RDONLY|0x80000, 0000) = -2
[P1:T1:node] trace: ---- shim_newfstatat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/x86_64", 0x6eef5d60, 0) = -2
[P1:T1:node] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", O_RDONLY|0x80000, 0000) = 0x3
[P1:T1:node] trace: ---- shim_read(3, 0x6eef5ec8, 0x340) ...
[P1:T1:node] trace: ---- return from shim_read(...) = 0x340
[P1:T1:node] trace: ---- shim_newfstatat(3, "", 0x6eef5d60, 4096) = 0x0
[P1:T1:node] trace: ---- shim_mmap(0, 0x1e1800, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0x0) ...
[P1:T1:node] trace: ---- return from shim_mmap(...) = 0x6ec93000
[P1:T1:node] trace: ---- shim_mprotect(0x6ed29000, 0x13b000, PROT_NONE) ...
[P1:T1:node] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:node] trace: ---- shim_mmap(0x6ed29000, 0xf1000, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x96000) ...
[P1:T1:node] trace: ---- return from shim_mmap(...) = 0x6ed29000
[P1:T1:node] trace: ---- shim_mmap(0x6ee1a000, 0x49000, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x187000) ...
[P1:T1:node] trace: ---- return from shim_mmap(...) = 0x6ee1a000
[P1:T1:node] trace: ---- shim_mmap(0x6ee64000, 0xe000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1d0000) ...
[P1:T1:node] trace: ---- return from shim_mmap(...) = 0x6ee64000
[P1:T1:node] trace: ---- shim_mmap(0x6ee72000, 0x2800, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_FIXED, -1, 0x0) ...
[P1:T1:node] trace: ---- return from shim_mmap(...) = 0x6ee72000
[P1:T1:node] trace: ---- shim_close(3) = 0x0
[P1:T1:node] debug: glibc register library /usr/lib/x86_64-linux-gnu/libstdc++.so.6 loaded at 0x6ec93000
[P1:T1:node] debug: append_r_debug: adding file://usr/lib/x86_64-linux-gnu/libstdc++.so.6 at 0x6ec93000
[P1:T1:node] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libm.so.6", O_RDONLY|0x80000, 0000) = 0x3
[P1:T1:node] trace: ---- shim_read(3, 0x6eef5ea8, 0x340) ...
[P1:T1:node] trace: ---- return from shim_read(...) = 0x340
[P1:T1:node] trace: ---- shim_newfstatat(3, "", 0x6eef5d40, 4096) = 0x0
[P1:T1:node] trace: ---- shim_mmap(0, 0xdd100, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0x0) ...
[P1:T1:node] trace: ---- return from shim_mmap(...) = 0x6ebb5000
[P1:T1:node] trace: ---- shim_mmap(0x6ebc5000, 0x71000, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x10000) ...
[P1:T1:node] trace: ---- return from shim_mmap(...) = 0x6ebc5000
[P1:T1:node] trace: ---- shim_mmap(0x6ec36000, 0x5b000, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x81000) ...
[P1:T1:node] trace: ---- return from shim_mmap(...) = 0x6ec36000
[P1:T1:node] trace: ---- shim_mmap(0x6ec91000, 0x2000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xdb000) ...
[P1:T1:node] trace: ---- return from shim_mmap(...) = 0x6ec91000
[P1:T1:node] trace: ---- shim_close(3) = 0x0
[P1:T1:node] debug: glibc register library /gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libm.so.6 loaded at 0x6ebb5000
[P1:T1:node] debug: append_r_debug: adding file://gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libm.so.6 at 0x6ebb5000
[P1:T1:node] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libgcc_s.so.1", O_RDONLY|0x80000, 0000) = -2
[P1:T1:node] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libgcc_s.so.1", O_RDONLY|0x80000, 0000) = 0x3
[P1:T1:node] trace: ---- shim_read(3, 0x6eef5e88, 0x340) ...
[P1:T1:node] trace: ---- return from shim_read(...) = 0x340
[P1:T1:node] trace: ---- shim_newfstatat(3, "", 0x6eef5d20, 4096) = 0x0
[P1:T1:node] trace: ---- shim_mmap(0, 0x1a448, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0x0) ...
[P1:T1:node] trace: ---- return from shim_mmap(...) = 0x6eb9a000
[P1:T1:node] trace: ---- shim_mmap(0x6eb9d000, 0x12000, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) ...
[P1:T1:node] trace: ---- return from shim_mmap(...) = 0x6eb9d000
[P1:T1:node] trace: ---- shim_mmap(0x6ebaf000, 0x4000, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x15000) ...
[P1:T1:node] trace: ---- return from shim_mmap(...) = 0x6ebaf000
[P1:T1:node] trace: ---- shim_mmap(0x6ebb3000, 0x2000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x18000) ...
[P1:T1:node] trace: ---- return from shim_mmap(...) = 0x6ebb3000
[P1:T1:node] trace: ---- shim_close(3) = 0x0
[P1:T1:node] debug: glibc register library /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 loaded at 0x6eb9a000
[P1:T1:node] debug: append_r_debug: adding file://usr/lib/x86_64-linux-gnu/libgcc_s.so.1 at 0x6eb9a000
[P1:T1:node] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libpthread.so.0", O_RDONLY|0x80000, 0000) = 0x3
[P1:T1:node] trace: ---- shim_read(3, 0x6eef5e68, 0x340) ...
[P1:T1:node] trace: ---- return from shim_read(...) = 0x340
[P1:T1:node] trace: ---- shim_newfstatat(3, "", 0x6eef5d00, 4096) = 0x0
[P1:T1:node] trace: ---- shim_mmap(0, 0x4028, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0x0) ...
[P1:T1:node] trace: ---- return from shim_mmap(...) = 0x6eb95000
[P1:T1:node] trace: ---- shim_mmap(0x6eb96000, 0x1000, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) ...
[P1:T1:node] trace: ---- return from shim_mmap(...) = 0x6eb96000
[P1:T1:node] trace: ---- shim_mmap(0x6eb97000, 0x1000, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) ...
[P1:T1:node] trace: ---- return from shim_mmap(...) = 0x6eb97000
[P1:T1:node] trace: ---- shim_mmap(0x6eb98000, 0x2000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) ...
[P1:T1:node] trace: ---- return from shim_mmap(...) = 0x6eb98000
[P1:T1:node] trace: ---- shim_close(3) = 0x0
[P1:T1:node] debug: glibc register library /gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libpthread.so.0 loaded at 0x6eb95000
[P1:T1:node] debug: append_r_debug: adding file://gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libpthread.so.0 at 0x6eb95000
[P1:T1:node] trace: ---- shim_openat(AT_FDCWD, "/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libc.so.6", O_RDONLY|0x80000, 0000) = 0x3
[P1:T1:node] trace: ---- shim_read(3, 0x6eef5e48, 0x340) ...
[P1:T1:node] trace: ---- return from shim_read(...) = 0x340
[P1:T1:node] trace: ---- shim_pread64(3, 0x6eef5870, 0x310, 0x40) ...
[P1:T1:node] trace: ---- return from shim_pread64(...) = 0x310
[P1:T1:node] trace: ---- shim_pread64(3, 0x6eef5830, 0x30, 0x350) ...
[P1:T1:node] trace: ---- return from shim_pread64(...) = 0x30
[P1:T1:node] trace: ---- shim_pread64(3, 0x6eef57e0, 0x44, 0x380) ...
[P1:T1:node] trace: ---- return from shim_pread64(...) = 0x44
[P1:T1:node] trace: ---- shim_newfstatat(3, "", 0x6eef5ce0, 4096) = 0x0
[P1:T1:node] trace: ---- shim_mmap(0, 0x2000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0x0) ...
[P1:T1:node] trace: ---- return from shim_mmap(...) = 0x6eb93000
[P1:T1:node] trace: ---- shim_pread64(3, 0x6eef5930, 0x310, 0x40) ...
[P1:T1:node] trace: ---- return from shim_pread64(...) = 0x310
[P1:T1:node] trace: ---- shim_mmap(0, 0x1f9a50, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0x0) ...
[P1:T1:node] trace: ---- return from shim_mmap(...) = 0x6e999000
[P1:T1:node] trace: ---- shim_mprotect(0x6e9c5000, 0x1c0000, PROT_NONE) ...
[P1:T1:node] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:node] trace: ---- shim_mmap(0x6e9c5000, 0x16a000, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2c000) ...
[P1:T1:node] trace: ---- return from shim_mmap(...) = 0x6e9c5000
[P1:T1:node] trace: ---- shim_mmap(0x6eb2f000, 0x55000, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x196000) ...
[P1:T1:node] trace: ---- return from shim_mmap(...) = 0x6eb2f000
[P1:T1:node] trace: ---- shim_mmap(0x6eb85000, 0x6000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1eb000) ...
[P1:T1:node] trace: ---- return from shim_mmap(...) = 0x6eb85000
[P1:T1:node] trace: ---- shim_mmap(0x6eb8b000, 0x7a50, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_FIXED, -1, 0x0) ...
[P1:T1:node] trace: ---- return from shim_mmap(...) = 0x6eb8b000
[P1:T1:node] trace: ---- shim_close(3) = 0x0
[P1:T1:node] debug: glibc register library /gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libc.so.6 loaded at 0x6e999000
[P1:T1:node] debug: append_r_debug: adding file://gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libc.so.6 at 0x6e999000
[P1:T1:node] trace: ---- shim_mmap(0, 0x3000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0x0) ...
[P1:T1:node] trace: ---- return from shim_mmap(...) = 0x6e996000
[P1:T1:node] trace: ---- shim_arch_prctl(4098, 0x6e996780) = 0x0
[P1:T1:node] trace: ---- shim_set_tid_address(0x6e996a50) = 0x1
[P1:T1:node] trace: ---- shim_set_robust_list(0x6e996a60, 0x18) = 0x0
[P1:T1:node] warning: Unsupported system call rseq
[P1:T1:node] trace: ---- shim_mprotect(0x6eb85000, 0x3000, PROT_READ) ...
[P1:T1:node] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:node] trace: ---- shim_mprotect(0x6eb98000, 0x1000, PROT_READ) ...
[P1:T1:node] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:node] trace: ---- shim_mprotect(0x6ebb3000, 0x1000, PROT_READ) ...
[P1:T1:node] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:node] trace: ---- shim_mprotect(0x6ec91000, 0x1000, PROT_READ) ...
[P1:T1:node] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:node] trace: ---- shim_mprotect(0x6ee64000, 0xb000, PROT_READ) ...
[P1:T1:node] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:node] trace: ---- shim_mprotect(0x6ee78000, 0x1000, PROT_READ) ...
[P1:T1:node] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:node] trace: ---- shim_mprotect(0x459f000, 0x3000, PROT_READ) ...
[P1:T1:node] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:node] trace: ---- shim_mprotect(0x6eeb2000, 0x2000, PROT_READ) ...
[P1:T1:node] trace: ---- return from shim_mprotect(...) = 0x0
[P1:T1:node] trace: ---- shim_prlimit64(0, 3, 0, 0x6eef68e0) = 0x0
[P1:T1:node] trace: ---- shim_getrandom(0x6eb90198, 0x8, GRND_NONBLOCK) = 0x8
[P1:T1:node] trace: ---- shim_brk(0) = 0x5f93000
[P1:T1:node] trace: ---- shim_brk(0x5fb4000) = 0x5fb4000
[P1:T1:node] trace: ---- shim_futex(0x6ee726bc, FUTEX_PRIVATE|FUTEX_WAKE, 2147483647, 0, 0x1, 202) ...
[P1:T1:node] trace: ---- return from shim_futex(...) = 0x0
[P1:T1:node] trace: ---- shim_futex(0x6ee726c8, FUTEX_PRIVATE|FUTEX_WAKE, 2147483647, 0, 0x1, 202) ...
[P1:T1:node] trace: ---- return from shim_futex(...) = 0x0
[P1:T1:node] trace: ---- shim_brk(0x5fd5000) = 0x5fb4000
[P1:T1:node] trace: ---- shim_mmap(0, 0x100000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0x0) ...
[P1:T1:node] trace: ---- return from shim_mmap(...) = 0x6e896000
[P1:T1:node] trace: ---- shim_clock_gettime(1, 0x6eef6c20) = 0x0
[P1:T1:node] trace: ---- shim_gettimeofday(0x6eef6c00, 0) = 0x0
[P1:T1:node] trace: ---- shim_futex(0x45bcc38, FUTEX_PRIVATE|FUTEX_WAKE, 2147483647, 0, 0x1999999999999999, 0) ...
[P1:T1:node] trace: ---- return from shim_futex(...) = 0x0
[P1:T1:node] trace: ---- shim_getuid() = 0x0
[P1:T1:node] trace: ---- shim_geteuid() = 0x0
[P1:T1:node] trace: ---- shim_getgid() = 0x0
[P1:T1:node] trace: ---- shim_getegid() = 0x0
[P1:T1:node] trace: ---- shim_rt_sigprocmask(SETMASK, [SIGUSR1,], NULL, 0x8) = 0x0
[P1:T1:node] trace: ---- shim_fstat(0, 0x45b4728) = 0x0
[P1:T1:node] trace: ---- shim_fstat(1, 0x45b4800) = 0x0
[P1:T1:node] trace: ---- shim_fstat(2, 0x45b48d8) = 0x0
[P1:T1:node] trace: ---- shim_rt_sigaction([SIGHUP], 0x6eef6570, 0, 0x8) = 0x0
[P1:T1:node] trace: ---- shim_rt_sigaction([SIGINT], 0x6eef6570, 0, 0x8) = 0x0
[P1:T1:node] trace: ---- shim_rt_sigaction([SIGQUIT], 0x6eef6570, 0, 0x8) = 0x0
[P1:T1:node] trace: ---- shim_rt_sigaction([SIGILL], 0x6eef6570, 0, 0x8) = 0x0
[P1:T1:node] trace: ---- shim_rt_sigaction([SIGTRAP], 0x6eef6570, 0, 0x8) = 0x0
[P1:T1:node] trace: ---- shim_rt_sigaction([SIGABRT], 0x6eef6570, 0, 0x8) = 0x0
[P1:T1:node] trace: ---- shim_rt_sigaction([SIGBUS], 0x6eef6570, 0, 0x8) = 0x0
[P1:T1:node] trace: ---- shim_rt_sigaction([SIGFPE], 0x6eef6570, 0, 0x8) = 0x0
[P1:T1:node] trace: ---- shim_rt_sigaction([SIGUSR1], 0x6eef6570, 0, 0x8) = 0x0
[P1:T1:node] trace: ---- shim_rt_sigaction([SIGSEGV], 0x6eef6570, 0, 0x8) = 0x0
[P1:T1:node] trace: ---- shim_rt_sigaction([SIGUSR2], 0x6eef6570, 0, 0x8) = 0x0
[P1:T1:node] trace: ---- shim_rt_sigaction([SIGPIPE], 0x6eef6570, 0, 0x8) = 0x0
[P1:T1:node] trace: ---- shim_rt_sigaction([SIGALRM], 0x6eef6570, 0, 0x8) = 0x0
[P1:T1:node] trace: ---- shim_rt_sigaction([SIGTERM], 0x6eef6570, 0, 0x8) = 0x0
[P1:T1:node] trace: ---- shim_rt_sigaction([SIGSTKFLT], 0x6eef6570, 0, 0x8) = 0x0
[P1:T1:node] trace: ---- shim_rt_sigaction([SIGCHLD], 0x6eef6570, 0, 0x8) = 0x0
[P1:T1:node] trace: ---- shim_rt_sigaction([SIGCONT], 0x6eef6570, 0, 0x8) = 0x0
[P1:T1:node] trace: ---- shim_rt_sigaction([SIGTSTP], 0x6eef6570, 0, 0x8) = 0x0
[P1:T1:node] trace: ---- shim_rt_sigaction([SIGTTIN], 0x6eef6570, 0, 0x8) = 0x0
[P1:T1:node] trace: ---- shim_rt_sigaction([SIGTTOU], 0x6eef6570, 0, 0x8) = 0x0
[P1:T1:node] trace: ---- shim_rt_sigaction([SIGURG], 0x6eef6570, 0, 0x8) = 0x0
[P1:T1:node] trace: ---- shim_rt_sigaction([SIGXCPU], 0x6eef6570, 0, 0x8) = 0x0
[P1:T1:node] trace: ---- shim_rt_sigaction([SIGXFSZ], 0x6eef6570, 0, 0x8) = 0x0
[P1:T1:node] trace: ---- shim_rt_sigaction([SIGVTALRM], 0x6eef6570, 0, 0x8) = 0x0
[P1:T1:node] trace: ---- shim_rt_sigaction([SIGPROF], 0x6eef6570, 0, 0x8) = 0x0
[P1:T1:node] trace: ---- shim_rt_sigaction([SIGWINCH], 0x6eef6570, 0, 0x8) = 0x0
[P1:T1:node] trace: ---- shim_rt_sigaction([SIGIO], 0x6eef6570, 0, 0x8) = 0x0
[P1:T1:node] trace: ---- shim_rt_sigaction([SIGPWR], 0x6eef6570, 0, 0x8) = 0x0
[P1:T1:node] trace: ---- shim_rt_sigaction([SIGSYS], 0x6eef6570, 0, 0x8) = 0x0
[P1:T1:node] trace: ---- shim_fcntl(0, F_GETFL, 0) = 0x0
[P1:T1:node] trace: ---- shim_ioctl(0, TCGETS, 0x6eef6560) ...
[P1:T1:node] trace: ---- return from shim_ioctl(...) = -38
[P1:T1:node] trace: ---- shim_fstat(0, 0x6eef6610) = 0x0
[P1:T1:node] trace: ---- shim_fcntl(1, F_GETFL, 0x6eef6610) = 0x401
[P1:T1:node] trace: ---- shim_ioctl(1, TCGETS, 0x6eef6560) ...
[P1:T1:node] trace: ---- return from shim_ioctl(...) = -38
[P1:T1:node] trace: ---- shim_fstat(1, 0x6eef6610) = 0x0
[P1:T1:node] trace: ---- shim_fcntl(2, F_GETFL, 0x6eef6610) = 0x401
[P1:T1:node] trace: ---- shim_ioctl(2, TCGETS, 0x6eef6560) ...
[P1:T1:node] trace: ---- return from shim_ioctl(...) = -38
[P1:T1:node] trace: ---- shim_fstat(2, 0x6eef6610) = 0x0
[P1:T1:node] trace: ---- shim_rt_sigaction([SIGINT], 0x6eef6570, 0, 0x8) = 0x0
[P1:T1:node] trace: ---- shim_rt_sigaction([SIGTERM], 0x6eef6570, 0, 0x8) = 0x0
[P1:T1:node] trace: ---- shim_rt_sigaction([SIGSEGV], 0x6eef6570, 0, 0x8) = 0x0
[P1:T1:node] trace: ---- shim_prlimit64(0, 7, 0, 0x6eef67f0) = 0x0
[P1:T1:node] trace: ---- shim_prlimit64(0, 7, 0x6eef67f0, 0) = 0x0
[P1:T1:node] trace: ---- shim_clock_gettime(1, 0x6eef68b0) = 0x0
[P1:T1:node] trace: ---- shim_ioctl(0, FIOCLEX, 0x16fa9abad06ddcb0) ...
[P1:T1:node] trace: ---- return from shim_ioctl(...) = 0x0
[P1:T1:node] trace: ---- shim_ioctl(1, FIOCLEX, 0x16fa9abad06ddcb0) ...
[P1:T1:node] trace: ---- return from shim_ioctl(...) = 0x0
[P1:T1:node] trace: ---- shim_ioctl(2, FIOCLEX, 0x16fa9abad06ddcb0) ...
[P1:T1:node] trace: ---- return from shim_ioctl(...) = 0x0
[P1:T1:node] trace: ---- shim_ioctl(3, FIOCLEX, 0x16fa9abad06ddcb0) ...
[P1:T1:node] trace: ---- return from shim_ioctl(...) = -9
[P1:T1:node] trace: ---- shim_ioctl(4, FIOCLEX, 0xffffffffffffff48) ...
[P1:T1:node] trace: ---- return from shim_ioctl(...) = -9
[P1:T1:node] trace: ---- shim_ioctl(5, FIOCLEX, 0xffffffffffffff48) ...
[P1:T1:node] trace: ---- return from shim_ioctl(...) = -9
[P1:T1:node] trace: ---- shim_ioctl(6, FIOCLEX, 0xffffffffffffff48) ...
[P1:T1:node] trace: ---- return from shim_ioctl(...) = -9
[P1:T1:node] trace: ---- shim_ioctl(7, FIOCLEX, 0xffffffffffffff48) ...
[P1:T1:node] trace: ---- return from shim_ioctl(...) = -9
[P1:T1:node] trace: ---- shim_ioctl(8, FIOCLEX, 0xffffffffffffff48) ...
[P1:T1:node] trace: ---- return from shim_ioctl(...) = -9
[P1:T1:node] trace: ---- shim_ioctl(9, FIOCLEX, 0xffffffffffffff48) ...
[P1:T1:node] trace: ---- return from shim_ioctl(...) = -9
[P1:T1:node] trace: ---- shim_ioctl(10, FIOCLEX, 0xffffffffffffff48) ...
[P1:T1:node] trace: ---- return from shim_ioctl(...) = -9
[P1:T1:node] trace: ---- shim_ioctl(11, FIOCLEX, 0xffffffffffffff48) ...
[P1:T1:node] trace: ---- return from shim_ioctl(...) = -9
[P1:T1:node] trace: ---- shim_ioctl(12, FIOCLEX, 0xffffffffffffff48) ...
[P1:T1:node] trace: ---- return from shim_ioctl(...) = -9
[P1:T1:node] trace: ---- shim_ioctl(13, FIOCLEX, 0xffffffffffffff48) ...
[P1:T1:node] trace: ---- return from shim_ioctl(...) = -9
[P1:T1:node] trace: ---- shim_ioctl(14, FIOCLEX, 0xffffffffffffff48) ...
[P1:T1:node] trace: ---- return from shim_ioctl(...) = -9
[P1:T1:node] trace: ---- shim_ioctl(15, FIOCLEX, 0xffffffffffffff48) ...
[P1:T1:node] trace: ---- return from shim_ioctl(...) = -9
[P1:T1:node] trace: ---- shim_ioctl(16, FIOCLEX, 0xffffffffffffff48) ...
[P1:T1:node] trace: ---- return from shim_ioctl(...) = -9
[P1:T1:node] trace: ---- shim_getuid() = 0x0
[P1:T1:node] trace: ---- shim_geteuid() = 0x0
[P1:T1:node] trace: ---- shim_getgid() = 0x0
[P1:T1:node] trace: ---- shim_getegid() = 0x0
[P1:T1:node] trace: ---- shim_getuid() = 0x0
[P1:T1:node] trace: ---- shim_geteuid() = 0x0
[P1:T1:node] trace: ---- shim_getgid() = 0x0
[P1:T1:node] trace: ---- shim_getegid() = 0x0
[P1:T1:node] trace: ---- shim_getuid() = 0x0
[P1:T1:node] trace: ---- shim_geteuid() = 0x0
[P1:T1:node] trace: ---- shim_getgid() = 0x0
[P1:T1:node] trace: ---- shim_getegid() = 0x0
[P1:T1:node] trace: ---- shim_getuid() = 0x0
[P1:T1:node] trace: ---- shim_geteuid() = 0x0
[P1:T1:node] trace: ---- shim_getgid() = 0x0
[P1:T1:node] trace: ---- shim_getegid() = 0x0
[P1:T1:node] trace: ---- shim_getuid() = 0x0
[P1:T1:node] trace: ---- shim_geteuid() = 0x0
[P1:T1:node] trace: ---- shim_getgid() = 0x0
[P1:T1:node] trace: ---- shim_getegid() = 0x0
[P1:T1:node] trace: ---- shim_getuid() = 0x0
[P1:T1:node] trace: ---- shim_geteuid() = 0x0
[P1:T1:node] trace: ---- shim_getgid() = 0x0
[P1:T1:node] trace: ---- shim_getegid() = 0x0
[P1:T1:node] trace: ---- shim_futex(0x45c3b60, FUTEX_PRIVATE|FUTEX_WAKE, 2147483647, 0, 0x1, 202) ...
[P1:T1:node] trace: ---- return from shim_futex(...) = 0x0
[P1:T1:node] trace: ---- shim_getuid() = 0x0
[P1:T1:node] trace: ---- shim_geteuid() = 0x0
[P1:T1:node] trace: ---- shim_getgid() = 0x0
[P1:T1:node] trace: ---- shim_getegid() = 0x0
[P1:T1:node] trace: ---- shim_clock_getres(6, 0x6eef68b0) = 0x0
[P1:T1:node] trace: ---- shim_clock_gettime(6, 0x6eef68b0) = 0x0
[P1:T1:node] trace: ---- shim_epoll_create1(524288) = 0x3
[P1:T1:node] debug: Creating pipe: pipe.srv:f7d577ab1ce4c825fcb59e703491e212020d39cd1a589b01c35d2ec66b0ca3ac
[P1:T1:node] trace: ---- shim_pipe2(0x6eef6828, 524288) = 0x0
[P1:T1:node] trace: ---- shim_write(5, 0x6eef685f, 0x1) ...
[P1:T1:node] trace: ---- return from shim_write(...) = 0x1
[P1:T1:node] trace: ---- shim_futex(0x45bcc30, FUTEX_PRIVATE|FUTEX_WAKE, 2147483647, 0, 0x1, 202) ...
[P1:T1:node] trace: ---- return from shim_futex(...) = 0x0
[P1:T1:node] debug: Creating pipe: pipe.srv:5e5126039a5fa2e8f98cda1e059fb9f54e40568f7e2c831895b7431304f6d01f
[P1:T1:node] trace: ---- shim_pipe2(0x6eef6888, 526336) = 0x0
[P1:T1:node] warning: The app tried to use eventfd, but it's turned off (sys.insecure__allow_eventfd = false)
[P1:T1:node] trace: ---- shim_eventfd2(0, 526336) = -38
[P1:T1:node] trace: ---- shim_close(6) = 0x0
[P1:T1:node] trace: ---- shim_close(7) = 0x0
[P1:T1:node] trace: ---- shim_getpid() = 0x1
[P1:T1:node] trace: ---- shim_futex(0x45bcce8, FUTEX_PRIVATE|FUTEX_WAKE, 2147483647, 0, 0x1, 202) ...
[P1:T1:node] trace: ---- return from shim_futex(...) = 0x0
node[1]: ../src/tracing/agent.cc:55:node::tracing::Agent::Agent(): Assertion `(uv_loop_init(&tracing_loop_)) == (0)' failed.
[P1:T1:node] trace: ---- shim_write(2, 0x6eef42b0, 0x7d) ...
[P1:T1:node] trace: ---- return from shim_write(...) = 0x7d
[P1:T1:node] trace: ---- shim_futex(0x6ebb41e0, FUTEX_PRIVATE|FUTEX_WAKE, 2147483647, 0, 0x1, 202) ...
[P1:T1:node] trace: ---- return from shim_futex(...) = 0x0
[P1:T1:node] trace: ---- shim_write(2, 0x6eef39f0, 0x22) ...
[P1:T1:node] trace: ---- return from shim_write(...) = 0x22
 1: 0xa3aaf0 node::Abort() [node]
 2: 0xa3ab6e  [node]
[P1:T1:node] trace: ---- shim_write(2, 0x6eef39f0, 0x15) ...
[P1:T1:node] trace: ---- return from shim_write(...) = 0x15
 3: 0xb257ea node::tracing::Agent::Agent() [node]
[P1:T1:node] trace: ---- shim_write(2, 0x6eef39f0, 0x32) ...
[P1:T1:node] trace: ---- return from shim_write(...) = 0x32
[P1:T1:node] trace: ---- shim_write(2, 0x6eef39f0, 0x40) ...
[P1:T1:node] trace: ---- return from shim_write(...) = 0x40
 4: 0xa03115 node::InitializeOncePerProcess(int, char**) [node]
 5: 0xa03711 node::Start(int, char**) [node]
[P1:T1:node] trace: ---- shim_write(2, 0x6eef39f0, 0x2d) ...
[P1:T1:node] trace: ---- return from shim_write(...) = 0x2d
[P1:T1:node] trace: ---- shim_write(2, 0x6eef39f0, 0x63) ...
[P1:T1:node] trace: ---- return from shim_write(...) = 0x63
 6: 0x6e9c653d  [/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libc.so.6]
[P1:T1:node] trace: ---- shim_write(2, 0x6eef39f0, 0x74) ...
[P1:T1:node] trace: ---- return from shim_write(...) = 0x74
 7: 0x6e9c6600 __libc_start_main [/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libc.so.6]
[P1:T1:node] trace: ---- shim_write(2, 0x6eef39f0, 0x15) ...
[P1:T1:node] trace: ---- return from shim_write(...) = 0x15
[P1:T1:node] trace: ---- shim_rt_sigprocmask(UNBLOCK, [SIGABRT,], NULL, 0x8) = 0x0
[P1:T1:node] trace: ---- shim_gettid() = 0x1
[P1:T1:node] trace: ---- shim_getpid() = 0x1
[P1:T1:node] trace: ---- shim_tgkill(1, 1, [SIGABRT]) = 0x0
[P1:T1:node] debug: killed by signal 6
 8: 0x98c58c  [node]
[P1:T1:node] debug: clearing POSIX locks for pid 1
[P1:T1:node] debug: sync client shutdown: closing handles
[P1:T1:node] debug: sync client shutdown: waiting for confirmation
[P1:T1:node] debug: sync client shutdown: finished
[P1:shim] debug: IPC worker: exiting worker thread
[P1:T1:node] debug: process 1 exited with status 134
debug: DkProcessExit: Returning exit code 134

Is there something wrong with my image?

dimakuv commented 2 years ago

node[1]: ../src/tracing/agent.cc:55:node::tracing::Agent::Agent(): Assertion(uv_loop_init(&tracingloop)) == (0)' failed.`

This is a known "bug" in NodeJS. Please see: https://github.com/gramineproject/examples/blob/master/nodejs/nodejs.manifest.template#L13-L14

So the fix is to add to your manifest file this:

sys.insecure__allow_eventfd = true

dimakuv commented 2 years ago

Unfortunately, NodeJS doesn't output some meaningful message like eventfd() failed. This would make it more apparent to users that Gramine's insecure emulation of eventfd should be enabled in the manifest. (And GSC by default doesn't enable it, so you need to explicitly specify it in your manifest file.)

XNinety9 commented 2 years ago

Ok, added the sys.insecure__allow_eventfd = true command to the manifest, it works a bit better. Still not completely though, here is the log (it's a bit too big to fit inline).

dimakuv commented 2 years ago

Why do you think that it doesn't work? The last few lines are like this:

[P1:T1:node] trace: ---- shim_fstat(17, 0x118be2210) = 0x0
[P1:T1:node] trace: ---- shim_read(17, 0x10bdb6010, 0x177f) ...

So it seams that your workload just waits for some input on file descriptor 17 -- which is probably a network socket. So the workload waits for inputs from the (remote) user.

XNinety9 commented 2 years ago

Darn, the gist cut the log by about a half...

Ok, here is the interesting part of the log (aka "the end of it"):

[P1:T1:node] trace: ---- shim_futex(0x9f4c0910, FUTEX_CLOCK_REALTIME|FUTEX_WAIT_BITSET, 2, 0, 0, -1) ...
[P1:T1:node] warning: Ignoring FUTEX_CLOCK_REALTIME flag
[P1:T1:node] warning: Non-private futexes are not supported, assuming implicit FUTEX_PRIVATE_FLAG
[P1:T2:node] trace: ---- return from shim_epoll_wait(...) = 0x1
[P1:T2:node] trace: ---- shim_clock_gettime(6, 0x9f4bcc20) = 0x0
[P1:T2:node] trace: ---- shim_read(12, 0x9f4bc810, 0x400) ...
[P1:T2:node] trace: ---- return from shim_read(...) = 0x8
[P1:T3:node] trace: ---- return from shim_futex(...) = 0x0
[P1:T3:node] trace: ---- shim_futex(0x9f4dbeb0, FUTEX_PRIVATE|FUTEX_WAKE, 1, 0, 0, -1622676192) ...
[P1:T3:node] trace: ---- return from shim_futex(...) = 0x0
[P1:T3:node] trace: ---- shim_rt_sigprocmask(BLOCK, [SIGHUP,SIGINT,SIGQUIT,SIGILL,SIGTRAP,SIGABRT,SIGBUS,SIGFPE,SIGKILL,SIGUSR1,SIGSEGV,SIGUSR2,SIGPIPE,SIGALRM,SIGTERM,SIGSTKFLT,SIGCHLD,SIGCONT,SIGSTOP,SIGTSTP,SIGTTIN,SIGTTOU,SIGURG,SIGXCPU,SIGXFSZ,SIGVTALRM,SIGPROF,SIGWINCH,
[P1:T3:node] trace: SIGIO,SIGPWR,SIGSYS,SIG32,SIG34,SIG35,SIG36,SIG37,SIG38,SIG39,SIG40,SIG41,SIG42,SIG43,SIG44,SIG45,SIG46,SIG47,SIG48,SIG49,SIG50,SIG51,SIG52,SIG53,SIG54,SIG55,SIG56,SIG57,SIG58,SIG59,SIG60,SIG61,SIG62,SIG63,SIG64,], NULL, 0x8) = 0x0
[P1:T3:node] trace: ---- shim_madvise(0x9f43f000, 0x3b000, MADV_DONTNEED) = -38
[P1:T3:node] debug: ---- shim_exit (returning 0)
[P1:T2:node] trace: ---- shim_close(10) = 0x0
[P1:T2:node] trace: ---- shim_close(11) = 0x0
[P1:T2:node] trace: ---- shim_close(12) = 0x0
[P1:T2:node] trace: ---- shim_close(9) = 0x0
[P1:T2:node] trace: ---- shim_rt_sigprocmask(BLOCK, [SIGHUP,SIGINT,SIGQUIT,SIGILL,SIGTRAP,SIGABRT,SIGBUS,SIGFPE,SIGKILL,SIGUSR1,SIGSEGV,SIGUSR2,SIGPIPE,SIGALRM,SIGTERM,SIGSTKFLT,SIGCHLD,SIGCONT,SIGSTOP,SIGTSTP,SIGTTIN,SIGTTOU,SIGURG,SIGXCPU,SIGXFSZ,SIGVTALRM,SIGPROF,SIGWINCH,
[P1:T2:node] trace: SIGIO,SIGPWR,SIGSYS,SIG32,SIG34,SIG35,SIG36,SIG37,SIG38,SIG39,SIG40,SIG41,SIG42,SIG43,SIG44,SIG45,SIG46,SIG47,SIG48,SIG49,SIG50,SIG51,SIG52,SIG53,SIG54,SIG55,SIG56,SIG57,SIG58,SIG59,SIG60,SIG61,SIG62,SIG63,SIG64,], NULL, 0x8) = 0x0
[P1:T2:node] trace: ---- shim_madvise(0x9f480000, 0x3b000, MADV_DONTNEED) = -38
[P1:T2:node] debug: ---- shim_exit (returning 0)
[P1:shim] debug: Async worker thread started
[P1:T4:node] debug: Installed async event at 1655903529125192
[P1:T6:node] debug: Installed async event at 1655903529125205
[P1:T3:node] debug: Installed async event at 1655903529128863
[P1:T2:node] debug: Installed async event at 1655903529129939
[P1:T5:node] debug: Installed async event at 1655903529125333
[P1:shim] debug: Thread exited, cleaning up
[P1:shim] debug: Thread exited, cleaning up
[P1:shim] debug: Thread exited, cleaning up
[P1:shim] debug: Thread exited, cleaning up
[P1:shim] debug: Thread exited, cleaning up
[P1:T1:node] trace: ---- return from shim_futex(...) = 0x0
[P1:T1:node] trace: ---- shim_futex(0x9f3fd910, FUTEX_CLOCK_REALTIME|FUTEX_WAIT_BITSET, 5, 0, 0, -1) ...
[P1:T1:node] warning: Ignoring FUTEX_CLOCK_REALTIME flag
[P1:T1:node] warning: Non-private futexes are not supported, assuming implicit FUTEX_PRIVATE_FLAG
[P1:T1:node] trace: ---- return from shim_futex(...) = 0x0
[P1:T1:node] trace: ---- shim_clock_gettime(6, 0x9fb20970) = 0x0
[P1:T1:node] debug: epoll: added 6 (0x1d7d04b80) to epoll handle 0x1d7d04538
[P1:T1:node] trace: ---- shim_epoll_ctl(3, ADD, 6, {.events=EPOLLIN, .data=0x6}) = 0x0
[P1:T1:node] debug: epoll: added 8 (0x1d7d04fb0) to epoll handle 0x1d7d04538
[P1:T1:node] trace: ---- shim_epoll_ctl(3, ADD, 8, {.events=EPOLLIN, .data=0x8}) = 0x0
[P1:T1:node] trace: ---- shim_epoll_wait(3, 0x9fb1d960, 1024, 0) ...
[P1:T1:node] trace: ---- return from shim_epoll_wait(...) = 0x0
[P1:T1:node] trace: ---- shim_clock_gettime(6, 0x9fb1d860) = 0x0
[P1:T1:node] trace: ---- shim_clock_gettime(6, 0x9fb20970) = 0x0
[P1:T1:node] trace: ---- shim_close(6) = 0x0
[P1:T1:node] trace: ---- shim_close(7) = 0x0
[P1:T1:node] trace: ---- shim_close(8) = 0x0
[P1:T1:node] trace: ---- shim_close(3) = 0x0
[P1:T1:node] trace: ---- shim_close(4) = 0x0
[P1:T1:node] trace: ---- shim_close(5) = 0x0
[P1:T1:node] trace: ---- shim_fstat(0, 0x9fb20930) = 0x0
[P1:T1:node] trace: ---- shim_fcntl(0, F_GETFL, 0x9fb20930) = 0x1
[P1:T1:node] trace: ---- shim_fstat(1, 0x9fb20930) = 0x0
[P1:T1:node] trace: ---- shim_fcntl(1, F_GETFL, 0x9fb20930) = 0x401
[P1:T1:node] trace: ---- shim_fstat(2, 0x9fb20930) = 0x0
[P1:T1:node] trace: ---- shim_fcntl(2, F_GETFL, 0x9fb20930) = 0x401
[P1:T1:node] debug: ---- shim_exit_group (returning 1)
[P1:T7:node] debug: Installed async event at 1655903529135821
[P1:T1:node] debug: clearing POSIX locks for pid 1
[P1:T1:node] debug: sync client shutdown: closing handles
[P1:T1:node] debug: sync client shutdown: waiting for confirmation
[P1:T1:node] debug: sync client shutdown: finished
[P1:shim] debug: Thread exited, cleaning up
[P1:shim] debug: IPC worker: exiting worker thread
[P1:shim] debug: Async worker thread terminated
[P1:T1:node] debug: process 1 exited with status 1
debug: DkProcessExit: Returning exit code 1
internal/validators.js:124
    throw new ERR_INVALID_ARG_TYPE(name, 'string', value);
    ^

TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received undefined
    at new NodeError (internal/errors.js:322:7)
    at validateString (internal/validators.js:124:11)
    at Object.join (path.js:1148:7)
    at Object.<anonymous> (/usr/src/node-red/node_modules/node-red/red.js:115:50)
    at Module._compile (internal/modules/cjs/loader.js:1085:14)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
    at Module.load (internal/modules/cjs/loader.js:950:32)
    at Function.Module._load (internal/modules/cjs/loader.js:790:12)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:75:12)
    at internal/main/run_main_module.js:17:47 {
  code: 'ERR_INVALID_ARG_TYPE'
}

dimakuv commented 2 years ago

TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received undefined

Do you know what it can refer to? Looks like something about command-line arguments? Did you try loader.insecure__use_cmdline_argv = true? See https://gramine.readthedocs.io/en/latest/manifest-syntax.html#command-line-arguments

XNinety9 commented 2 years ago

Same result, no Js error this time though:

[P1:T1:node] trace: ---- return from shim_futex(...) = 0x0
[P1:T1:node] trace: ---- shim_futex(0xc3826910, FUTEX_CLOCK_REALTIME|FUTEX_WAIT_BITSET, 4, 0, 0, -1) ...
[P1:T1:node] warning: Ignoring FUTEX_CLOCK_REALTIME flag
[P1:T1:node] warning: Non-private futexes are not supported, assuming implicit FUTEX_PRIVATE_FLAG
[P1:T1:node] trace: ---- return from shim_futex(...) = 0x0
[P1:T1:node] trace: ---- shim_futex(0xc37e5910, FUTEX_CLOCK_REALTIME|FUTEX_WAIT_BITSET, 5, 0, 0, -1) ...
[P1:T1:node] warning: Ignoring FUTEX_CLOCK_REALTIME flag
[P1:T1:node] warning: Non-private futexes are not supported, assuming implicit FUTEX_PRIVATE_FLAG
[P1:T1:node] trace: ---- return from shim_futex(...) = 0x0
[P1:T1:node] trace: ---- shim_clock_gettime(6, 0xc3f08970) = 0x0
[P1:T1:node] debug: epoll: added 6 (0x1d7d04b80) to epoll handle 0x1d7d04538
[P1:T1:node] trace: ---- shim_epoll_ctl(3, ADD, 6, {.events=EPOLLIN, .data=0x6}) = 0x0
[P1:T1:node] debug: epoll: added 8 (0x1d7d04fb0) to epoll handle 0x1d7d04538
[P1:T1:node] trace: ---- shim_epoll_ctl(3, ADD, 8, {.events=EPOLLIN, .data=0x8}) = 0x0
[P1:T1:node] trace: ---- shim_epoll_wait(3, 0xc3f05960, 1024, 0) ...
[P1:T1:node] trace: ---- return from shim_epoll_wait(...) = 0x0
[P1:T1:node] trace: ---- shim_clock_gettime(6, 0xc3f05860) = 0x0
[P1:T1:node] trace: ---- shim_clock_gettime(6, 0xc3f08970) = 0x0
[P1:T1:node] trace: ---- shim_close(6) = 0x0
[P1:T1:node] trace: ---- shim_close(7) = 0x0
[P1:T1:node] trace: ---- shim_close(8) = 0x0
[P1:T1:node] trace: ---- shim_close(3) = 0x0
[P1:T1:node] trace: ---- shim_close(4) = 0x0
[P1:T1:node] trace: ---- shim_close(5) = 0x0
[P1:T1:node] trace: ---- shim_fstat(0, 0xc3f08930) = 0x0
[P1:T1:node] trace: ---- shim_fcntl(0, F_GETFL, 0xc3f08930) = 0x1
[P1:T1:node] trace: ---- shim_fstat(1, 0xc3f08930) = 0x0
[P1:T1:node] trace: ---- shim_fcntl(1, F_GETFL, 0xc3f08930) = 0x401
[P1:T1:node] trace: ---- shim_fstat(2, 0xc3f08930) = 0x0
[P1:T1:node] trace: ---- shim_fcntl(2, F_GETFL, 0xc3f08930) = 0x401
[P1:T1:node] debug: ---- shim_exit_group (returning 1)
[P1:T7:node] debug: Installed async event at 1655982843926858
[P1:T1:node] debug: clearing POSIX locks for pid 1
[P1:T1:node] debug: sync client shutdown: closing handles
[P1:T1:node] debug: sync client shutdown: waiting for confirmation
[P1:T1:node] debug: sync client shutdown: finished
[P1:shim] debug: Thread exited, cleaning up
[P1:shim] debug: IPC worker: exiting worker thread
[P1:shim] debug: Async worker thread terminated
[P1:T1:node] debug: process 1 exited with status 1
debug: DkProcessExit: Returning exit code 1

I'm starting to think that shoving NodeRed+NodeJs into an enclave might be way too much, even with the help of Gramine.

dimakuv commented 2 years ago

Well, but you're making progress :)

Any error from NodeRed? The Gramine logs look fine. Where is the error?

XNinety9 commented 2 years ago

I activated the most verbose logging mode in node red, launched the docker image as-is, NodeRed indeed logs a lot of info to the console and launches fine.

Gamine build/sign the image, launch it, I only get Gramine's logs, same as above. From what I see NodeRed doesn't even start.

Here is my manifest, if it can help...

loader.pal_internal_mem_size = "512M"

sgx.enclave_size = "8G"
sgx.thread_num = 128

sgx.debug = true
sys.insecure__allow_eventfd = true

loader.insecure__use_cmdline_argv = true

sgx.trusted_files = [
  "file:/gramine/app_files/entrypoint.manifest",  # unused entry, only to test merging of manifests
]

mkow commented 2 years ago

Aren't you missing the entrypoint from it? How can Gramine know what binary to start? I assume you just copied the manifest incorrectly?

XNinety9 commented 2 years ago

From the doc, gsc build uses the docker image's entrypoint. The OpenVino example confirms that too.

mkow commented 2 years ago

Oh, I missed that you're using GSC, not vanilla Gramine.

dimakuv commented 2 years ago

@p-vernaeckt Your manifest has a couple problems, which may be the cause of your NodeRed not running properly.

sgx.debug = true -- this is wrong. In GSC, you're supposed to give a -d option to gsc build. GSC always ignores the sgx.debug option specified in the manifest. It shouldn't harm in your tests, but is redundant in your manifest.
loader.insecure__use_cmdline_argv = true -- this is wrong. In GSC, you're supposed to give a --insecure-args option to gsc build. GSC always ignores the loader.insecure__use_cmdline_argv option specified in the manifest. So what happens is that GSC builds your NodeRed with command-line arguments hard-coded in the original Docker image (inside ENTRYPOINT and CMD).
sgx.trusted_files = ... -- this is harmless but redundant. Just remove these lines.

Hm, ok, GSC is not that user-friendly :) We'll need to fix this.

XNinety9 commented 2 years ago

I removed a few lines from my manifest, which looks like this now:

loader.pal_internal_mem_size = "512M"

sgx.enclave_size = "8G"
sgx.thread_num = 128

sys.insecure__allow_eventfd = true

However, I already use the CLI args you mention:

./gsc build -d -c config.yaml --insecure-args IMAGE_ID nodered.manifest

dimakuv commented 2 years ago

Yes, this looks good now.

Well, your workload will need more debugging and analysis then...

mkow commented 2 months ago

This issue seems outdated and I assume @XNinety9 lost interest in debugging this, if that's true then I'll close this issue. @XNinety9: If you want us to help in debugging this, please provide us with a minimized reproducer (no GSC, minimal JS dependencies, etc)

gramineproject / gramine

Packaging a complete Node app with Gramine #655