allvphx commented 1 year ago

Describe the bug

I got [ERROR] occlum-pal: occlum_ecall_init returns EFAULT when running MySQL demo with occlum gdb.

To reproduce

Steps to reproduce the behavior:

Replace line 40 in run_mysql_server.sh with the command: occlum gdb /bin/${MYSQLD}, and run it.
Enable sgx_emmt and execute run --user=root inside gdb.

After making these modifications, I observe the following outcome:

Starting program: /root/demos/mysql/occlum_instance/build/bin/occlum-run -user=root
warning: Error disabling address space randomization: Operation not permitted
detect urts is loaded, initializing
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
detect urts is loaded, initializing
add-symbol-file '/root/demos/mysql/occlum_instance/build/lib/libocclum-libos.signed.so' 0x7f380004a040 -s .interp 0x7f38000002e0  -s .note.gnu.build-id 0x7f38000002fc  -s .gnu.hash 0x7f3800000320  -s .dynsym 0x7f3800000358  -s .dynstr 0x7f3800000400  -s .gnu.version 0x7f380000047a  -s .gnu.version_d 0x7f3800000488  -s .rela.dyn 0x7f38000004c0  -s .plt 0x7f380004a000  -s .plt.got 0x7f380004a010  -s .nipx 0x7f38002f86d0  -s .rodata 0x7f38002fa000  -s .niprod 0x7f380035ce40  -s .builtin_config 0x7f380035d740  -s .eh_frame_hdr 0x7f380035d770  -s .eh_frame 0x7f3800368180  -s .gcc_except_table 0x7f38003acbac  -s .tbss 0x7f38003c4be0  -s .init_array 0x7f38003c4be0  -s .fini_array 0x7f38003c4bf8  -s .data.rel.ro 0x7f38003c4c60  -s .dynamic 0x7f38003f8720  -s .got 0x7f38003f88c0  -s .data 0x7f38003fa000  -s .nipd 0x7f38003fd340  -s .bss 0x7f38003fd380 
thread '<unnamed>' panicked at 'explicit panic', src/fs/rootfs.rs:31:13
stack backtrace:
0: rust_begin_unwind
1: core::panicking::panic_fmt
         at library/core/src/panicking.rs:142
2: core::panicking::panic
         at library/core/src/panicking.rs:48
3: spin::once::Once<T>::call_once
4: occlum_libos_core_rs::fs::fs_view::FsView::lookup_inode
5: occlum_libos_core_rs::fs::fs_view::FsView::open_file
6: occlum_libos_core_rs::util::host_file_util::write_host_file
7: occlum_libos_core_rs::entry::parse_host_files
note: Some details are omitted, call backtrace::enable_backtrace() with 'PrintFormat::Full' for a verbose backtrace.
[ERROR] occlum-pal: occlum_ecall_init returns EFAULT (line 141, file src/pal_api.c)
Enclave: "/root/demos/mysql/occlum_instance/build/lib/libocclum-libos.signed.so"
[Peak stack used]: 16 KB
[Peak heap used]:  32 KB
[Peak reserved memory used]:  0 KB
remove-symbol-file -a 139878495199296
[Inferior 1 (process 66742) exited with code 01]

Expected behavior

Logs

Environment

HW: SGX2
OS: Ubuntu20.04
Occlum version: 0.29.6

Additional context

The configuration file:

{
  "resource_limits": {
    "kernel_space_heap_size": "1000MB",
    "kernel_space_stack_size": "1MB",
    "user_space_size": "8000MB",
    "max_num_of_threads": 96
  },
  "process": {
    "default_stack_size": "4MB",
    "default_heap_size": "32MB",
    "default_mmap_size": "100MB"
  },
  "entry_points": [
    "/bin"
  ],
  "env": {
    "default": [
      "OCCLUM=yes"
    ],
    "untrusted": [
      "EXAMPLE"
    ]
  },
  "metadata": {
    "product_id": 0,
    "version_number": 0,
    "debuggable": true,
    "enable_kss": false,
    "family_id": {
      "high": "0x0",
      "low": "0x0"
    },
    "ext_prod_id": {
      "high": "0x0",
      "low": "0x0"
    },
    "pkru": 0
  },
  "mount": [
    {
      "target": "/",
      "type": "unionfs",
      "options": {
        "layers": [
          {
            "target": "/",
            "type": "sefs",
            "source": "./build/mount/__ROOT",
            "options": {
              "MAC": ""
            }
          },
          {
            "target": "/",
            "type": "sefs",
            "source": "./run/mount/__ROOT"
          }
        ]
      }
    },
    {
      "target": "/host",
      "type": "hostfs",
      "source": "."
    }
  ]
}

Possible solution/Implementation

Should I make adjustments to the config file to resolve this issue? If yes, could you guide me on the specific modifications required?"

lucassong-mh commented 1 year ago

Hi @allvphx , thanks for using Occlum and try some demos. According to your error log, it seems the Occlum failed to initialize the rootfs at a very early stage. In fact, we haven't tested occlum gdb on MySQL yet, not to mention sgx_emmt. I don't know your specific requirement, but enable sgx_emmt would inform you enclave memory usage mixed by MySQL-the app and Occlum-the os, this can be confused. If you'd like to track memory consumption, you can build a BusyBox along with MySQL showed here, then enter the instance directory after Occlum&MySQL started, run occlum exec /bin/busybox free or occlum exec /bin/busybox cat /proc/meminfo to see memory usage of MySQL.

allvphx commented 1 year ago

Thank you for your prompt reply. To clarify, I'm interested in determining the total enclave memory usage of both MySQL and Occlum.

May I ask if the busybox method includes the enclave memory used by Occlum? Also, I'd like to know usually how much extra enclave memory is required to run an application with Occlum.

jessehui commented 1 year ago

What type of memory concerns you? Are you talking about EPC or untrusted memory usage? Are you talking about the physical memory in use or the virtual memory?

If you want to know the extra memory used by Occlum, you can configure the value of kernel_heap_size and kernel_stack_size to get the minimum value to run Occlum. And these values are the memory amount that can be regarded as the EPC memory used by Occlum.

occlum / occlum

[BUG] Got error when run mysql demo with gdb. #1342