search

gramineproject / gramine

A library OS for Linux multi-process applications, with Intel SGX support
GNU Lesser General Public License v3.0
591 stars 195 forks source link

Error whern running MySQL #382

Closed JorelHan closed 2 years ago

JorelHan commented 2 years ago

Description of the problem

I encountered some problems when testing MySQL performance in a Gramine environment.

Version information:

os version: Ubuntu 20.04
kernel version: 4.15.0.147-generic

  1. Start MySQL： MySQL run command：

        gramine-sgx-sign --key /enclave-key.pem --manifest entrypoint.manifest --output entrypoint.manifest.sgx && \
    gramine-sgx-get-token -o /entrypoint.token -s /entrypoint.sig && \
    gramine-sgx /entrypoint -c "gosu mysql mysqld --defaults-file=/var/lib/mysql/mysql.cnf --datadir /tmp/mysql"

    MySQL log：

    Attributes:
    size:        0x200000000
    thread_num:  1024
    isv_prod_id: 0
    isv_svn:     0
    attr.flags:  0x4
    attr.xfrm:   0x3
    misc_select: 0x0
    Memory:
        00000001ffcfd000-0000000200000000 [REG:R--] (manifest) measured
        00000001fdcfd000-00000001ffcfd000 [REG:RW-] (ssa) measured
        00000001fd8fd000-00000001fdcfd000 [TCS:---] (tcs) measured
        00000001fd4fd000-00000001fd8fd000 [REG:RW-] (tls) measured
        00000001fd4bd000-00000001fd4fd000 [REG:RW-] (stack) measured
        ......
        00000001ed57d000-00000001ed5bd000 [REG:RW-] (stack) measured
        00000001ed53d000-00000001ed57d000 [REG:RW-] (stack) measured
        00000001ed4fd000-00000001ed53d000 [REG:RW-] (stack) measured
        00000001ed4ed000-00000001ed4fd000 [REG:RW-] (sig_stack) measured
        00000001ed4dd000-00000001ed4ed000 [REG:RW-] (sig_stack) measured
        ......
        00000001e953d000-00000001e954d000 [REG:RW-] (sig_stack) measured
        00000001e952d000-00000001e953d000 [REG:RW-] (sig_stack) measured
        00000001e951d000-00000001e952d000 [REG:RW-] (sig_stack) measured
        00000001e950d000-00000001e951d000 [REG:RW-] (sig_stack) measured
        00000001e94fd000-00000001e950d000 [REG:RW-] (sig_stack) measured
        00000001e50d7000-00000001e5124000 [REG:R-X] (code) measured
        00000001e5124000-00000001e94fd000 [REG:RW-] (data) measured
        0000000000010000-00000001e50d7000 [REG:RWX] (free)
    Measurement:
        1212e78d7c616774039c391e068192aeee14b08b5f833be5cbf307852bf433dd
    Attributes:
        mr_enclave:  1212e78d7c616774039c391e068192aeee14b08b5f833be5cbf307852bf433dd
        mr_signer:   e7b1215e487b2e89f0862e0daa0a0d019635ab156b49a2bbc7657ab389cc2784
        isv_prod_id: 0
        isv_svn:     0
        attr.flags:  0000000000000004
        attr.xfrm:   00000000000000e7
        mask.flags:  ffffffffffffffff
        mask.xfrm:   ffffffffffffff1b
        misc_select: 00000000
        misc_mask:   ffffffff
        modulus:     57b540f9f44d06690246d353f24af9ab...
        exponent:    3
        signature:   ed4aab6f1692f07e3dc7e422e0241050...
        date:        2022-02-07
    2022-02-07T05:02:57.280571Z 0 [ERROR] [MY-010986] [Server] Unable to determine CAP_SYS_NICE capability.
    2022-02-07T05:02:57.283293Z 0 [Warning] [MY-010101] [Server] Insecure configuration for --secure-file-priv: Location is accessible to all OS users. Consider choosing a different directory.
    2022-02-07T05:02:57.283379Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.27-0ubuntu0.20.04.1) starting as process 1
    2022-02-07T05:02:57.290658Z 0 [Warning] [MY-010122] [Server] One can only use the --user switch if running as root
    2022-02-07T05:02:57.355376Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
    2022-02-07T05:02:57.428048Z 1 [ERROR] [MY-012585] [InnoDB] Linux Native AIO interface is not supported on this platform. Please check your OS documentation and install appropriate binary of InnoDB.
    2022-02-07T05:02:57.428203Z 1 [Warning] [MY-012654] [InnoDB] Linux Native AIO disabled.
    2022-02-07T05:02:58.611650Z 1 [Warning] [MY-012128] [InnoDB] posix_fallocate(): Failed to preallocate data for file ./#innodb_temp/temp_1.ibt, desired size 81920 Operating system error number 38 - Function not implemented. Check that the disk is not full or a disk quota exceeded. Make sure the file system supports this function. Refer to your operating system documentation for operating system error code information.
    2022-02-07T05:02:58.619057Z 1 [Warning] [MY-012579] [InnoDB] fallocate(14, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, 0, 16384) returned errno: 38
    2022-02-07T05:02:58.629421Z 1 [Warning] [MY-012128] [InnoDB] posix_fallocate(): Failed to preallocate data for file ./#innodb_temp/temp_2.ibt, desired size 81920 Operating system error number 38 - Function not implemented. Check that the disk is not full or a disk quota exceeded. Make sure the file system supports this function. Refer to your operating system documentation for operating system error code information.
    2022-02-07T05:02:58.637465Z 1 [Warning] [MY-012579] [InnoDB] fallocate(14, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, 0, 16384) returned errno: 38
    2022-02-07T05:02:58.647851Z 1 [Warning] [MY-012128] [InnoDB] posix_fallocate(): Failed to preallocate data for file ./#innodb_temp/temp_3.ibt, desired size 81920 Operating system error number 38 - Function not implemented. Check that the disk is not full or a disk quota exceeded. Make sure the file system supports this function. Refer to your operating system documentation for operating system error code information.
    2022-02-07T05:02:58.655900Z 1 [Warning] [MY-012579] [InnoDB] fallocate(14, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, 0, 16384) returned errno: 38
    2022-02-07T05:02:58.666189Z 1 [Warning] [MY-012128] [InnoDB] posix_fallocate(): Failed to preallocate data for file ./#innodb_temp/temp_4.ibt, desired size 81920 Operating system error number 38 - Function not implemented. Check that the disk is not full or a disk quota exceeded. Make sure the file system supports this function. Refer to your operating system documentation for operating system error code information.
    2022-02-07T05:02:58.674323Z 1 [Warning] [MY-012579] [InnoDB] fallocate(14, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, 0, 16384) returned errno: 38
    2022-02-07T05:02:58.684706Z 1 [Warning] [MY-012128] [InnoDB] posix_fallocate(): Failed to preallocate data for file ./#innodb_temp/temp_5.ibt, desired size 81920 Operating system error number 38 - Function not implemented. Check that the disk is not full or a disk quota exceeded. Make sure the file system supports this function. Refer to your operating system documentation for operating system error code information.
    2022-02-07T05:02:58.692758Z 1 [Warning] [MY-012579] [InnoDB] fallocate(14, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, 0, 16384) returned errno: 38
    2022-02-07T05:02:58.703113Z 1 [Warning] [MY-012128] [InnoDB] posix_fallocate(): Failed to preallocate data for file ./#innodb_temp/temp_6.ibt, desired size 81920 Operating system error number 38 - Function not implemented. Check that the disk is not full or a disk quota exceeded. Make sure the file system supports this function. Refer to your operating system documentation for operating system error code information.
    2022-02-07T05:02:58.711208Z 1 [Warning] [MY-012579] [InnoDB] fallocate(14, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, 0, 16384) returned errno: 38
    2022-02-07T05:02:58.721598Z 1 [Warning] [MY-012128] [InnoDB] posix_fallocate(): Failed to preallocate data for file ./#innodb_temp/temp_7.ibt, desired size 81920 Operating system error number 38 - Function not implemented. Check that the disk is not full or a disk quota exceeded. Make sure the file system supports this function. Refer to your operating system documentation for operating system error code information.
    2022-02-07T05:02:58.729881Z 1 [Warning] [MY-012579] [InnoDB] fallocate(14, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, 0, 16384) returned errno: 38
    2022-02-07T05:02:58.740183Z 1 [Warning] [MY-012128] [InnoDB] posix_fallocate(): Failed to preallocate data for file ./#innodb_temp/temp_8.ibt, desired size 81920 Operating system error number 38 - Function not implemented. Check that the disk is not full or a disk quota exceeded. Make sure the file system supports this function. Refer to your operating system documentation for operating system error code information.
    2022-02-07T05:02:58.748163Z 1 [Warning] [MY-012579] [InnoDB] fallocate(14, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, 0, 16384) returned errno: 38
    2022-02-07T05:02:58.758642Z 1 [Warning] [MY-012128] [InnoDB] posix_fallocate(): Failed to preallocate data for file ./#innodb_temp/temp_9.ibt, desired size 81920 Operating system error number 38 - Function not implemented. Check that the disk is not full or a disk quota exceeded. Make sure the file system supports this function. Refer to your operating system documentation for operating system error code information.
    2022-02-07T05:02:58.766676Z 1 [Warning] [MY-012579] [InnoDB] fallocate(14, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, 0, 16384) returned errno: 38
    2022-02-07T05:02:58.777000Z 1 [Warning] [MY-012128] [InnoDB] posix_fallocate(): Failed to preallocate data for file ./#innodb_temp/temp_10.ibt, desired size 81920 Operating system error number 38 - Function not implemented. Check that the disk is not full or a disk quota exceeded. Make sure the file system supports this function. Refer to your operating system documentation for operating system error code information.
    2022-02-07T05:02:58.785159Z 1 [Warning] [MY-012579] [InnoDB] fallocate(14, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, 0, 16384) returned errno: 38
    2022-02-07T05:02:59.667687Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
    2022-02-07T05:03:00.756230Z 0 [Warning] [MY-013746] [Server] A deprecated TLS version TLSv1 is enabled for channel mysql_main
    2022-02-07T05:03:00.756337Z 0 [Warning] [MY-013746] [Server] A deprecated TLS version TLSv1.1 is enabled for channel mysql_main
    2022-02-07T05:03:00.758712Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
    2022-02-07T05:03:00.758864Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
    2022-02-07T05:03:00.765887Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/tmp' in the path is accessible to all OS users. Consider choosing a different directory.
    2022-02-07T05:03:01.180664Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock
    2022-02-07T05:03:01.181347Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.27-0ubuntu0.20.04.1'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu).

  2. TPS and QPS are very low when using sysbench for insert and update operations. Threads: 24 TPS：690 QPS：3453 sysbench run command：

    sysbench --mysql-host=$MYSQLHOST --mysql-port=$MYSQLPORT --mysql-user=$USER --mysql-password=$PWD --mysql-db=sysbench_test ./mysqltt_use_prepare.lua --report-interval=10 --time=120 --table_size=10000 --threads=24

    MySQL log：

    2022-02-07T05:04:09.662888Z 8 [Warning] [MY-012128] [InnoDB] posix_fallocate(): Failed to preallocate data for file ./sbtest/sbtest.ibd, desired size 114688 Operating system error number 38 - Function not implemented. Check that the disk is not full or a disk quota exceeded. Make sure the file system supports this function. Refer to your operating system documentation for operating system error code information.
2022-02-07T05:04:09.673467Z 8 [Warning] [MY-012579] [InnoDB] fallocate(38, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, 0, 16384) returned errno: 38
2022-02-07T05:04:09.700575Z 8 [ERROR] [MY-012144] [InnoDB] posix_fallocate(): Failed to preallocate data for file ./sbtest/sbtest.ibd, desired size 16384 bytes. Operating system error number 38. Check that the disk is not full or a disk quota exceeded. Make sure the file system supports this function. Refer to your operating system documentation for operating system error code information.
......
2022-02-07T05:04:18.363721Z 17 [ERROR] [MY-012144] [InnoDB] posix_fallocate(): Failed to preallocate data for file ./sbtest/sbtest.ibd, desired size 393216 bytes. Operating system error number 38. Check that the disk is not full or a disk quota exceeded. Make sure the file system supports this function. Refer to your operating system documentation for operating system error code information.
2022-02-07T05:04:18.375154Z 17 [ERROR] [MY-012144] [InnoDB] posix_fallocate(): Failed to preallocate data for file ./sbtest/sbtest.ibd, desired size 1048576 bytes. Operating system error number 38. Check that the disk is not full or a disk quota exceeded. Make sure the file system supports this function. Refer to your operating system documentation for operating system error code information.
......
2022-02-07T05:04:35.001157Z 28 [ERROR] [MY-012144] [InnoDB] posix_fallocate(): Failed to preallocate data for file ./sbtest/sbtest.ibd, desired size 4194304 bytes. Operating system error number 38. Check that the disk is not full or a disk quota exceeded. Make sure the file system supports this function. Refer to your operating system documentation for operating system error code information.
.....

  3. Sysbench error and terminated mysql process, when sysbench threads is 64. sysbench run command：

    sysbench --mysql-host=172.18.165.50 --mysql-port=307 --db-driver=mysql --mysql-user=root --mysql-password=root --mysql-db=sbtest ./mysqltt_use_prepare.lua --report-interval=10 --time=120 --table_size=10000 --threads=64 prepare && \
sysbench --mysql-host=172.18.165.50 --mysql-port=307 --db-driver=mysql --mysql-user=root --mysql-password=root --mysql-db=sbtest ./mysqltt_use_prepare.lua --report-interval=10 --time=120 --table_size=10000 --threads=64 run

    sysbench log：

    sysbench 1.0.11 (using system LuaJIT 2.1.0-beta3)

Creating database sbtest
Droping table sbtest.sbtest
Creating table sbtest.sbtest
Prepare OK!

+ sysbench --mysql-host=172.18.165.50 --mysql-port=307 --db-driver=mysql --mysql-user=root --mysql-password=root --mysql-db=sbtest ./mysqltt_use_prepare.lua --report-interval=10 --time=120 --table_size=10000 --threads=64 run
sysbench 1.0.11 (using system LuaJIT 2.1.0-beta3)

Running the test with following options:
Number of threads: 64
Report intermediate results every 10 second(s)
Initializing random number generator from current time

Initializing worker threads...

Threads started!

[ 10s ] thds: 64 tps: 478.77 qps: 2419.05 (r/w/o: 485.07/970.14/963.84) lat (ms,95%): 248.83 err/s: 0.00 reconn/s: 0.00
[ 20s ] thds: 64 tps: 484.93 qps: 2407.23 (r/w/o: 478.73/959.45/969.05) lat (ms,95%): 282.25 err/s: 0.00 reconn/s: 0.00
[ 30s ] thds: 64 tps: 444.19 qps: 2238.75 (r/w/o: 450.49/898.98/889.28) lat (ms,95%): 253.35 err/s: 0.00 reconn/s: 0.00
[ 40s ] thds: 64 tps: 468.31 qps: 2338.65 (r/w/o: 468.01/934.02/936.62) lat (ms,95%): 253.35 err/s: 0.00 reconn/s: 0.00
[ 50s ] thds: 64 tps: 443.20 qps: 2218.70 (r/w/o: 443.50/888.80/886.40) lat (ms,95%): 267.41 err/s: 0.00 reconn/s: 0.00
FATAL: mysql_stmt_execute() returned error 1114 (The table 'sbtest' is full) for query 'insert into sbtest.sbtest(
            transmsn_dt_tm,     acq_ins_id_cd,              sys_tra_no,              tfr_in_in,             related_key,
            acq_ins_tp,         fwd_ins_tp,                 rcv_ins_tp,              iss_ins_tp,            rsn_cd, 
            sti_in,             sti_takeout_in,             fee_in,                  cross_dist_in,         disc_cd, 
            dom_allot_cd,       dif_allot_cd,               trans_rcv_ts,            to_ts,                 moni_dist, 
            risk_in,            trans_id,                   trans_id_conv,           trans_seq,             trans_seq_conv, 
            trans_tp,           resnd_num,                  settle_dt,               settle_mon,            settle_d, 
            cycle_no,           sms_dms_conv_in,            msg_tp,                  msg_tp_conv,           pri_acct_no, 
            pri_acct_no_conv,   bin,                        cups_card_in,            cups_sig_card_in,      card_class, 
            card_attr,          trans_chnl,                 card_media,              card_brand,            proc_cd,
            proc_cd_conv,       trans_at,                   trans_curr_cd,           fwd_settle_at,         rcv_settle_at, 
            fwd_settle_conv_rt, rcv_settle_conv_rt,         fwd_settle_curr_cd,      rcv_settle_curr_cd,    cdhd_at, 
            cdhd_conv_rt,       cdhd_curr_cd,               fwd_disc_at,             rcv_disc_at,           disc_curr_cd, 
            fee_dir_in,         dms_trans_id,               iss_ins_id_cd,           related_ins_id_cd,     related_bin, 
            loc_trans_tm,       loc_trans_dt,               conv_dt,                 mchnt_tp,              acq_ins_cntry_cd, 
            ext_pan_cntry_cd,   pos_entry_md_cd,            card_seq_id,             pos_cond_cd,           pos_cond_cd_conv, 
            pos_pin_capture_cd, fwd_ins_id_cd,              retri_ref_no,            term_id,               mchnt_cd, 
            card_accptr_nm_loc, sec_related_ctrl_inf,       addn_pos_inf,            orig_msg_tp,           orig_sys_tra_no, 
            orig_sys_tra_no_conv,  orig_transmsn_dt_tm,     orig_acq_ins_id_cd,      orig_fwd_ins_id_cd,    rcv_ins_id_cd, 
            resv_fld1,             resv_fld2,               fwd_proc_in,             rcv_proc_in,           trans_st, 
            vfy_rslt,              vfy_fee_cd,              trans_fin_ts,            expire_dt,             auth_id_resp_cd, 
            iss_resp_cd,           iss_resp_cd_conv,        acq_resp_cd,             acq_resp_cd_conv,      addn_pvt_dat, 
            addn_at,               iss_addn_dat,            ic_res_dat,              repl_at,               iss_ins_res, 
            auth_resp,             moni_in,                 auth_id_resp_cd_conv,    ic_flds,               cups_def_fld, 
            ext_header,            track_2_dat,             track_3_dat,             id_no,                 tfr_in_acct_id, 
            tfr_out_acct_id,       cups_res,                acq_ins_res,             addn_priv_dat,         resv_fld3, 
            resv_fld4,             resv_fld5,               resv_fld6)
            value( ?,     ?,     ?,     ?,    '',
                '',    '',    '',    '',    '',
                '',    '',    '',    '',    '',
                '',    '', now(), now(),     ?,
                '',    '',    '',     0,     0,
                '',     0,     ?,    '',    '',
                ?,    '',    '',    '',     ?,
                '',    '',    '',    '',    '',
                '',    '',    '',    '',    '',
                '',     0,    '',     0,     0,
                0,     0,    '',    '',     0,
                0,    '',     0,     0,    '',
                '',    '',    '',    '',    '',
                '',    '',    '',    '',    '',
                '',    '',    '',    '',    '',
                '',     ?,    '',     ?,     ?,
                '',    '',    '',    '',    '',
                FATAL: `thread_run' function failed: ./mysqltt_common.lua:661: SQL error, errno = 1114, state = 'HY000': The table 'sbtest' is full

    MySQL log：

    2022-02-07T06:16:26.536579Z 383 [Warning] [MY-012128] [InnoDB] posix_fallocate(): Failed to preallocate data for file ./sbtest/sbtest.ibd, desired size 114688 Operating system error number 38 - Function not implemented. Check that the disk is not full or a disk quota exceeded. Make sure the file system supports this function. Refer to your operating system documentation for operating system error code information.
2022-02-07T06:16:26.548445Z 383 [Warning] [MY-012579] [InnoDB] fallocate(38, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, 0, 16384) returned errno: 38
2022-02-07T06:16:26.571380Z 383 [ERROR] [MY-012144] [InnoDB] posix_fallocate(): Failed to preallocate data for file ./sbtest/sbtest.ibd, desired size 16384 bytes. Operating system error number 38. Check that the disk is not full or a disk quota exceeded. Make sure the file system supports this function. Refer to your operating system documentation for operating system error code information.
......
2022-02-07T06:16:27.391214Z 436 [ERROR] [MY-012144] [InnoDB] posix_fallocate(): Failed to preallocate data for file ./sbtest/sbtest.ibd, desired size 393216 bytes. Operating system error number 38. Check that the disk is not full or a disk quota exceeded. Make sure the file system supports this function. Refer to your operating system documentation for operating system error code information.
2022-02-07T06:16:27.435303Z 436 [ERROR] [MY-012144] [InnoDB] posix_fallocate(): Failed to preallocate data for file ./sbtest/sbtest.ibd, desired size 1048576 bytes. Operating system error number 38. Check that the disk is not full or a disk quota exceeded. Make sure the file system supports this function. Refer to your operating system documentation for operating system error code information.
......
2022-02-07T06:16:52.844651Z 424 [ERROR] [MY-012144] [InnoDB] posix_fallocate(): Failed to preallocate data for file ./sbtest/sbtest.ibd, desired size 4194304 bytes. Operating system error number 38. Check that the disk is not full or a disk quota exceeded. Make sure the file system supports this function. Refer to your operating system documentation for operating system error code information
.......
2022-02-07T06:17:12.468205Z 437 [ERROR] [MY-011904] [InnoDB] Space id and page number stored in the page read in are [page id: space=4294967279, page number=605], should be [page id: space=4294967279, page number=603]
2022-02-07T06:17:12.469264Z 437 [ERROR] [MY-013183] [InnoDB] Assertion failure: trx0undo.cc:890:mach_read_from_2(undo_page + TRX_UNDO_PAGE_HDR + TRX_UNDO_PAGE_TYPE) == TRX_UNDO_INSERT thread 2000930368
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
06:17:12 UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
Thread pointer: 0x194049aa0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7743bcf0 thread_stack 0x100000
/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x41) [0x9b42a011]
/usr/sbin/mysqld(handle_fatal_signal+0x31b) [0x9a2cdb7b]
/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libc.so.6(+0x41bb0) [0x9f472bb0]
/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libc.so.6(+0x8d573) [0x9f4be573]
/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libc.so.6(raise+0x16) [0x9f472b06]
/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libc.so.6(abort+0xc6) [0x9f45d7a6]
/usr/sbin/mysqld(+0xeb0606) [0x9a003606]
/usr/sbin/mysqld(+0x25cbb28) [0x9b71eb28]
/usr/sbin/mysqld(trx_undo_assign_undo(trx_t*, trx_undo_ptr_t*, unsigned long)+0x8e2) [0x9b721332]
/usr/sbin/mysqld(trx_undo_report_row_operation(unsigned long, unsigned long, que_thr_t*, dict_index_t*, dtuple_t const*, upd_t const*, unsigned long, unsigned char const*, unsigned long const*, unsigned long*)+0x890) [0x9b6f5800]
/usr/sbin/mysqld(btr_cur_optimistic_insert(unsigned long, btr_cur_t*, unsigned long**, mem_block_info_t**, dtuple_t*, unsigned char**, big_rec_t**, que_thr_t*, mtr_t*)+0x996) [0x9b76a9b6]
/usr/sbin/mysqld(row_ins_clust_index_entry_low(unsigned int, unsigned long, dict_index_t*, unsigned long, dtuple_t*, que_thr_t*, bool)+0x465) [0x9b64d675]
/usr/sbin/mysqld(row_ins_clust_index_entry(dict_index_t*, dtuple_t*, que_thr_t*, bool)+0xc1) [0x9b651b51]
/usr/sbin/mysqld(row_ins_step(que_thr_t*)+0x56a) [0x9b652cfa]
/usr/sbin/mysqld(+0x2505a2d) [0x9b658a2d]
/usr/sbin/mysqld(ha_innobase::write_row(unsigned char*)+0x299) [0x9b529399]
/usr/sbin/mysqld(handler::ha_write_row(unsigned char*)+0x1a8) [0x9a3e80f8]
/usr/sbin/mysqld(write_record(THD*, TABLE*, COPY_INFO*, COPY_INFO*)+0x66d) [0x9a65da7d]
/usr/sbin/mysqld(Sql_cmd_insert_values::execute_inner(THD*)+0x7f9) [0x9a65eca9]
/usr/sbin/mysqld(Sql_cmd_dml::execute(THD*)+0x186) [0x9a1e2456]
/usr/sbin/mysqld(mysql_execute_command(THD*, bool)+0x9f0) [0x9a182730]
/usr/sbin/mysqld(Prepared_statement::execute(String*, bool)+0x8b0) [0x9a1b1e00]
/usr/sbin/mysqld(Prepared_statement::execute_loop(String*, bool)+0x130) [0x9a1b6510]
/usr/sbin/mysqld(mysqld_stmt_execute(THD*, Prepared_statement*, bool, unsigned long, PS_PARAM*)+0x1b1) [0x9a1b6b21]
/usr/sbin/mysqld(dispatch_command(THD*, COM_DATA const*, enum_server_command)+0x251b) [0x9a189c5b]
/usr/sbin/mysqld(do_command(THD*)+0x1a4) [0x9a18a5e4]
/usr/sbin/mysqld(+0x116ba68) [0x9a2bea68]
/usr/sbin/mysqld(+0x2885bc9) [0x9b9d8bc9]
/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libc.so.6(+0x8b977) [0x9f4bc977]
/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/runtime/glibc/libc.so.6(clone+0x51) [0x9f53e3a1]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (194010270): is an invalid pointer
Connection ID (thread ID): 437
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
dimakuv commented 2 years ago
  1. What is your Intel CPU? Is it a Skylake, Icelake client (ICL), Icelake server (ICX), or something else?

Please note that prior to Icelake server, Intel SGX had only ~100MB of physical enclave cache size (EPC size). So if you have an older/weaker CPU, then SGX enclaves will exhibit very poor performance on memory-intensive workloads like MySQL.

  1. What is the output of is-sgx-available (this is a tool of Gramine that show SGX-relevant info)?

  2. How does you manifest file for MySQL look like? The error that you observe (The table 'sbtest' is full) looks like there is not enough enclave and/or stack memory allocated. Please try to increase enclave memory via sgx.enclave_size and stack memory via sys.stack.size.

dimakuv commented 2 years ago
  1. What is your Gramine version? Did you clone and build from GitHub? What is the commit hash that you used?
JorelHan commented 2 years ago

Thank you for your reply! We set sgx.enclave_size = 8G, sgx.thread.num=1024. When set sys.stack.size = 12M, worker threads failed to initialize within 30 seconds. I don't know how to look at Gramine version. view the version of Gramine if have the command, pls tell me. Thanks!

Below is the information I collected.

is-sgx-available output:

SGX supported by CPU: true
SGX1 (ECREATE, EENTER, ...): true
SGX2 (EAUG, EACCEPT, EMODPR, ...): true
Flexible Launch Control (IA32_SGXPUBKEYHASH{0..3} MSRs): true
SGX extensions for virtualizers (EINCVIRTCHILD, EDECVIRTCHILD, ESETCONTEXT): true
Extensions for concurrent memory management (ETRACKC, ELDBC, ELDUC, ERDINFO): true
CET enclave attributes support (See Table 37-5 in the SDM): false
Key separation and sharing (KSS) support (CONFIGID, CONFIGSVN, ISVEXTPRODID, ISVFAMILYID report fields): true
Max enclave size (32-bit): 0x80000000
Max enclave size (64-bit): 0x100000000000000
EPC size: 0x1fcf800000
SGX driver loaded: true
AESMD installed: false
SGX PSW/libsgx installed: false

entrypoint --version

GNU bash, version 5.0.17(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Docker images information is as follows:

[
    {
        "Id": "sha256:6158c778e53835b38c3dca7a21d29c9914d753ea746fd600ee3d610fcb970f68",
        "RepoTags": [
            "gar-registry.caas.intel.com/cpio/gsc-ubuntu18.04-mysql:latest"
        ],

        "Created": "2021-12-14T09:34:59.392410222Z",
        "Container": "e28a742661a847ba8b3f0f6550cdb2b8cb07f14f1761ba053b5f759d6a171a76",
        "ContainerConfig": {
            "Hostname": "e28a742661a8",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "3306/tcp": {},
                "33060/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/gramine/meson_build_output/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "PYTHONPATH=:/gramine/meson_build_output/lib/python3.8/site-packages",
                "GSC=2"
            ],
            "Cmd": [
                "/bin/sh",
                "-c",
                "#(nop) ",
                "ENTRYPOINT [\"./apploader.sh\"]"
            ],
            "Image": "sha256:4c0df41d9e706f1049acf8989dc39e4f9d8579cb339857cc18991199c4538099",
            "Volumes": {
                "/var/lib/mysql": {}
            },
            "WorkingDir": "",
            "Entrypoint": [
                "./apploader.sh"
            ],
            "OnBuild": null,
            "Labels": {}
        },
        "DockerVersion": "20.10.8",
        "Author": "",
        "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "3306/tcp": {},
                "33060/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/gramine/meson_build_output/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "PYTHONPATH=:/gramine/meson_build_output/lib/python3.8/site-packages",
                "GSC=2"
            ],
            "Cmd": null,
            "Image": "sha256:4c0df41d9e706f1049acf8989dc39e4f9d8579cb339857cc18991199c4538099",
            "Volumes": {
                "/var/lib/mysql": {}
            },
            "WorkingDir": "",
            "Entrypoint": [
                "./apploader.sh"
            ],
            "OnBuild": null,
            "Labels": null
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 1084894048,
        "VirtualSize": 1084894048,

        "Metadata": {
            "LastTagTime": "2021-12-30T15:04:25.608575254+08:00"
        }
    }
]
dimakuv commented 2 years ago

I don't know how to look at Gramine version. view the version of Gramine if have the command, pls tell me. Thanks!

It looks like you're using GSC to build graminized Docker images. What are the contents of config.yaml in your GSC?

dimakuv commented 2 years ago

Since you're using GSC, please add -d to the gsc build command (see https://gramine.readthedocs.io/projects/gsc/en/latest/#cmdoption-gsc-build-d).

This will enable the debug logs, which will give you more information to analyze.

JorelHan commented 2 years ago

This mirror image is not made by me. What information I need to provide you to better help you judge the problem.

dimakuv commented 2 years ago

You need to ask a person who generated this GSC Docker image to re-generate it with -d. And you need to analyze the resulting Gramine logs to figure out the root cause.

aneessahib commented 2 years ago

@dimakuv assign this one to me pls

aneessahib commented 2 years ago

i was able to build and run mysql with GSC, however i ran into an issue of running scripts from Gramine thats not supported today (i'm working on getting this added now). Below is procedure to get to a state where I am at now.

create a base image using the below docker file. 

From mysql:latest

ENTRYPOINT ["/bin/bash", "/usr/local/bin/docker-entrypoint.sh"]

and then create a GSC build (refer to the GSC documentation). After creation of a successful GSC image, run as below

docker run --device=/dev/sgx_enclave <gsc-build-name> "mysqld"
aneessahib commented 2 years ago

submitted https://github.com/gramineproject/gramine/pull/459 to add support for executing scripts. Now the mysql server starts, but exits after the below output. Looking into this further now

2022-03-15 06:49:17+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.28-1debian10 started. 2022-03-15 06:49:24+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'

qqing-li commented 2 years ago

Hi anees and dimakuv, I tried to run mysql in gramine 1.1 and also met similar errors. I didn't use gsc or a docker image. Here are the steps to reproduce: OS: ubuntu20.04 Kernel:5.11

  1. Install mysql-server-8.0 on baremetal: apt-get install mysql-server comment log in /etc/mysql/mysql.conf.d/mysqld.cnf to see the logs on console: #log_error = /var/log/mysql/error.log
  2. Stop mysql service, we need to manually run mysql with mysqld: systemctl stop mysql.service; mkdir /var/run/mysqld && chown -R mysql:mysql /var/run/mysqld
  3. Prepare new datadir: mkdir /tmp/mysql && chown -R mysql:mysql /tmp/mysql add the following 2 lines to /etc/apparmor.d/usr.sbin.mysqld: /tmp/mysql r, /tmp/mysql/** rwk, and restart apparmor: service apparmor restart
  4. Initialize mysql: mysqld --initialize-insecure --user=mysql --datadir=/tmp/mysql rm /tmp/mysql/undo*
  5. Run mysql in gramine: Here are the Makefile and manifest.template: mysql.zip cd gramine/CI-Examples/mysql make gramine-direct mysql --datadir /tmp/mysql Then mysql would output: 
    2022-03-24T16:40:46.284335Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 8.0.28-0ubuntu0.20.04.3) initializing of server in progress as process 207406
2022-03-24T16:40:46.296736Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-03-24T16:40:46.832322Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2022-03-24T16:40:48.288941Z 6 [Warning] [MY-010453] [Server] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.
2022-03-24T16:40:51.268092Z 0 [Warning] [MY-010939] [Server] Failed to get absolute path of program executable mysql
2022-03-24T16:41:57.832530Z 0 [System] [MY-010116] [Server] mysql (mysqld 8.0.28-0ubuntu0.20.04.3) starting as process 1
2022-03-24T16:41:57.840972Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-03-24T16:41:57.854539Z 1 [ERROR] [MY-012585] [InnoDB] Linux Native AIO interface is not supported on this platform. Please check your OS documentation and install appropriate binary of InnoDB.
2022-03-24T16:41:57.854626Z 1 [Warning] [MY-012654] [InnoDB] Linux Native AIO disabled.
2022-03-24T16:41:58.158630Z 1 [Warning] [MY-012128] [InnoDB] posix_fallocate(): Failed to preallocate data for file ./#innodb_temp/temp_1.ibt, desired size 81920 Operating system error number 38 - Function not implemented. Check that the disk is not full or a disk quota exceeded. Make sure the file system supports this function. Refer to your operating system documentation for operating system error code information.
2022-03-24T16:41:58.160318Z 1 [Warning] [MY-012579] [InnoDB] fallocate(14, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, 0, 16384) returned errno: 38
2022-03-24T16:41:58.161658Z 1 [Warning] [MY-012128] [InnoDB] posix_fallocate(): Failed to preallocate data for file ./#innodb_temp/temp_2.ibt, desired size 81920 Operating system error number 38 - Function not implemented. Check that the disk is not full or a disk quota exceeded. Make sure the file system supports this function. Refer to your operating system documentation for operating system error code information.
...
2022-03-24T16:41:58.481722Z 2 [ERROR] [MY-012144] [InnoDB] posix_fallocate(): Failed to preallocate data for file ./#innodb_temp/temp_10.ibt, desired size 16384 bytes. Operating system error number 38. Check that the disk is not full or a disk quota exceeded. Make sure the file system supports this function. Refer to your operating system documentation for operating system error code information.
...
2022-03-24T16:41:58.639969Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2022-03-24T16:41:58.640069Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2022-03-24T16:41:58.640953Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/tmp' in the path is accessible to all OS users. Consider choosing a different directory.
2022-03-24T16:41:58.673889Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '127.0.0.1' port: 33060, socket: /var/run/mysqld/mysqlx.sock
2022-03-24T16:41:58.674405Z 0 [System] [MY-010931] [Server] mysql: ready for connections. Version: '8.0.28-0ubuntu0.20.04.3'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu).
  6. Now create sbtest database and run sysbench: mysqladmin -h 127.0.0.1 -P 3306 create sbtest; apt-get install sysbench sysbench --db-driver=mysql --mysql-host=127.0.0.1 --mysql-port=3306 --mysql-user=root --mysql-db=sbtest /usr/share/sysbench/oltp_common.lua --tables=10 --table_size=100000 prepare sysbench --db-driver=mysql --mysql-host=127.0.0.1 --mysql-port=3306 --mysql-user=root --mysql-db=sbtest --threads=4 --time=20 --report-interval=5 /usr/share/sysbench/oltp_read_write.lua --tables=10 --table_size=100000 run Then mysql would output errors and abort: 
    
...
2022-03-24T16:12:54.502268Z 9 [ERROR] [MY-012144] [InnoDB] posix_fallocate(): Failed to preallocate data for file ./sbtest/sbtest7.ibd, desired size 16384 bytes. Operating system error number 38. Check that the disk is not full or a disk quota exceeded. Make sure the file system supports this function. Refer to your operating system documentation for operating system error code information.
2022-03-24T16:12:54.504173Z 9 [ERROR] [MY-012144] [InnoDB] posix_fallocate(): Failed to preallocate data for file ./sbtest/sbtest7.ibd, desired size 16384 bytes. Operating system error number 38. Check that the disk is not full or a disk quota exceeded. Make sure the file system supports this function. Refer to your operating system documentation for operating system error code information.
2022-03-24T16:12:54.506305Z 9 [ERROR] [MY-012144] [InnoDB] posix_fallocate(): Failed to preallocate data for file ./sbtest/sbtest7.ibd, desired size 16384 bytes. Operating system error number 38. Check that the disk is not full or a disk quota exceeded. Make sure the file system supports this function. Refer to your operating system documentation for operating system error code information.
2022-03-24T16:12:54.508279Z 9 [ERROR] [MY-012144] [InnoDB] posix_fallocate(): Failed to preallocate data for file ./sbtest/sbtest7.ibd, desired size 16384 bytes. Operating system error number 38. Check that the disk is not full or a disk quota exceeded. Make sure the file system supports this function. Refer to your operating system documentation for operating system error code information.
mysql(my_print_stacktrace(unsigned char const*, unsigned long)+0x41) [0x674965a56e91]
mysql(print_fatal_signal(int)+0x2fb) [0x6749648e332b]
mysql(handle_fatal_signal+0xa5) [0x6749648e33e5]
/lib/libc.so.6(+0x41b40) [0x67496292db40]
mysql(+0x2658040) [0x674965e08040]
mysql(+0x265eefc) [0x674965e0eefc]
mysql(dict_stats_update(dict_table_t*, dict_stats_upd_option_t)+0xf07) [0x674965e11bd7]
mysql(dict_stats_thread()+0xab2) [0x674965e13b32]
mysql(std::thread::_State_impl<std::thread::_Invoker<std::tuple<Detached_thread, void (*)()> > >::_M_run()+0xbc) [0x674965b5e29c]
/usr//lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6de4) [0x674962cb4de4]
/lib/libc.so.6(+0x8b907) [0x674962977907]
/lib/libc.so.6(clone+0x51) [0x6749629f92e1]

Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0): is an invalid pointer Connection ID (thread ID): 0 Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash.

aneessahib commented 2 years ago

@qqing-li - could you share the manifest file that you used

qqing-li commented 2 years ago

@qqing-li - could you share the manifest file that you used

Already attached a zip file in step 5. Can you download it?

aneessahib commented 2 years ago

i'm able to get it working further ahead

intel@intel-M50CYP2SB2U:~/aneessahib/gramine/CI-Examples/bash$ gramine-direct bash -c "/usr/sbin/mysqld --datadir /tmp/mysql1" 2022-03-31T09:04:19.454128Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details). 2022-03-31T09:04:19.455677Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.37-0ubuntu0.18.04.1) starting as process 1 ... 2022-03-31T09:04:19.455751Z 0 [ERROR] Can't find error-message file '/usr/share/mysql/errmsg.sys'. Check error-message file location and 'lc-messages-dir' configuration directive. 2022-03-31T09:04:19.459442Z 0 [Note] InnoDB: PUNCH HOLE support available 2022-03-31T09:04:19.459470Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2022-03-31T09:04:19.459474Z 0 [Note] InnoDB: Uses event mutexes 2022-03-31T09:04:19.459479Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier 2022-03-31T09:04:19.459482Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 2022-03-31T09:04:19.459487Z 0 [Note] InnoDB: Using Linux native AIO 2022-03-31T09:04:19.459671Z 0 [Note] InnoDB: Number of pools: 1 2022-03-31T09:04:19.459747Z 0 [Note] InnoDB: Using CPU crc32 instructions Emulating a raw system/supervisor call. This degrades performance, consider patching your application to use Gramine syscall API. 2022-03-31T09:04:19.460140Z 0 [ERROR] InnoDB: Linux Native AIO interface is not supported on this platform. Please check your OS documentation and install appropriate binary of InnoDB. 2022-03-31T09:04:19.460148Z 0 [Note] InnoDB: You can disable Linux Native AIO by setting innodb_use_native_aio = 0 in my.cnf 2022-03-31T09:04:19.460152Z 0 [Warning] InnoDB: Linux Native AIO disabled. 2022-03-31T09:04:19.460861Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M 2022-03-31T09:04:19.466472Z 0 [Note] InnoDB: Completed initialization of buffer pool 2022-03-31T09:04:19.471938Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2022-03-31T09:04:19.483653Z 0 [Note] InnoDB: Highest supported file format is Barracuda. 2022-03-31T09:04:19.492634Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2022-03-31T09:04:19.492684Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2022-03-31T09:04:19.550791Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2022-03-31T09:04:19.551614Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active. 2022-03-31T09:04:19.551628Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active. 2022-03-31T09:04:19.555056Z 0 [Note] InnoDB: 5.7.37 started; log sequence number 2750229 2022-03-31T09:04:19.555614Z 0 [Note] InnoDB: Loading buffer pool(s) from /tmp/mysql1/ib_buffer_pool 2022-03-31T09:04:19.557091Z 0 [Note] Plugin 'FEDERATED' is disabled. 2022-03-31T09:04:19.560011Z 0 [Note] InnoDB: Buffer pool(s) load completed at 220331 14:34:19 2022-03-31T09:04:19.566594Z 0 [Note] Found ca.pem, server-cert.pem and server-key.pem in data directory. Trying to enable SSL support using them. 2022-03-31T09:04:19.566624Z 0 [Note] Skipping generation of SSL certificates as certificate files are present in data directory. 2022-03-31T09:04:19.566636Z 0 [Warning] 2022-03-31T09:04:19.566645Z 0 [Warning] 2022-03-31T09:04:19.567525Z 0 [Warning] CA certificate ca.pem is self signed. 2022-03-31T09:04:19.567581Z 0 [Note] Skipping generation of RSA key pair as key files are present in data directory. 2022-03-31T09:04:19.567720Z 0 [Note] Server hostname (bind-address): '127.0.0.1'; port: 3306 2022-03-31T09:04:19.567748Z 0 [Note] - '127.0.0.1' resolves to '127.0.0.1'; 2022-03-31T09:04:19.567767Z 0 [Note] Server socket created on IP: '127.0.0.1'. 2022-03-31T09:04:19.585401Z 0 [Note] Event Scheduler: Loaded 0 events 2022-03-31T09:04:19.586574Z 0 [Note]

aneessahib commented 2 years ago

@qqing-li I will try with sysbench. @HtSmy - for the issue that you are facing with mysql containers, the issue is that the docker-entrypoint runs mysqld switching to another user. Gramine does not support switching users at this point, but I'm trying to see if any workaround is possible with GSC.

qqing-li commented 2 years ago

@aneessahib Thanks anees, I've tried the latest gramine instead of 1.1, now mysql starts successfully, and won't crash when using sysbench, but still continue outputting error logs like: 2022-04-01T10:47:05.958326Z 7 [ERROR] InnoDB: posix_fallocate(): Failed to preallocate data for file ./sbtest/sbtest3.ibd, desired size 4194304 bytes. Operating system error number 38. Check that the disk is not full or a disk quota exceeded. Make sure the file system supports this function. Some operating system error numbers are described at http://dev.mysql.com/doc/refman/5.7/en/operating-system-error-codes.html I'm not sure if it matters.

dimakuv commented 2 years ago

@qqing-li Regarding your last error InnoDB: posix_fallocate(). This function calls the underlying fallocate() system call, which is not implemented in Gramine: https://github.com/gramineproject/gramine/blob/dbb843a515be5869f30c87b6b3035c7e46740a65/LibOS/shim/src/arch/x86_64/shim_table.c#L303

Therefore MySQL complains about this (...Make sure the file system supports this function). This may be a problem for your MySQL workload. Does MySQL continue running? Does MySQL do what you're expecting it to do? Then you can ignore this error (maybe MySQL has a fall-back in this case, so it just continues running).

@pwmarcz Any idea how hard would it be to implement fallocate()? To me, it feels like we can always return success, because in our threat model we can't control the underlying disk space anyway (the attacker can always pretend to have more/less disk space at any moment in time).

mkow commented 2 years ago

Please try with https://github.com/gramineproject/gramine/pull/522.

boryspoplawski commented 2 years ago

@HtSmy What's the state of this? Why did you reopen?