Gramine-Direct Fuzzing: Failed to send IPC msg to 1: Broken pipe

anjalirai-intel commented 1 year ago

Description of the problem

Syzkaller has been modified in order to run with Gramine (https://github.com/JaewonHur/syzkaller). We were able to run it with Gramine v1.5 for Gramine-Direct and below crash is reported by the tool.

(libos_ipc.c:234:ipc_send_message_to_conn) [P2:T2:syz-executor] error: Failed to send IPC msg to 1: Broken pipe (EPIPE) (libos_ipc_pid.c:336:ipc_release_id_range) [P2:T2:syz-executor] debug: ipc_send_message: Broken pipe (EPIPE) (libos_pid.c:177:release_id) [P2:T2:syz-executor] warning: IPC pid release failed

These issues are very rarely reproducible but it is reported by syzcaller fuzzer several times

Steps to reproduce

Extract the zip file and cd to the drive
Build the Gramine(v1.5) in DEBUG mode
make DEBUG=1 CRASH=1c475b4a78976243
gramine-direct crash crash-1c475b4a78976243.zip

Expected results

No response

Actual results

(libos_parser.c:1628:buf_write_all) [P2:T2:syz-executor] trace: ---- return from futex(...) = 0x0
(libos_parser.c:1628:buf_write_all) [P2:T2:syz-executor] trace: ---- write(248, 0x6a9546396550, 0x2c) ...
(libos_parser.c:1628:buf_write_all) [P2:T2:syz-executor] trace: ---- return from write(...) = 0x2c
(libos_exit.c:212:libos_syscall_exit_group) [P2:T2:syz-executor] debug: ---- exit_group (returning 1)
(libos_ipc.c:222:ipc_send_message_to_conn) [P2:T2:syz-executor] debug: Sending ipc message to 1
(libos_ipc_worker.c:193:receive_ipc_messages) [P1:libos] debug: IPC worker: received IPC message from 2: code=17 size=21 seq=2
(libos_fs_lock.c:785:file_lock_clear_pid) [P1:libos] debug: clearing file (POSIX) locks for pid 2
(libos_ipc.c:222:ipc_send_message_to_conn) [P1:libos] debug: Sending ipc message to 2
(libos_ipc.c:257:wait_for_response) [P2:T2:syz-executor] debug: Waiting for a response to 2
(libos_ipc_worker.c:193:receive_ipc_messages) [P2:libos] debug: IPC worker: received IPC message from 1: code=0 size=21 seq=2
(libos_ipc.c:264:wait_for_response) [P2:T2:syz-executor] debug: Waiting finished: Success (PAL_ERROR_SUCCESS)
(libos_ipc.c:341:ipc_response_callback) [P2:libos] debug: Got an IPC response from 1, seq: 2
(libos_ipc.c:222:ipc_send_message_to_conn) [P2:T2:syz-executor] debug: Sending ipc message to 1
(libos_ipc_worker.c:193:receive_ipc_messages) [P1:libos] debug: IPC worker: received IPC message from 2: code=2 size=37 seq=0
(libos_ipc_child.c:59:ipc_cld_exit_callback) [P1:libos] debug: IPC callback from 2: IPC_MSG_CHILDEXIT(1, 2, 1, 0)
(libos_ipc_child.c:63:ipc_cld_exit_callback) [P1:libos] debug: Child process (pid: 2) died
(libos_parser.c:1628:buf_write_all) [P1:T1:syz-executor] trace: ---- return from wait4(...) = -512
(libos_parser.c:1628:buf_write_all) [P1:T1:syz-executor] trace: ---- wait4(-1, 0x6a95463967a4, |0x40000000, 0) ...
(libos_parser.c:1628:buf_write_all) [P1:T1:syz-executor] trace: ---- return from wait4(...) = 0x2
(libos_parser.c:1628:buf_write_all) [P1:T1:syz-executor] trace: ---- write(248, 0x6a9546396824, 0xc) ...
(libos_parser.c:1628:buf_write_all) [P1:T1:syz-executor] trace: ---- return from write(...) = 0xc
(libos_exit.c:212:libos_syscall_exit_group) [P1:T1:syz-executor] debug: ---- exit_group (returning 1)
(libos_sync_client.c:331:shutdown_sync_client) [P2:T2:syz-executor] debug: sync client shutdown: closing handles
(libos_sync_client.c:346:shutdown_sync_client) [P2:T2:syz-executor] debug: sync client shutdown: waiting for confirmation
(libos_sync_client.c:359:shutdown_sync_client) [P2:T2:syz-executor] debug: sync client shutdown: finished
(libos_ipc_pid.c:333:ipc_release_id_range) [P2:T2:syz-executor] debug: sending a request: [2..2]
(libos_ipc.c:222:ipc_send_message_to_conn) [P2:T2:syz-executor] debug: Sending ipc message to 1
(libos_ipc.c:234:ipc_send_message_to_conn) [P2:T2:syz-executor] error: Failed to send IPC msg to 1: Broken pipe (EPIPE)
(libos_ipc_pid.c:336:ipc_release_id_range) [P2:T2:syz-executor] debug: ipc_send_message: Broken pipe (EPIPE)
(libos_pid.c:177:release_id) [P2:T2:syz-executor] warning: IPC pid release failed
.....

Gramine commit hash

0b552696d4f402c291c8c2cce525e7117d06d397

kailun-qin commented 1 year ago

Hi @anjalirai-intel!

I cannot reproduce this issue w/ both Gramine master: b6fc88de1b0ccfa6602c7abbdc4936664f90d980 and v1.5: 0b552696d4f402c291c8c2cce525e7117d06d397 using the provided program on two of my local setups. Would you pls help double check? Thanks!

anjalirai-intel commented 1 year ago

@kailun-qin IPC issues are very rarely reproducible, but syzcaller reports this crash several times

kailun-qin commented 1 year ago

I see, thanks @anjalirai-intel!

IPC issues are very rarely reproducible, but syzcaller reports this crash several times

I guess it's also very rarely reproducible for https://github.com/gramineproject/gramine/issues/1520?

If so, would you pls update both issue descriptions w/ this additional info on reproducability?

anjalirai-intel commented 1 year ago

I see, thanks @anjalirai-intel!

IPC issues are very rarely reproducible, but syzcaller reports this crash several times

I guess it's also very rarely reproducible for #1520?

If so, would you pls update both issue descriptions w/ this additional info on reproducability?

Done

mkow commented 1 year ago

Isn't this just https://github.com/gramineproject/gramine/issues/21 ? I guess the parent died for some reason.

dimakuv commented 1 week ago

Closing this as it is a duplicate of #21

gramineproject / gramine