search

skupperproject / skupper-router

An application-layer router for Skupper networks
https://skupper.io
Apache License 2.0
14 stars 18 forks source link

AddressSanitizer: use-after-poison in qd_message_extend: `qd_message_content_t *content = MSG_CONTENT(msg);` #107

Open jiridanek opened 2 years ago

jiridanek commented 2 years ago

Configure the router as in system_tests_http2.py::Http2TestOneStandaloneRouter::test_post_upload_large_image_jpg, then repeatedly run

curl --http2-prior-knowledge http://127.0.0.1:21581/upload -X POST -H 'Content-Type: multipart/form-data' -F data=@/home/jdanek/repos/skupper-router/tests/images/test.jpg
=================================================================
==909120==ERROR: AddressSanitizer: use-after-poison on address 0x614000047ef0 at pc 0x000000530e58 bp 0x7f360fb20900 sp 0x7f360fb208f8
READ of size 8 at 0x614000047ef0 thread T3
    #0 0x530e57 in qd_message_extend /home/jdanek/repos/skupper-router/src/message.c:2399
    #1 0x53c318 in qd_message_stream_data_append /home/jdanek/repos/skupper-router/src/message.c:3028
    #2 0x68ab09 in on_data_chunk_recv_callback /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:560
    #3 0x7f3625a6c753 in nghttp2_session_mem_recv (/lib64/libnghttp2.so.14+0xe753)
    #4 0x6946cb in handle_incoming_http /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:2133
    #5 0x6b1852 in handle_connection_event /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:2542
    #6 0x662fd4 in handle_event_with_context /home/jdanek/repos/skupper-router/src/server.c:780
    #7 0x662fd4 in do_handle_raw_connection_event /home/jdanek/repos/skupper-router/src/server.c:786
    #8 0x662fd4 in handle /home/jdanek/repos/skupper-router/src/server.c:1063
    #9 0x66a6a5 in thread_run /home/jdanek/repos/skupper-router/src/server.c:1095
    #10 0x7f3624f6fb19 in start_thread (/lib64/libc.so.6+0x8db19)
    #11 0x7f3624ff464f in __GI___clone3 (/lib64/libc.so.6+0x11264f)

0x614000047ef0 is located 176 bytes inside of 384-byte region [0x614000047e40,0x614000047fc0)
allocated by thread T4 here:
    #0 0x7f3625f654fc in __interceptor_posix_memalign (/lib64/libasan.so.6+0xaf4fc)
    #1 0x4bb49d in qd_alloc /home/jdanek/repos/skupper-router/src/alloc_pool.c:391
    #2 0x522268 in new_qd_message_t /home/jdanek/repos/skupper-router/src/message.c:93
    #3 0x522268 in qd_message /home/jdanek/repos/skupper-router/src/message.c:1011
    #4 0x68329c in create_http2_stream_data /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:478
    #5 0x6ae964 in qdr_http_deliver /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:2032
    #6 0x611d26 in qdr_link_process_deliveries /home/jdanek/repos/skupper-router/src/router_core/transfer.c:194
    #7 0x58a621 in qdr_connection_process /home/jdanek/repos/skupper-router/src/router_core/connections.c:401
    #8 0x6affbe in handle_connection_event /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:2545
    #9 0x662fd4 in handle_event_with_context /home/jdanek/repos/skupper-router/src/server.c:780
    #10 0x662fd4 in do_handle_raw_connection_event /home/jdanek/repos/skupper-router/src/server.c:786
    #11 0x662fd4 in handle /home/jdanek/repos/skupper-router/src/server.c:1063
    #12 0x66a6a5 in thread_run /home/jdanek/repos/skupper-router/src/server.c:1095
    #13 0x7f3624f6fb19 in start_thread (/lib64/libc.so.6+0x8db19)

Thread T3 created by T0 here:
    #0 0x7f3625f0c866 in pthread_create (/lib64/libasan.so.6+0x56866)
    #1 0x55f155 in sys_thread /home/jdanek/repos/skupper-router/src/posix/threading.c:181
    #2 0x66cc1c in qd_server_run /home/jdanek/repos/skupper-router/src/server.c:1489
    #3 0x6b633d in main_process /home/jdanek/repos/skupper-router/router/src/main.c:105
    #4 0x432332 in main /home/jdanek/repos/skupper-router/router/src/main.c:359
    #5 0x7f3624f0f55f in __libc_start_call_main (/lib64/libc.so.6+0x2d55f)

Thread T4 created by T0 here:
    #0 0x7f3625f0c866 in pthread_create (/lib64/libasan.so.6+0x56866)
    #1 0x55f155 in sys_thread /home/jdanek/repos/skupper-router/src/posix/threading.c:181
    #2 0x66cc1c in qd_server_run /home/jdanek/repos/skupper-router/src/server.c:1489
    #3 0x6b633d in main_process /home/jdanek/repos/skupper-router/router/src/main.c:105
    #4 0x432332 in main /home/jdanek/repos/skupper-router/router/src/main.c:359
    #5 0x7f3624f0f55f in __libc_start_call_main (/lib64/libc.so.6+0x2d55f)

SUMMARY: AddressSanitizer: use-after-poison /home/jdanek/repos/skupper-router/src/message.c:2399 in qd_message_extend
Shadow bytes around the buggy address:
  0x0c2880000f80: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
  0x0c2880000f90: 00 00 00 00 00 00 00 00 00 00 f7 f7 f7 f7 f7 f7
  0x0c2880000fa0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x0c2880000fb0: f7 f7 f7 f7 00 00 00 00 fa fa fa fa fa fa fa fa
  0x0c2880000fc0: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
=>0x0c2880000fd0: 00 00 00 00 00 00 00 00 00 00 f7 f7 f7 f7[f7]f7
  0x0c2880000fe0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x0c2880000ff0: f7 f7 f7 f7 00 00 00 00 fa fa fa fa fa fa fa fa
  0x0c2880001000: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2880001010: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2880001020: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==909120==ABORTING

From the same test, I've also seen the following. See last leak for addr2line decoded trace

addr2line --exe router/qdrouterd --pretty-print 0x<addr>
alloc.c: Items of type 'qd_http2_buffer_t' remain allocated at shutdown: 10
Leak: 2022-02-27 14:54:31.996089 +0100 type: qd_http2_buffer_t address: 0x62900007d290
/lib64/libasan.so.6(+0x44701) [0x7fb03d518701]
qdrouterd(qd_alloc+0x1fe) [0x4ba4ce]
qdrouterd() [0x697522]
qdrouterd() [0x698111]
/lib64/libnghttp2.so.14(+0xad1b) [0x7fb03d086d1b]
/lib64/libnghttp2.so.14(nghttp2_session_send+0x69) [0x7fb03d088e89]
qdrouterd() [0x6950ca]
qdrouterd() [0x6b036e]
qdrouterd() [0x662fd5]
qdrouterd() [0x66a6a6]

Leak: 2022-02-27 14:54:31.996285 +0100 type: qd_http2_buffer_t address: 0x629000082290
/lib64/libasan.so.6(+0x44701) [0x7fb03d518701]
qdrouterd(qd_alloc+0x1fe) [0x4ba4ce]
qdrouterd() [0x697522]
qdrouterd() [0x698111]
/lib64/libnghttp2.so.14(+0xad1b) [0x7fb03d086d1b]
/lib64/libnghttp2.so.14(nghttp2_session_send+0x69) [0x7fb03d088e89]
qdrouterd() [0x6950ca]
qdrouterd() [0x6b036e]
qdrouterd() [0x662fd5]
qdrouterd() [0x66a6a6]

Leak: 2022-02-27 14:54:31.996481 +0100 type: qd_http2_buffer_t address: 0x629000168290
/lib64/libasan.so.6(+0x44701) [0x7fb03d518701]
qdrouterd(qd_alloc+0x1fe) [0x4ba4ce]
qdrouterd() [0x697522]
qdrouterd() [0x698111]
/lib64/libnghttp2.so.14(+0xad1b) [0x7fb03d086d1b]
/lib64/libnghttp2.so.14(nghttp2_session_send+0x69) [0x7fb03d088e89]
qdrouterd() [0x6950ca]
qdrouterd() [0x6b036e]
qdrouterd() [0x662fd5]
qdrouterd() [0x66a6a6]

Leak: 2022-02-27 14:54:31.996670 +0100 type: qd_http2_buffer_t address: 0x6290000af290
/lib64/libasan.so.6(+0x44701) [0x7fb03d518701]
qdrouterd(qd_alloc+0x1fe) [0x4ba4ce]
qdrouterd() [0x697522]
qdrouterd() [0x698111]
/lib64/libnghttp2.so.14(+0xad1b) [0x7fb03d086d1b]
/lib64/libnghttp2.so.14(nghttp2_session_send+0x69) [0x7fb03d088e89]
qdrouterd() [0x6950ca]
qdrouterd() [0x6b036e]
qdrouterd() [0x662fd5]
qdrouterd() [0x66a6a6]

Leak: 2022-02-27 14:54:31.996820 +0100 type: qd_http2_buffer_t address: 0x6290000e1290
/lib64/libasan.so.6(+0x44701) [0x7fb03d518701]
qdrouterd(qd_alloc+0x1fe) [0x4ba4ce]
qdrouterd() [0x697522]
qdrouterd() [0x698111]
/lib64/libnghttp2.so.14(+0xad1b) [0x7fb03d086d1b]
/lib64/libnghttp2.so.14(nghttp2_session_send+0x69) [0x7fb03d088e89]
qdrouterd() [0x6950ca]
qdrouterd() [0x6b036e]
qdrouterd() [0x662fd5]
qdrouterd() [0x66a6a6]

Leak: 2022-02-27 14:54:31.996987 +0100 type: qd_http2_buffer_t address: 0x6290002e9290
/lib64/libasan.so.6(+0x44701) [0x7fb03d518701]
qdrouterd(qd_alloc+0x1fe) [0x4ba4ce]
qdrouterd() [0x697522]
qdrouterd() [0x698111]
/lib64/libnghttp2.so.14(+0xad1b) [0x7fb03d086d1b]
/lib64/libnghttp2.so.14(nghttp2_session_send+0x69) [0x7fb03d088e89]
qdrouterd() [0x6950ca]
qdrouterd() [0x6b036e]
qdrouterd() [0x662fd5]
qdrouterd() [0x66a6a6]

Leak: 2022-02-27 14:54:31.997152 +0100 type: qd_http2_buffer_t address: 0x629000325290
/lib64/libasan.so.6(+0x44701) [0x7fb03d518701]
qdrouterd(qd_alloc+0x1fe) [0x4ba4ce]
qdrouterd() [0x697522]
qdrouterd() [0x698111]
/lib64/libnghttp2.so.14(+0xad1b) [0x7fb03d086d1b]
/lib64/libnghttp2.so.14(nghttp2_session_send+0x69) [0x7fb03d088e89]
qdrouterd() [0x6950ca]
qdrouterd() [0x6b036e]
qdrouterd() [0x662fd5]
qdrouterd() [0x66a6a6]

Leak: 2022-02-27 14:54:31.997321 +0100 type: qd_http2_buffer_t address: 0x629000069290
/lib64/libasan.so.6(+0x44701) [0x7fb03d518701]
qdrouterd(qd_alloc+0x1fe) [0x4ba4ce]
qdrouterd() [0x697522]
qdrouterd() [0x698111]
/lib64/libnghttp2.so.14(+0xad1b) [0x7fb03d086d1b]
/lib64/libnghttp2.so.14(nghttp2_session_send+0x69) [0x7fb03d088e89]
qdrouterd() [0x6950ca]
qdrouterd() [0x6b036e]
qdrouterd() [0x662fd5]
qdrouterd() [0x66a6a6]

Leak: 2022-02-27 14:54:31.997487 +0100 type: qd_http2_buffer_t address: 0x6290001ef290
/lib64/libasan.so.6(+0x44701) [0x7fb03d518701]
qdrouterd(qd_alloc+0x1fe) [0x4ba4ce]
qdrouterd() [0x697522]
qdrouterd() [0x698111]
/lib64/libnghttp2.so.14(+0xad1b) [0x7fb03d086d1b]
/lib64/libnghttp2.so.14(nghttp2_session_send+0x69) [0x7fb03d088e89]
qdrouterd() [0x6950ca]
qdrouterd() [0x6b036e]
qdrouterd() [0x662fd5]
qdrouterd() [0x66a6a6]

Leak: 2022-02-27 14:54:31.997619 +0100 type: qd_http2_buffer_t address: 0x6290001e0290
/lib64/libasan.so.6(+0x44701) [0x7fb03d518701]
qdrouterd(qd_alloc+0x1fe) [0x4ba4ce] /home/jdanek/repos/skupper-router/src/alloc_pool.c:349
qdrouterd() [0x697522] /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:51
qdrouterd() [0x698111] /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:154
/lib64/libnghttp2.so.14(+0xad1b) [0x7fb03d086d1b]
/lib64/libnghttp2.so.14(nghttp2_session_send+0x69) [0x7fb03d088e89]
qdrouterd() [0x6950ca] /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:2173
qdrouterd() [0x6b036e] /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:2555
qdrouterd() [0x662fd5] /home/jdanek/repos/skupper-router/src/server.c:780
qdrouterd() [0x66a6a6] /home/jdanek/repos/skupper-router/src/server.c:1095
jiridanek commented 2 years ago 
qdrouterd: /home/jdanek/repos/skupper-router/src/router_core/delivery.c:440: qdr_delete_delivery_internal_CT: Assertion `sys_atomic_get(&delivery->ref_count) == 0' failed.
curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)

I also saw a leak from pn_record_def

==929325==ERROR: LeakSanitizer: detected memory leaks

Indirect leak of 48 byte(s) in 2 object(s) allocated from:
    #0 0x7fafd949ec98 in __interceptor_realloc (/lib64/libasan.so.6+0xaec98)
    #1 0x7fafd93ad809 in pn_record_def (/lib64/libqpid-proton-core.so.10+0x15809) 

-----------------------------------------------------
Suppressions used:
  count      bytes template
      1         56 ^IoAdapter_init$
      1         24 ^pn_condition$
      1       1536 ^pn_raw_connection$
      5        248 ^pn_object_new$
      1        128 ^pn_list$
    467     471233 /libpython3.*.so
-----------------------------------------------------
jiridanek commented 2 years ago 
/home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:192:60: runtime error: member access within null pointer of type 'struct qdr_http2_adaptor_t'
    #0 0x680b4b in qdr_http2_q2_unblocked_handler /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:192
    #1 0x534c6b in qd_message_free /home/jdanek/repos/skupper-router/src/message.c:1095
    #2 0x5f696e in qdr_core_free /home/jdanek/repos/skupper-router/src/router_core/router_core.c:263
    #3 0x6534bd in qd_router_free /home/jdanek/repos/skupper-router/src/router_node.c:2125
    #4 0x4f60a2 in qd_dispatch_free /home/jdanek/repos/skupper-router/src/dispatch.c:355
    #5 0x6b6354 in main_process /home/jdanek/repos/skupper-router/router/src/main.c:109
    #6 0x432332 in main /home/jdanek/repos/skupper-router/router/src/main.c:359
    #7 0x7f43d1a8155f in __libc_start_call_main (/lib64/libc.so.6+0x2d55f)
    #8 0x7f43d1a8160b in __libc_start_main_alias_1 (/lib64/libc.so.6+0x2d60b)
    #9 0x4325a4 in _start (/home/jdanek/repos/skupper-router/cmake-build-relwithdebinfo-asan/router/qdrouterd+0x4325a4)
jiridanek commented 2 years ago

Freeing stacktrace from patched router at https://github.com/skupperproject/skupper-router/pull/110

=================================================================
==1091403==ERROR: AddressSanitizer: heap-use-after-free on address 0x615000000888 at pc 0x0000006877b6 bp 0x7f5a603b9840 sp 0x7f5a603b9838
READ of size 8 at 0x615000000888 thread T3
    #0 0x6877b5 in read_data_callback /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:1178
    #1 0x7f5a76317b04 in nghttp2_session_mem_send_internal (/lib64/libnghttp2.so.14+0xbb04)
    #2 0x7f5a76318e88 in nghttp2_session_send (/lib64/libnghttp2.so.14+0xce88)
    #3 0x68e8a2 in qdr_http_delivery_update /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:1601
    #4 0x58a22d in qdr_connection_process /home/jdanek/repos/skupper-router/src/router_core/connections.c:390
    #5 0x6afdce in handle_connection_event /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:2545
    #6 0x662de4 in handle_event_with_context /home/jdanek/repos/skupper-router/src/server.c:780
    #7 0x662de4 in do_handle_raw_connection_event /home/jdanek/repos/skupper-router/src/server.c:786
    #8 0x662de4 in handle /home/jdanek/repos/skupper-router/src/server.c:1063
    #9 0x66a4b5 in thread_run /home/jdanek/repos/skupper-router/src/server.c:1095
    #10 0x7f5a7581db19 in start_thread (/lib64/libc.so.6+0x8db19)
    #11 0x7f5a758a264f in __GI___clone3 (/lib64/libc.so.6+0x11264f)

0x615000000888 is located 200 bytes inside of 448-byte region [0x6150000007c0,0x615000000980)
freed by thread T3 here:
    #0 0x7f5a76812627 in free (/lib64/libasan.so.6+0xae627)
    #1 0x4bf092 in qd_dealloc /home/jdanek/repos/skupper-router/src/alloc_pool.c:497
    #2 0x68f77d in free_http2_stream_data /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:344
    #3 0x68f77d in qdr_http_delivery_update /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:1631
    #4 0x58a22d in qdr_connection_process /home/jdanek/repos/skupper-router/src/router_core/connections.c:390
    #5 0x6afdce in handle_connection_event /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:2545
    #6 0x662de4 in handle_event_with_context /home/jdanek/repos/skupper-router/src/server.c:780
    #7 0x662de4 in do_handle_raw_connection_event /home/jdanek/repos/skupper-router/src/server.c:786
    #8 0x662de4 in handle /home/jdanek/repos/skupper-router/src/server.c:1063
    #9 0x66a4b5 in thread_run /home/jdanek/repos/skupper-router/src/server.c:1095
    #10 0x7f5a7581db19 in start_thread (/lib64/libc.so.6+0x8db19)

previously allocated by thread T3 here:
    #0 0x7f5a768134fc in __interceptor_posix_memalign (/lib64/libasan.so.6+0xaf4fc)
    #1 0x4bb49d in qd_alloc /home/jdanek/repos/skupper-router/src/alloc_pool.c:393
    #2 0x682ff0 in new_qdr_http2_stream_data_t /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:49
    #3 0x682ff0 in create_http2_stream_data /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:471
    #4 0x6ae774 in qdr_http_deliver /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:2032
    #5 0x611b36 in qdr_link_process_deliveries /home/jdanek/repos/skupper-router/src/router_core/transfer.c:194
    #6 0x58a431 in qdr_connection_process /home/jdanek/repos/skupper-router/src/router_core/connections.c:401
    #7 0x6afdce in handle_connection_event /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:2545
    #8 0x662de4 in handle_event_with_context /home/jdanek/repos/skupper-router/src/server.c:780
    #9 0x662de4 in do_handle_raw_connection_event /home/jdanek/repos/skupper-router/src/server.c:786
    #10 0x662de4 in handle /home/jdanek/repos/skupper-router/src/server.c:1063
    #11 0x66a4b5 in thread_run /home/jdanek/repos/skupper-router/src/server.c:1095
    #12 0x7f5a7581db19 in start_thread (/lib64/libc.so.6+0x8db19)

Thread T3 created by T0 here:
    #0 0x7f5a767ba866 in pthread_create (/lib64/libasan.so.6+0x56866)
    #1 0x55ef65 in sys_thread /home/jdanek/repos/skupper-router/src/posix/threading.c:181
    #2 0x66ca2c in qd_server_run /home/jdanek/repos/skupper-router/src/server.c:1489
    #3 0x6b614d in main_process /home/jdanek/repos/skupper-router/router/src/main.c:105
    #4 0x432332 in main /home/jdanek/repos/skupper-router/router/src/main.c:359
    #5 0x7f5a757bd55f in __libc_start_call_main (/lib64/libc.so.6+0x2d55f)

SUMMARY: AddressSanitizer: heap-use-after-free /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:1178 in read_data_callback
Shadow bytes around the buggy address:
  0x0c2a7fff80c0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c2a7fff80d0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c2a7fff80e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2a7fff80f0: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c2a7fff8100: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x0c2a7fff8110: fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c2a7fff8120: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c2a7fff8130: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2a7fff8140: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2a7fff8150: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2a7fff8160: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==1091403==ABORTING
jiridanek commented 2 years ago 
=================================================================
==1166308==ERROR: AddressSanitizer: heap-use-after-free on address 0x6160000d3d28 at pc 0x0000006a1e65 bp 0x7ff2d3550a50 sp 0x7ff2d3550a48
READ of size 4 at 0x6160000d3d28 thread T4
    #0 0x6a1e64 in on_frame_recv_callback /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:1078
    #1 0x7ff2ea7ce040 in session_process_data_frame (/lib64/libnghttp2.so.14+0x9040)
    #2 0x7ff2ea7d309e in nghttp2_session_mem_recv (/lib64/libnghttp2.so.14+0xe09e)
    #3 0x6944db in handle_incoming_http /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:2133
    #4 0x6b017d in handle_connection_event /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:2554
    #5 0x662de4 in handle_event_with_context /home/jdanek/repos/skupper-router/src/server.c:780
    #6 0x662de4 in do_handle_raw_connection_event /home/jdanek/repos/skupper-router/src/server.c:786
    #7 0x662de4 in handle /home/jdanek/repos/skupper-router/src/server.c:1063
    #8 0x66a4b5 in thread_run /home/jdanek/repos/skupper-router/src/server.c:1095
    #9 0x7ff2e9cd6b19 in start_thread (/lib64/libc.so.6+0x8db19)
    #10 0x7ff2e9d5b64f in __GI___clone3 (/lib64/libc.so.6+0x11264f)

0x6160000d3d28 is located 424 bytes inside of 512-byte region [0x6160000d3b80,0x6160000d3d80)
freed by thread T1 here:
    #0 0x7ff2eaccb627 in free (/lib64/libasan.so.6+0xae627)
    #1 0x4bf092 in qd_dealloc /home/jdanek/repos/skupper-router/src/alloc_pool.c:497
    #2 0x5fbc27 in router_core_thread /home/jdanek/repos/skupper-router/src/router_core/router_core_thread.c:236
    #3 0x7ff2e9cd6b19 in start_thread (/lib64/libc.so.6+0x8db19)

previously allocated by thread T4 here:
    #0 0x7ff2eaccc4fc in __interceptor_posix_memalign (/lib64/libasan.so.6+0xaf4fc)
    #1 0x4bb49d in qd_alloc /home/jdanek/repos/skupper-router/src/alloc_pool.c:393
    #2 0x60f6cc in qdr_link_deliver /home/jdanek/repos/skupper-router/src/router_core/transfer.c:63
    #3 0x69b3aa in compose_and_deliver /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:924
    #4 0x69ed70 in route_delivery /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:945
    #5 0x6a2eb0 in qdr_http_flow /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:1488
    #6 0x6a2eb0 in qdr_http_flow /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:1480
    #7 0x58b122 in qdr_connection_process /home/jdanek/repos/skupper-router/src/router_core/connections.c:409
    #8 0x6afdce in handle_connection_event /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:2545
    #9 0x662de4 in handle_event_with_context /home/jdanek/repos/skupper-router/src/server.c:780
    #10 0x662de4 in do_handle_raw_connection_event /home/jdanek/repos/skupper-router/src/server.c:786
    #11 0x662de4 in handle /home/jdanek/repos/skupper-router/src/server.c:1063
    #12 0x66a4b5 in thread_run /home/jdanek/repos/skupper-router/src/server.c:1095
    #13 0x7ff2e9cd6b19 in start_thread (/lib64/libc.so.6+0x8db19)

Thread T4 created by T0 here:
    #0 0x7ff2eac73866 in pthread_create (/lib64/libasan.so.6+0x56866)
    #1 0x55ef65 in sys_thread /home/jdanek/repos/skupper-router/src/posix/threading.c:181
    #2 0x66ca2c in qd_server_run /home/jdanek/repos/skupper-router/src/server.c:1489
    #3 0x6b614d in main_process /home/jdanek/repos/skupper-router/router/src/main.c:105
    #4 0x432332 in main /home/jdanek/repos/skupper-router/router/src/main.c:359
    #5 0x7ff2e9c7655f in __libc_start_call_main (/lib64/libc.so.6+0x2d55f)

Thread T1 created by T0 here:
    #0 0x7ff2eac73866 in pthread_create (/lib64/libasan.so.6+0x56866)
    #1 0x55ef65 in sys_thread /home/jdanek/repos/skupper-router/src/posix/threading.c:181
    #2 0x5e54d3 in qdr_core /home/jdanek/repos/skupper-router/src/router_core/router_core.c:122
    #3 0x652e22 in qd_router_setup_late /home/jdanek/repos/skupper-router/src/router_node.c:2088
    #4 0x7ff2e5a5cc03 in ffi_call_unix64 (/lib64/libffi.so.6+0x6c03)
    #5 0x7ffd56db61ff  ([stack]+0x1e1ff)

SUMMARY: AddressSanitizer: heap-use-after-free /home/jdanek/repos/skupper-router/src/adaptors/http2/http2_adaptor.c:1078 in on_frame_recv_callback
Shadow bytes around the buggy address:
  0x0c2c80012750: 00 00 fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2c80012760: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2c80012770: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c2c80012780: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c2c80012790: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x0c2c800127a0: fd fd fd fd fd[fd]fd fd fd fd fd fd fd fd fd fd
  0x0c2c800127b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2c800127c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2c800127d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c2c800127e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c2c800127f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==1166308==ABORTING
jiridanek commented 2 years ago

Possibly related to https://github.com/skupperproject/skupper-router/issues/109, another http2 issue opened at the same time.

jiridanek commented 6 months ago

I haven't seen this for a long time. Maybe just close as out-of-date?

jiridanek commented 4 weeks ago 
57: Router EdgeB output file:
57: >>>>
57: ==================
57: WARNING: ThreadSanitizer: data race (pid=6969)
57:   Read of size 8 at 0x724c00096f08 by thread T6:
57:     #0 handle_event_with_context /home/runner/work/skupper-router/skupper-router/skupper-router/src/server.c:107 (skrouterd+0x51b88c) (BuildId: ff494da932379fc07ddd8270f2e95dd00f09c918)
57:     #1 handle_raw_connection_event /home/runner/work/skupper-router/skupper-router/skupper-router/src/server.c:115 (skrouterd+0x51b88c)
57:     #2 handle_raw_connection_event /home/runner/work/skupper-router/skupper-router/skupper-router/src/server.c:111 (skrouterd+0x51b88c)
57:     #3 proactor_thread /home/runner/work/skupper-router/skupper-router/skupper-router/src/server.c:194 (skrouterd+0x51b9f8) (BuildId: ff494da932379fc07ddd8270f2e95dd00f09c918)
57:     #4 _thread_init /home/runner/work/skupper-router/skupper-router/skupper-router/src/posix/threading.c:207 (skrouterd+0x4bf2dd) (BuildId: ff494da932379fc07ddd8270f2e95dd00f09c918)
57: 
57:   Previous write of size 8 at 0x724c00096f08 by thread T2:
57:     #0 free_qdr_http2_connection /home/runner/work/skupper-router/skupper-router/skupper-router/src/adaptors/http2/http2_adaptor.c:446 (skrouterd+0x459e09) (BuildId: ff494da932379fc07ddd8270f2e95dd00f09c918)
57:     #1 qdr_del_http2_connection_CT /home/runner/work/skupper-router/skupper-router/skupper-router/src/adaptors/http2/http2_adaptor.c:2892 (skrouterd+0x45a0ee) (BuildId: ff494da932379fc07ddd8270f2e95dd00f09c918)
57:     #2 qdr_del_http2_connection_CT /home/runner/work/skupper-router/skupper-router/skupper-router/src/adaptors/http2/http2_adaptor.c:2881 (skrouterd+0x45a0ee)
57:     #3 router_core_thread /home/runner/work/skupper-router/skupper-router/skupper-router/src/router_core/router_core_thread.c:252 (skrouterd+0x507177) (BuildId: ff494da932379fc07ddd8270f2e95dd00f09c918)
57:     #4 _thread_init /home/runner/work/skupper-router/skupper-router/skupper-router/src/posix/threading.c:207 (skrouterd+0x4bf2dd) (BuildId: ff494da932379fc07ddd8270f2e95dd00f09c918)
57: 
57:   Location is heap block of size 448 at 0x724c00096f00 allocated by thread T4:
57:     #0 posix_memalign <null> (libtsan.so.2+0x556b6) (BuildId: 7cbb0e7a8424da37537f8f181125fcd8fd60706e)
57:     #1 qd_alloc /home/runner/work/skupper-router/skupper-router/skupper-router/src/alloc_pool.c:348 (skrouterd+0x48779a) (BuildId: ff494da932379fc07ddd8270f2e95dd00f09c918)
57:     #2 new_qdr_http2_connection_t /home/runner/work/skupper-router/skupper-router/skupper-router/src/adaptors/http2/http2_adaptor.c:53 (skrouterd+0x460f86) (BuildId: ff494da932379fc07ddd8270f2e95dd00f09c918)
57:     #3 qdr_http_connection_egress /home/runner/work/skupper-router/skupper-router/skupper-router/src/adaptors/http2/http2_adaptor.c:3053 (skrouterd+0x460f86)
57:     #4 qd_http2_configure_connector /home/runner/work/skupper-router/skupper-router/skupper-router/src/adaptors/http2/http2_adaptor.c:3496 (skrouterd+0x46428a) (BuildId: ff494da932379fc07ddd8270f2e95dd00f09c918)
57:     #5 qd_dispatch_configure_http_connector /home/runner/work/skupper-router/skupper-router/skupper-router/src/adaptors/http_common.c:212 (skrouterd+0x43e23d) (BuildId: ff494da932379fc07ddd8270f2e95dd00f09c918)
57:     #6 ffi_call_unix64 <null> (libffi.so.8+0x9055) (BuildId: a190bf03e644181cadab122962ab83ae96271696)
57:     #7 qdr_forward_on_message /home/runner/work/skupper-router/skupper-router/skupper-router/src/router_core/forwarder.c:375 (skrouterd+0x4fc648) (BuildId: ff494da932379fc07ddd8270f2e95dd00f09c918)
57:     #8 qdr_general_handler /home/runner/work/skupper-router/skupper-router/skupper-router/src/router_core/router_core.c:1032 (skrouterd+0x4fa179) (BuildId: ff494da932379fc07ddd8270f2e95dd00f09c918)
57:     #9 qd_timer_visit /home/runner/work/skupper-router/skupper-router/skupper-router/src/timer.c:317 (skrouterd+0x52ea21) (BuildId: ff494da932379fc07ddd8270f2e95dd00f09c918)
57:     #10 handle_proactor_other_event /home/runner/work/skupper-router/skupper-router/skupper-router/src/server.c:142 (skrouterd+0x52ce44) (BuildId: ff494da932379fc07ddd8270f2e95dd00f09c918)
57:     #11 proactor_thread /home/runner/work/skupper-router/skupper-router/skupper-router/src/server.c:194 (skrouterd+0x51b9f8) (BuildId: ff494da932379fc07ddd8270f2e95dd00f09c918)
57:     #12 _thread_init /home/runner/work/skupper-router/skupper-router/skupper-router/src/posix/threading.c:207 (skrouterd+0x4bf2dd) (BuildId: ff494da932379fc07ddd8270f2e95dd00f09c918)
57: 
57:   Thread T6 'wrkr_3' (tid=6979, running) created by main thread at:
57:     #0 pthread_create <null> (libtsan.so.2+0x5a6e6) (BuildId: 7cbb0e7a8424da37537f8f181125fcd8fd60706e)
57:     #1 sys_thread /home/runner/work/skupper-router/skupper-router/skupper-router/src/posix/threading.c:229 (skrouterd+0x4c021a) (BuildId: ff494da932379fc07ddd8270f2e95dd00f09c918)
57:     #2 qd_server_run /home/runner/work/skupper-router/skupper-router/skupper-router/src/server.c:298 (skrouterd+0x52d381) (BuildId: ff494da932379fc07ddd8270f2e95dd00f09c918)
57:     #3 main_process /home/runner/work/skupper-router/skupper-router/skupper-router/router/src/main.c:111 (skrouterd+0x430480) (BuildId: ff494da932379fc07ddd8270f2e95dd00f09c918)
57:     #4 main /home/runner/work/skupper-router/skupper-router/skupper-router/router/src/main.c:365 (skrouterd+0x42a20e) (BuildId: ff494da932379fc07ddd8270f2e95dd00f09c918)
57: 
57:   Thread T2 'core_thread' (tid=6973, running) created by main thread at:
57:     #0 pthread_create <null> (libtsan.so.2+0x5a6e6) (BuildId: 7cbb0e7a8424da37537f8f181125fcd8fd60706e)
57:     #1 sys_thread /home/runner/work/skupper-router/skupper-router/skupper-router/src/posix/threading.c:229 (skrouterd+0x4c021a) (BuildId: ff494da932379fc07ddd8270f2e95dd00f09c918)
57:     #2 qdr_core /home/runner/work/skupper-router/skupper-router/skupper-router/src/router_core/router_core.c:130 (skrouterd+0x50995c) (BuildId: ff494da932379fc07ddd8270f2e95dd00f09c918)
57:     #3 qd_router_setup_late /home/runner/work/skupper-router/skupper-router/skupper-router/src/dispatch.c:430 (skrouterd+0x49da79) (BuildId: ff494da932379fc07ddd8270f2e95dd00f09c918)
57:     #4 ffi_call_unix64 <null> (libffi.so.8+0x9055) (BuildId: a190bf03e644181cadab122962ab83ae96271696)
57:     #5 main_process /home/runner/work/skupper-router/skupper-router/skupper-router/router/src/main.c:101 (skrouterd+0x430422) (BuildId: ff494da932379fc07ddd8270f2e95dd00f09c918)
57:     #6 main /home/runner/work/skupper-router/skupper-router/skupper-router/router/src/main.c:365 (skrouterd+0x42a20e) (BuildId: ff494da932379fc07ddd8270f2e95dd00f09c918)
57: 
57:   Thread T4 'wrkr_1' (tid=6977, running) created by main thread at:
57:     #0 pthread_create <null> (libtsan.so.2+0x5a6e6) (BuildId: 7cbb0e7a8424da37537f8f181125fcd8fd60706e)
57:     #1 sys_thread /home/runner/work/skupper-router/skupper-router/skupper-router/src/posix/threading.c:229 (skrouterd+0x4c021a) (BuildId: ff494da932379fc07ddd8270f2e95dd00f09c918)
57:     #2 qd_server_run /home/runner/work/skupper-router/skupper-router/skupper-router/src/server.c:298 (skrouterd+0x52d381) (BuildId: ff494da932379fc07ddd8270f2e95dd00f09c918)
57:     #3 main_process /home/runner/work/skupper-router/skupper-router/skupper-router/router/src/main.c:111 (skrouterd+0x430480) (BuildId: ff494da932379fc07ddd8270f2e95dd00f09c918)
57:     #4 main /home/runner/work/skupper-router/skupper-router/skupper-router/router/src/main.c:365 (skrouterd+0x42a20e) (BuildId: ff494da932379fc07ddd8270f2e95dd00f09c918)
57: 
57: SUMMARY: ThreadSanitizer: data race /home/runner/work/skupper-router/skupper-router/skupper-router/src/server.c:107 in handle_event_with_context
57: ==================
57: ThreadSanitizer: reported 1 warnings

https://github.com/skupperproject/skupper-router/actions/runs/9664630485/job/26659738876#step:34:5066