Abortion with tracing JIT

danog commented 12 months ago

Description

The following code: https://github.com/nicelocal/psalm/tree/rector_pass (a fork of Psalm, commit 9d3fee47afa90f3eb53043a26f01e587d2dd34e5)

Running using (the xdebug env is used to avoid overwriting the opcache config, xdebug is not installed nor enabled):

PSALM_ALLOW_XDEBUG=1 USE_ZEND_ALLOC=0 gdb --args php-asan ./psalm --no-cache

With this config:

memory_limit = -1
zend.assertions = 1
display_errors = On
display_startup_errors = On
;extension=uv
extension=gmp
extension=iconv
[opcache]
zend_extension=opcache
opcache.memory_consumption=4096M
opcache.enable=1
opcache.enable_cli=1
opcache.jit=tracing
opcache.validate_timestamps=0
opcache.jit_buffer_size=1G
opcache.file_update_protection=0
opcache.max_accelerated_files=1000000
opcache.interned_strings_buffer=64

;opcache.jit_debug=28672
;opcache.jit_debug=16384
opcache.jit_debug=256

opcache.jit_prof_threshold=0.000000001
opcache.jit_max_root_traces=  30000000
opcache.jit_max_side_traces=  30000000
opcache.jit_max_exit_counters=30000000
opcache.jit_hot_loop=1
opcache.jit_hot_func=1
opcache.jit_hot_return=1
opcache.jit_hot_side_exit=1

opcache.jit_blacklist_root_trace=255
opcache.jit_blacklist_side_trace=255

opcache.protect_memory=1

Resulted in this output:

Program received signal SIGABRT, Aborted.
__pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
44        return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;

(gdb) back
#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1  0x00007ffff6ffa8a3 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at pthread_kill.c:78
#2  0x00007ffff6faa668 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3  0x00007ffff6f924b8 in __GI_abort () at abort.c:79
#4  0x00007fffb180a457 in TRACE-1550$Psalm\Type\Union::getSingleAtomic$1527 () at unknown:1
#5  0x00007ffff48035f8 in ?? ()
#6  0x00007ffff48035f0 in ?? ()
#7  0x00007ffff48035f0 in ?? ()
#8  0x00000000f48035f0 in ?? ()
#9  0x5dfaca7550574600 in ?? ()
#10 0x0000000173596764 in ?? ()
#11 0x00007ffff4803800 in ?? ()
#12 0x0000555557d28798 in ?? ()
#13 0x00007ffff7ffd000 in ?? () from /lib64/ld-linux-x86-64.so.2
#14 0x00007fffffffde88 in ?? ()
#15 0x0000000000000000 in ?? ()

(gdb) zback
[0x7ffff4803830] Psalm\Type\Union->getSingleAtomic() /home/daniil/repos/psalm/src/Psalm/Type/UnionTrait.php:1527
[0x7ffff48035f0] Psalm\Internal\Analyzer\Statements\Expression\SimpleTypeInferer->infer(object[0x7ffff4803640], object[0x7ffff4803650],
object[0x7ffff4803660], object[0x7ffff4803670], NULL, array(0)[0x7ffff4803690], "Composer\Autoload\ComposerStaticInitee60cb38244924a30b9a4f1eb3dba

But I expected this output instead: no abortion.

ping @dstogov

PHP Version

master

Operating System

Arch linux

danog commented 11 months ago

A better reproducer: https://github.com/danog/jit_bugs (test 2).

dstogov commented 11 months ago

Thanks for the https://github.com/danog/jit_bugs repo. The test 2 should be fixed now. Test 1 is actually reported at https://github.com/php/php-src/issues/11795 Should I take a look into others?

danog commented 11 months ago

Hi @dstogov, thanks! It seems like issue 2 is still present on master (a heap overflow with ASAN, since the fix only affected the old JIT). Thanks for the info on issue 1, regarding the other issues, I think they can mostly be ignored, except for the last one (5), which is actually causing a GC infinite loop hang after several hours of execution, it's particularly nasty to reproduce, will try to whip up a better reproducer...

danog commented 11 months ago

Additionally, there's an issue with fibers unrelated to JIT or opcache that causes memory corruption in certain cases if a fiber is resumed after an exception in the shutdown handler, I've been seeing a lot of reports including in the amphp chat, and got the issue myself several times (an impossible TypeError on corrupted return values), but haven't managed to reliably reproduce yet...

The only way of sometimes reproducing the issue is running reproducer 3 and randomly hitting ctrl-c multiple times during the handshake, but still it's super hard to reproduce reliably...

dstogov commented 11 months ago

Hi @dstogov, thanks! It seems like issue 2 is still present on master (a heap overflow with ASAN, since the fix only affected the old JIT). Thanks for the info on issue 1, regarding the other issues, I think they can mostly be ignored, except for the last one (5), which is actually causing a GC infinite loop hang after several hours of execution, it's particularly nasty to reproduce, will try to whip up a better reproducer...

I think this is a different issue. New JIT avoided this error out of the box. BTW I also see a heap-buffer-overflow on test 2 with PHP-8.3. This is definitely a different problem.

==1==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6100000a6904 at pc 0x7fd17fc3e492 bp 0x7ffd5cbefbb0 sp 0x7ffd5cbefba8
READ of size 4 at 0x6100000a6904 thread T0
LLVMSymbolizer: error reading file: No such file or directory
    #0 0x7fd17fc3e491 in zend_jit_assign_to_typed_prop /php-src/ext/opcache/jit/zend_jit_helpers.c:2567:30
    #1 0x7fd13f9a5230  (/dev/zero (deleted)+0x100696230)
    #2 0x56521f81bbf7 in zend_execute /php-src/Zend/zend_vm_execute.h:61584:2
    #3 0x56521f767588 in zend_execute_scripts /php-src/Zend/zend.c:1876:4
    #4 0x56521f56d89f in php_execute_script /php-src/main/main.c:2492:13
    #5 0x56521fcf9847 in do_cli /php-src/sapi/cli/php_cli.c:966:5
    #6 0x56521fcf73da in main /php-src/sapi/cli/php_cli.c:1340:18
    #7 0x7fd1845eed8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: a43bfc8428df6623cd498c9c0caeb91aec9be4f9)

0x6100000a6904 is located 12 bytes after 184-byte region [0x6100000a6840,0x6100000a68f8)
allocated by thread T0 here:
    #0 0x7fd18539ea4e in __interceptor_malloc (/usr/lib/llvm-16/lib/clang/16/lib/linux/libclang_rt.asan-x86_64.so+0xe3a4e) (BuildId: 8c0845aee3ffdc74a2f46d1eb4f0a349cdc69d67)
    #1 0x56521f68d6d4 in __zend_malloc /php-src/Zend/zend_alloc.c:3130:14
    #2 0x56521f68cf2f in _malloc_custom /php-src/Zend/zend_alloc.c:2493:10
    #3 0x56521f68ce01 in _emalloc /php-src/Zend/zend_alloc.c:2612:10
    #4 0x56521fb047c3 in zend_objects_new /php-src/Zend/zend_objects.c:189:24
    #5 0x56521fb05dbd in zend_objects_clone_obj /php-src/Zend/zend_objects.c:291:15
    #6 0x56521f967883 in ZEND_CLONE_SPEC_TMPVAR_HANDLER /php-src/Zend/zend_vm_execute.h:14772:2
    #7 0x7fd13f9a51ee  (/dev/zero (deleted)+0x1006961ee)
    #8 0x56521f81bbf7 in zend_execute /php-src/Zend/zend_vm_execute.h:61584:2
    #9 0x56521f767588 in zend_execute_scripts /php-src/Zend/zend.c:1876:4
    #10 0x56521f56d89f in php_execute_script /php-src/main/main.c:2492:13
    #11 0x56521fcf9847 in do_cli /php-src/sapi/cli/php_cli.c:966:5
    #12 0x56521fcf73da in main /php-src/sapi/cli/php_cli.c:1340:18
    #13 0x7fd1845eed8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: a43bfc8428df6623cd498c9c0caeb91aec9be4f9)

The problem that this error occurs after an hour of work :(

dstogov commented 11 months ago

The last heap-buffer-overflow should be fixed now in PHP-8.1 and above

php / php-src