Loading and executing one of the attached files (s4.zip) results either in SIGILL (invalid instruction in extra_buffer) or SIGSEGV.
For SIGILL-causing file (cpuinstr1.z64) the invalid instruction generation seems to be caused by emit_writeword with rt=-1 argument (it seems to be wrong since regname[rt] is used inside this function, and it should not receive negative index)
$ gdb -ex run --args Bin/Release/RMG /tmp/cpuinstr1.z64
GNU gdb (Ubuntu 13.1-2ubuntu2) 13.1
...
Thread 14 "Thread::Emulati" received signal SIGILL, Illegal instruction.
[Switching to Thread 0x7fffa63fd6c0 (LWP 204015)]
0x00007fffcc9ee05e in g_dev () from /tmp/RMG/Bin/Release/Core/libmupen64plus.so
(gdb) disas /r $pc,$pc+20
Dump of assembler code from 0x7fffcc9ee05e to 0x7fffcc9ee072:
=> 0x00007fffcc9ee05e <g_dev+9441374>: ce (bad)
0x00007fffcc9ee05f <g_dev+9441375>: 00 00 add BYTE PTR [rax],al
0x00007fffcc9ee061 <g_dev+9441377>: 02 89 1d 98 00 00 add cl,BYTE PTR [rcx+0x981d]
0x00007fffcc9ee067 <g_dev+9441383>: 02 50 51 add dl,BYTE PTR [rax+0x51]
0x00007fffcc9ee06a <g_dev+9441386>: 48 83 c4 90 add rsp,0xffffffffffffff90
0x00007fffcc9ee06e <g_dev+9441390>: bf 98 05 00 a4 mov edi,0xa4000598
End of assembler dump.
(gdb) set $offset = $pc-g_dev.r4300.extra_memory
(gdb) b new_dyna_start
Breakpoint 1 at 0x7fffcc0bf70b
(gdb) r
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /tmp/RMG/Bin/Release/RMG /tmp/cpuinstr1.z64
...
Thread 14 "Thread::Emulati" hit Breakpoint 1, 0x00007fffc80bf70b in new_dyna_start () from /tmp/RMG/Bin/Release/Core/libmupen64plus.so
(gdb) p/x $offset
$1 = 0x5e
(gdb) watch *(g_dev.r4300.extra_memory+$offset)
Hardware watchpoint 2: *(g_dev.r4300.extra_memory+$offset)
(gdb) c
Continuing.
Thread 14 "Thread::Emulati" hit Hardware watchpoint 2: *(g_dev.r4300.extra_memory+$offset)
Old value = 0 '\000'
New value = -50 '\316'
output_w32 (word=33554638) at /tmp/RMG/Source/3rdParty/mupen64plus-core/src/device/r4300/new_dynarec/x64/assem_x64.c:747
747 out+=4;
(gdb) bt
#0 output_w32 (word=33554638) at /tmp/RMG/Source/3rdParty/mupen64plus-core/src/device/r4300/new_dynarec/x64/assem_x64.c:747
#1 0x00007fffc808690e in emit_writeword (rt=-1, addr=140736592798000) at /tmp/RMG/Source/3rdParty/mupen64plus-core/src/device/r4300/new_dynarec/x64/assem_x64.c:2315
#2 0x00007fffc8096d76 in do_readstub (n=0) at /tmp/RMG/Source/3rdParty/mupen64plus-core/src/device/r4300/new_dynarec/new_dynarec.c:4817
#3 0x00007fffc80bd59c in new_recompile_block (addr=-1543503808) at /tmp/RMG/Source/3rdParty/mupen64plus-core/src/device/r4300/new_dynarec/new_dynarec.c:11680
#4 0x00007fffc80bf723 in new_dyna_start () at /tmp/RMG/Bin/Release/Core/libmupen64plus.so
#5 0x0000000000000000 in ()
When it comes to SIGSEGV-causing file, I don't know the reason. Please note that the similar issue may exist in related files too (e.g. assem_x86.c).
Loading and executing one of the attached files (s4.zip) results either in SIGILL (invalid instruction in extra_buffer) or SIGSEGV.
For SIGILL-causing file (cpuinstr1.z64) the invalid instruction generation seems to be caused by
emit_writewordwithrt=-1argument (it seems to be wrong sinceregname[rt]is used inside this function, and it should not receive negative index)https://github.com/mupen64plus/mupen64plus-core/blob/f500eb58f76e636e231c3cc2b3d904210f0677c9/src/device/r4300/new_dynarec/x64/assem_x64.c#L2309-L2316
It may be the reason of the SIGILL.
Debugging cpuinstr1.z64 execution:
When it comes to SIGSEGV-causing file, I don't know the reason. Please note that the similar issue may exist in related files too (e.g. assem_x86.c).
Test platform
CFLAGS += -O0 -g-O0and-gfor better crash backtraces in Release build