backtrace segmentation fault

[abmusse]

While running mysql_install_db I ran into the following error:

/QOpenSys/pkgs/bin/mysql_install_db[2]: 8628721 Memory fault(coredump)

Examining the core file I found:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0900000000985dc0 in backtrace () from /QOpenSys/pkgs/lib/libutil.so.2(shr_64.o)
(gdb) bt
#0  0x0900000000985dc0 in backtrace () from /QOpenSys/pkgs/lib/libutil.so.2(shr_64.o)
#1  0x0000000100031c48 in sf_malloc (size=1480, my_flags=16)
    at /home/AMUSSE/rpmbuild/BUILD/mariadb-10.3.14/mysys/safemalloc.c:141
#2  0x000000010003074c in my_malloc (size=1480, my_flags=16)
    at /home/AMUSSE/rpmbuild/BUILD/mariadb-10.3.14/mysys/my_malloc.c:101
#3  0x0000000100f042ec in init_queue (queue=0x180cee7e8 <_thralarm.bss_3_+312>, max_elements=184, 
    offset_to_key=0, max_at_top=0 '�', compare=@0x1803cc4d8: 0x10129fe08 <compare_ulong>, first_cmp_arg=0x0, 
    offset_to_queue_pos=25, auto_extent=10) at /home/AMUSSE/rpmbuild/BUILD/mariadb-10.3.14/mysys/queues.c:79
#4  0x000000010129ff00 in init_thr_alarm (max_alarms=183)
    at /home/AMUSSE/rpmbuild/BUILD/mariadb-10.3.14/mysys/thr_alarm.c:92
#5  0x0000000100009dec in signal_hand (arg=0x0) at /home/AMUSSE/rpmbuild/BUILD/mariadb-10.3.14/sql/mysqld.cc:3512
#6  0x0900000000583da4 in _pthread_body () from /QOpenSys/usr/lib/libpthreads.a(shr_xpg5_64.o)
#7  0x0000000000000000 in ?? ()

[abmusse]

I added some print statements to help diagnose the issue:

size_t backtrace(void** frames, size_t count)
{
    size_t i;
    int debug = 0;
    // We ignore the current stack frame (this function)
    void** sp = getsp()[0];

    // Walk the stack up to count times or we hit the bottom
    // of the stack (whichever is first)
    for(i = 0; i < count && sp; ++i, sp = (void**) sp[0]) {
        if(debug){
          fprintf(stderr, "loop\n");
        }
        void *lr = sp[2];
        frames[i] = lr; 
        // This might be a signal handler frame, which means the back chain is
        // useless (missing/in the weeds), so look at what's in the frame. In
        // this case, what would be the back chain is in one of the fields of
        // the frame. We need to guess if this is a signal handler frame
        // though; our heuristic is the address being lower than the base of
        // text; the signal trampoline is around ~0x3680 under PASE, but AIX
        // has it at a different address around ~0x4800. The value we use for
        // the displacement is verified to be the proper offset by GDB, and we
        // employ a similar heuristic.
        // XXX: What about syscalls?
        if (lr < (void*)TEXTORG) {
            debug = 1;
            sp = (void*)((uint64_t)sp + TRAMPOLINE_OFFSET);
        }
        if (debug) {
            fprintf(stderr, "i: %d, count: %d, lr: %p, sp: %p\n", i, count, lr, sp);   
        }
    }   

    if (debug) {
      fprintf(stderr, "Backtrace\n");
   }   

    return i;

When I ran mysql_install_db gain I got the following output:

...

i: 9, count: 10, lr: 0, sp: 1929a98b8
Backtrace
i: 9, count: 10, lr: 0, sp: 1929a98b8
Backtrace
i: 6, count: 10, lr: 0, sp: 192a0c8b8
loop
/QOpenSys/pkgs/bin/mysql_install_db[2]: 8628789 Memory fault(coredump)

Each time if (lr < (void*)TEXTORG) check was entered lr = 0. Right before the memory fault error lr = 0 and we re-enter the loop and attempt to access invalid memory with void *lr = sp[2]; therefore causing the memory fault.

I was able to resolve this issue by ensuring lr !=0 before adding the offset to the stack pointer:

        if (lr < (void*)TEXTORG && lr != 0) {
            sp = (void*)((uint64_t)sp + TRAMPOLINE_OFFSET);
        }

This time when I ran mysql_install_db I did not run into the memory fault.