Occasional Encoder crash on aacEncClose

Hello Martin.

I'm writing a sample program that basically recreates a single AACENCODER many times internally within a single process. The purpose of the program is to encode input wav files, each using its own dedicated AACENCODER instance. Everything works and behave as expected and go on like this even when iterating over 800 files.

On rare occasions, independent of input size/index, the entire program crashes with a coredump. Looking at the coredump, I can see the crash takes place on aacEncClose

#1  0x00007f3e92e00f5d in __GI_abort () at abort.c:90
#2  0x00007f3e92e4928d in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7f3e92f70528 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:181
#3  0x00007f3e92e5064a in malloc_printerr (action=<optimized out>, str=0x7f3e92f6cdee "corrupted size vs. prev_size", ptr=<optimized out>, ar_ptr=<optimized out>) at malloc.c:5426
#4  0x00007f3e92e5304a in _int_free (av=0x7f3de0000020, p=<optimized out>, have_lock=0) at malloc.c:4337
#5  0x00007f3e92e5744e in __GI___libc_free (mem=<optimized out>) at malloc.c:3145
#6  0x00007f3e113921e9 in FDKfree (ptr=0x7f3de009df60) at libSYS/src/genericStds.cpp:233
#7  0x00007f3e1130d7d3 in Free_AacEncoder (p=0x7f3de0115740) at libAACenc/src/aacenc_lib.cpp:407
#8  0x00007f3e1130fbb3 in aacEncClose (phAacEncoder=0x7f3de0115740) at libAACenc/src/aacenc_lib.cpp:1395

The crash always happen in exactly the same place Now I understand "corrupted size vs. prev_size" is an indication to some sort of heap abuse.

Looking at a verbose dump, I spotted something odd:

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
        set = {__val = {4, 6378670679680, 645636045657660056, 90523359816, 139904561311072, 292199584, 139903730612120, 139903730611784, 139904561311088, 1460617926600, 47573685816, 4119199860131166208, 
            139904593745464, 139904553224483, 139904561311136, 288245657}}
        pid = <optimized out>
        tid = <optimized out>
#1  0x00007f3e92e00f5d in __GI_abort () at abort.c:90
        save_stage = 2
        act = {__sigaction_handler = {sa_handler = 0x7f3de026db10, sa_sigaction = 0x7f3de026db10}, sa_mask = {__val = {139903730540556, 19, 30064771092, 812522497172832284, 139903728706672, 1887866374039011357, 
              139900298780168, 3775732748407067896, 763430436865, 35180077121538, 4119199860131166208, 139904561311552, 139904553065676, 1, 139904561311584, 139904561312192}}, sa_flags = 4096, 
          sa_restorer = 0x14}
        sigs = {__val = {32, 0 <repeats 15 times>}}
#2  0x00007f3e92e4928d in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7f3e92f70528 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:181
        ap = {{gp_offset = 40, fp_offset = 32574, overflow_arg_area = 0x7f3e11adf1d0, reg_save_area = 0x7f3e11adf160}}
        fd = <optimized out>
        list = <optimized out>
        nlist = <optimized out>
        cp = <optimized out>
        written = <optimized out>
#3  0x00007f3e92e5064a in malloc_printerr (action=<optimized out>, str=0x7f3e92f6cdee "corrupted size vs. prev_size", ptr=<optimized out>, ar_ptr=<optimized out>) at malloc.c:5426
        buf = "00007f3de009e9f0"
        cp = <optimized out>
        ar_ptr = <optimized out>
        ptr = <optimized out>
        str = 0x7f3e92f6cdee "corrupted size vs. prev_size"
        action = <optimized out>
#4  0x00007f3e92e5304a in _int_free (av=0x7f3de0000020, p=<optimized out>, have_lock=0) at malloc.c:4337
        size = 2720
        fb = <optimized out>
        nextchunk = 0x7f3de009e9f0
        nextsize = 736
        nextinuse = <optimized out>
        prevsize = <optimized out>
        bck = <optimized out>
        fwd = <optimized out>
        errstr = 0x0
        locked = <optimized out>
#5  0x00007f3e92e5744e in __GI___libc_free (mem=<optimized out>) at malloc.c:3145
        ar_ptr = <optimized out>
        p = <optimized out>
        hook = <optimized out>
#6  0x00007f3e113921e9 in FDKfree (ptr=0x7f3de009df60) at libSYS/src/genericStds.cpp:233
No locals.
#7  0x00007f3e1130d7d3 in Free_AacEncoder (p=0x7f3de0115740) at libAACenc/src/aacenc_lib.cpp:407
No locals.
#8  0x00007f3e1130fbb3 in aacEncClose (phAacEncoder=0x7f3de0115740) at libAACenc/src/aacenc_lib.cpp:1395
        hAacEncoder = 0x7f3de009df60
        err = AACENC_OK

Notice the address of hAacEncoder - 0x7f3de009df60
Its expected size is 2720, or 0xaa0
If you add 0xaa0 to 0x7f3de009df60 you get 0x7f3de009ea00
However, looking at nextchunk inside _int_free it points to 0x7f3de009e9f0, which is smaller than 0x7f3de009ea00 by 16 (0x10)
I know for a fact (looking at glibc source code) that "corrupted size vs. prev_size" is a result of an invalid size comparison between the prevsize of the next chunk and the current chunk size.

Any help/advice would be great.

* As a side note, the event loop is run from a Java program which links to your library via JNA. AACEncoder memory allocations are, of course, still done using aacEncOpen and aacEncClose* .

mstorsjo / fdk-aac

Occasional Encoder crash on aacEncClose #82

Never mind. This issue was related to JNA mapping field alignment ( fields were out of bounds, causing bad overall structure sizing). Fixed in my code.