SIGSEGV yami decode MD5 ouput mode (-m -2)

[stripes416]

Since the following commit in libyami decode, MD5 output mode (-m -2) segfaults (see stack trace below).

commit a088df6e6f762a27f7b13b8342260ad100920fbf
Author: wudping <dongpingx.wu@intel.com>
Date:   Thu Mar 30 11:03:07 2017 +0800

    com_unittest: modify over range resolution

    When the width or hight overflows the maxmum value of int,
    guessResolution() returns false.

    Signed-off-by: wudping <dongpingx.wu@intel.com>

Starting program: /opt/media/install/bin/yamidecode -i /opt/media/src/media_smoke/media_streams/decodebitstreams/vp9_10bit/basic/vp9_kf_2frm_SplitLvl3_10bit_fastfood_64x64.ivf -m -2 -f P010
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
libva info: VA-API version 0.40.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /opt/media/install/lib/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_0_40
libva info: va_openDriver() returns 0

Program received signal SIGSEGV, Segmentation fault.
___fprintf_chk (fp=0x5f336c764c74696c, flag=flag@entry=1, format=format@entry=0x417ced "%s\n") at fprintf_chk.c:30
30  fprintf_chk.c: No such file or directory.
(gdb) bt full
#0  ___fprintf_chk (fp=0x5f336c764c74696c, flag=flag@entry=1, format=format@entry=0x417ced "%s\n") at fprintf_chk.c:30
        _IO_acquire_lock_file = 0x5f336c764c74696c
        ap = <error reading variable ap (Attempt to dereference a generic pointer.)>
        done = <optimized out>
#1  0x0000000000406e1d in fprintf (__fmt=0x417ced "%s\n", __stream=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/stdio2.h:98
No locals.
#2  DecodeOutputMD5::writeToFile[abi:cxx11](MD5Context&) (this=this@entry=0x6526b0, t_ctx=...) at decodeoutput.cpp:402
        temp = "b6\000"
        result = "i\017\037<\213\206\263u\245/\263\305\303|\224\266"
        strMD5 = "690f1f3c8b86b375a52fb3c5c37c94b6"
#3  0x0000000000408ab5 in DecodeOutputMD5::output (this=0x6526b0, frame=...) at decodeoutput.cpp:429
        frameMD5 = {state = {0, 0, 0, 0}, count = 0, buffer = '\000' <repeats 63 times>}
        frame = <optimized out>
        this = 0x6526b0
#4  0x00000000004047cb in DecodeTest::run (this=this@entry=0x7fffffffdf20) at decode.cpp:83
        fps = {m_frames = 0, m_start = 0, m_netStart = 4287696, static NET_FPS_START = 5}
        src = std::tr1::shared_ptr (count 1, weak 0) 0x6626f0
        count = 0
#5  0x0000000000404061 in main (argc=<optimized out>, argv=<optimized out>) at decode.cpp:106
        decode = {m_output = std::tr1::shared_ptr (count 1, weak 0) 0x6526b0, m_nativeDisplay = std::tr1::shared_ptr (count 2, weak 0) 0x6339c0, m_vppInput = 
    std::tr1::shared_ptr (count 1, weak 0) 0x660e90, m_params = {
            inputFile = 0x7fffffffe3ae "/opt/media/src/media_smoke/media_streams/decodebitstreams/vp9_10bit/basic/vp9_kf_2frm_SplitLvl3_10bit_fastfood_64x64.ivf", 
            width = 64, height = 64, renderMode = -2, waitBeforeQuit = 1, renderFrames = 4294967295, renderFourcc = 808530000, outputFile = "./", useCAPI = false, 
            temporalLayer = 0, spacialLayer = 0, qualityLayer = 0}}
(gdb)

[uartie]

I am only able to reproduce with HEVC 10 bit and VP9 10 bit streams. I could not reproduce with other codecs.

[uartie]

m_file is an invalid pointer.... looks like someone forgot to initialize it to NULL.

[uartie]

C++'s std::ofstream would be better for file operations instead of raw C-style FILE* constructs.

[wudping]

Thanks @stripes416 for rising up this issue that I make. I am sorry for that. Thanks @uartie to fix this.

[xuguangxin]

Hi @wudping , do we have a regression test for -2?

[wudping]

Yes, @xuguangxin, we have. I have double-checked it on my end. But @uartie help to fix this bug with the pull request #98.

[uartie]

@xuguangxin and @wudping ... The commit mentioned in the description above did not actually introduce this bug... this bug has always been there. It was only coincidence that this patch changed memory layout enough in a way that exposed the problem. Any commit would have inevitably exposed this at some point. It's a surprise that the bug never surfaced until now.

[uartie]

Furthermore, uninitialized class member variables get arbitrary values... we've only been lucky until now that the arbitrary value was 0 (NULL) previously.

[xuguangxin]

it's little wired, static scan tool should report issues if we use a uninitialised memory. But it does not report it...