search

ffvvc / FFmpeg

VVC Decoder for ffmpeg
Other
48 stars 12 forks source link

Stack overflow on 32 bit #188

Closed frankplow closed 4 months ago

frankplow commented 5 months ago

32 bit systems seem to suffer widely from stack overflows. Below is an address sanitizer log for ARMv7, but x86_32-mingw and x86_32-linux are certainly both affected.

ffmpeg version N-113412-g0b8e51b584 Copyright (c) 2000-2024 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.4.0-1ubuntu1~20.04.2)
  configuration: --arch=armv7 --cc=arm-linux-gnueabihf-gcc --enable-gpl --samples=/home/martin/fate/samples --extra-cflags='-fsanitize=address,undefined' --extra-ldflags='-fsanitize=address,undefined'
  libavutil      58. 36.101 / 58. 36.101
  libavcodec     60. 38.100 / 60. 38.100
  libavformat    60. 20.100 / 60. 20.100
  libavdevice    60.  4.100 / 60.  4.100
  libavfilter     9. 17.100 /  9. 17.100
  libswscale      7.  6.100 /  7.  6.100
  libswresample   4. 13.100 /  4. 13.100
  libpostproc    57.  4.100 / 57.  4.100
src/libavcodec/vvc/vvc_intra_template.c:972:22: runtime error: shift exponent 32 is too large for 32-bit type 'int'
Input #0, vvc, from '/home/martin/fate/samples/vvc-conformance/APSALF_A_2.bit':
  Duration: N/A, bitrate: N/A
  Stream #0:0: Video: vvc (Main 10), yuv420p10le(tv), 1280x720, 25 fps, 1200k tbr, 1200k tbn
src/fftools/ffmpeg_opt.c:105:15: runtime error: pointer index expression with base 0xffbf63f0 overflowed to 0x040aeb1d
Stream mapping:
  Stream #0:0 -> #0:0 (vvc (native) -> rawvideo (native))
=================================================================
==3195793==ERROR: AddressSanitizer: stack-buffer-overflow on address 0xf24fe994 at pc 0x080d95f3 bp 0xf24fe200 sp 0xf24fe200
WRITE of size 2 at 0xf24fe994 thread T2
    #0 0x80d95f0 in prepare_intra_edge_params_10 src/libavcodec/vvc/vvc_intra_template.c:574
    #1 0x80d95f0 in intra_pred_10 src/libavcodec/vvc/vvc_intra_template.c:627
    #2 0x8f8af86 in predict_intra src/libavcodec/vvc/vvc_intra.c:258
    #3 0x8f8af86 in reconstruct src/libavcodec/vvc/vvc_intra.c:575
    #4 0x8f8af86 in ff_vvc_reconstruct src/libavcodec/vvc/vvc_intra.c:603
    #5 0x9018208 in run_recon src/libavcodec/vvc/vvc_thread.c:459
    #6 0x901f02c in task_run_stage src/libavcodec/vvc/vvc_thread.c:606
    #7 0x901f02c in task_run src/libavcodec/vvc/vvc_thread.c:633
    #8 0x961e962 in run_one_task src/libavutil/executor.c:84
    #9 0x961e962 in executor_worker_task src/libavutil/executor.c:102
    #10 0xf7921d8e in asan_thread_start ../../../../src/libsanitizer/asan/asan_interceptors.cc:199

Address 0xf24fe994 is located in stack of thread T2 at offset 1684 in frame
    #0 0x80d5074 in intra_pred_10 src/libavcodec/vvc/vvc_intra_template.c:597

  This frame has 2 object(s):
    [48, 68) 'intra_hor_ver_dist_thres' (line 559)
    [112, 1684) 'edge' (line 625) <== Memory access at offset 1684 overflows this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
      (longjmp and C++ exceptions *are* supported)
Thread T2 created by T0 here:
    #0 0xf7921dea in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cc:208

SUMMARY: AddressSanitizer: stack-buffer-overflow src/libavcodec/vvc/vvc_intra_template.c:574 in prepare_intra_edge_params_10
Shadow bytes around the buggy address:
  0x3e49fce0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x3e49fcf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x3e49fd00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x3e49fd10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x3e49fd20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x3e49fd30: 00 00[04]f3 f3 f3 f3 f3 f3 f3 f3 f3 f3 f3 f3 f3
  0x3e49fd40: f3 f3 f3 f3 00 00 00 00 00 00 00 00 00 00 00 00
  0x3e49fd50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x3e49fd60: 00 00 00 00 f1 f1 f1 f1 f1 f1 04 f2 04 f2 04 f2
  0x3e49fd70: 04 f3 f3 f3 00 00 00 00 00 00 00 00 00 00 00 00
  0x3e49fd80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==3195793==ABORTING
threads=1
nuomi2021 commented 5 months ago

@frankplow , thank you for the report. I can't reproduce the issue on x86 with a 32-bit arch. (./configure --extra-cflags=-m32 --extra-ldflags=-m32) could you help change the code like this?

IntraEdgeParams edge = (IntraEdgeParams )lc->edge_emu_buffer

nuomi2021 commented 5 months ago

@frankplow , is this related to the log2 patch? Actually, I can reproduce the crash. But after I fix the log2 things, I can't reproduce it anymore. So I think it may be a compiler issue since https://en.cppreference.com/w/c/language/conversion. Maybe, I am wrong.

frankplow commented 5 months ago

could you help change the code like this?

IntraEdgeParams edge = (IntraEdgeParams )lc->edge_emu_buffer

This fixes the problem for me. I also got Martin to test it as he has a variety of runners set up, and he said it mostly fixed the problem for him.. He says he still gets the error below on i686-mingw32-clang, but it looks seperate to me.

ffmpeg version N-113423-g00b288da73 Copyright (c) 2000-2024 the FFmpeg developers
  built with clang version 18.0.0 (https://github.com/llvm/llvm-project.git 89efffd463ecc39cb17440ef7bb66d26141322aa)
  configuration: --cross-prefix=i686-w64-mingw32- --target-os=mingw32 --arch=i686 --enable-gpl --samples=../fate-samples --extra-cflags='-fsanitize=address,undefined' --extra-ldflags='-fsanitize=address,undefined' --disable-stripping
  libavutil      58. 36.101 / 58. 36.101
  libavcodec     60. 38.100 / 60. 38.100
  libavformat    60. 20.100 / 60. 20.100
  libavdevice    60.  4.100 / 60.  4.100
  libavfilter     9. 17.100 /  9. 17.100
  libswscale      7.  6.100 /  7.  6.100
  libswresample   4. 13.100 /  4. 13.100
  libpostproc    57.  4.100 / 57.  4.100
C:/code/ffmpeg/libavcodec/cbs_sei_syntax_template.c:185:9: runtime error: call to function cbs_h266_read_sei_decoded_picture_hash through pointer to incorrect function type 'int (*)(struct CodedBitstreamContext *, struct GetBitContext *, void *, struct SEIMessageState *)'
C:/code/ffmpeg/libavcodec/cbs_h266_syntax_template.c:3427: note: cbs_h266_read_sei_decoded_picture_hash defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior C:/code/ffmpeg/libavcodec/cbs_sei_syntax_template.c:185:9 in 
C:/code/ffmpeg/libavcodec/vvc/vvc_intra_template.c:972:22: runtime error: shift exponent 32 is too large for 32-bit type 'int'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior C:/code/ffmpeg/libavcodec/vvc/vvc_intra_template.c:972:22 in 
Input #0, vvc, from '../fate-samples/vvc-conformance/APSALF_A_2.bit':
  Duration: N/A, bitrate: N/A
  Stream #0:0: Video: vvc (Main 10), yuv420p10le(tv), 1280x720, 25 fps, 1200k tbr, 1200k tbn
Stream mapping:
  Stream #0:0 -> #0:0 (vvc (native) -> rawvideo (native))
C:/code/ffmpeg/libavcodec/vvc/vvc_ctu.c:260:26: runtime error: -nan(ind) is outside the range of representable values of type 'int'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior C:/code/ffmpeg/libavcodec/vvc/vvc_ctu.c:260:26 in 
C:/code/ffmpeg/libavcodec/vvc/vvc_ctu.c:261:26: runtime error: -nan(ind) is outside the range of representable values of type 'int'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior C:/code/ffmpeg/libavcodec/vvc/vvc_ctu.c:261:26 in 
C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2000:27: runtime error: signed integer overflow: -2147483648 + -2147483648 cannot be represented in type 'int'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2000:27 in 
C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2012:52: runtime error: signed integer overflow: -2147483648 + -2147483648 cannot be represented in type 'int'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2012:52 in 
C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2012:30: runtime error: shift exponent -2 is negative
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2012:30 in 
C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2015:53: runtime error: signed integer overflow: -2147483648 + -2147483648 cannot be represented in type 'int'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2015:53 in 
C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2018:61: runtime error: signed integer overflow: -2147483648 - 1 cannot be represented in type 'int'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2018:61 in 
C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2018:25: runtime error: index 2147483647 out of bounds for type 'const uint8_t[5][5][256]' (aka 'const unsigned char[5][5][256]')
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2018:25 in 
C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2018:25: runtime error: addition of unsigned offset to 0x08397fa0 overflowed to 0x08397aa0
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2018:25 in 
C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2018:92: runtime error: signed integer overflow: -2147483648 - 1 cannot be represented in type 'int'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2018:92 in 
C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2018:25: runtime error: index 2147483647 out of bounds for type 'const uint8_t[5][256]' (aka 'const unsigned char[5][256]')
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2018:25 in 
C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2018:25: runtime error: addition of unsigned offset to 0x08397aa0 overflowed to 0x083979a0
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2018:25 in 
C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2019:61: runtime error: signed integer overflow: -2147483648 - 1 cannot be represented in type 'int'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2019:61 in 
C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2019:25: runtime error: index 2147483647 out of bounds for type 'const uint8_t[5][5][256]' (aka 'const unsigned char[5][5][256]')
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2019:25 in 
C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2019:25: runtime error: addition of unsigned offset to 0x08399ee0 overflowed to 0x083999e0
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2019:25 in 
C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2019:92: runtime error: signed integer overflow: -2147483648 - 1 cannot be represented in type 'int'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2019:92 in 
C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2019:25: runtime error: index 2147483647 out of bounds for type 'const uint8_t[5][256]' (aka 'const unsigned char[5][256]')
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2019:25 in 
C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2019:25: runtime error: addition of unsigned offset to 0x083999e0 overflowed to 0x083998e0
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2019:25 in 
C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2026:49: runtime error: signed integer overflow: -2147483648 - 1 cannot be represented in type 'int'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2026:49 in 
C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2026:28: runtime error: shift exponent 2147483647 is too large for 32-bit type 'int'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2026:28 in 
C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2027:50: runtime error: signed integer overflow: -2147483648 - 1 cannot be represented in type 'int'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2027:50 in 
C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2027:28: runtime error: shift exponent 2147483647 is too large for 32-bit type 'int'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2027:28 in 
C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2028:42: runtime error: signed integer overflow: -2147483648 * -2147483648 cannot be represented in type 'int'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2028:42 in 
C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:1676:32: runtime error: left shift of negative value -2147483648
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:1676:32 in 
C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:1682:44: runtime error: signed integer overflow: -2147483648 - 1 cannot be represented in type 'int'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:1682:44 in 
C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:1682:22: runtime error: index 2147483647 out of bounds for type 'const int[6]'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:1682:22 in 
C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:1682:22: runtime error: addition of unsigned offset to 0x1c6ee630 overflowed to 0x1c6ee62c
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:1682:22 in 
=================================================================
==3444==ERROR: AddressSanitizer: stack-buffer-underflow on address 0x1c6ee62c at pc 0x07835773 bp 0x1c6ee594 sp 0x1c6ee590
READ of size 4 at 0x1c6ee62c thread T2
    #0 0x7835772 in last_significant_coeff_xy_prefix C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:1682:22
    #1 0x7835772 in last_significant_coeff_x_prefix_decode C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:1697:12
    #2 0x7835772 in last_significant_coeff_x_y_decode C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2335:32
    #3 0x7835772 in hls_residual_coding C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2380:5
    #4 0x7835772 in ff_vvc_residual_coding C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2412:50
    #5 0x64fcd4c in hls_transform_unit C:/code/ffmpeg/libavcodec/vvc/vvc_ctu.c:392:19

Address 0x1c6ee62c is located in stack of thread T2 at offset 12 in frame
    #0 0x7834257 in ff_vvc_residual_coding C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:2408

  This frame has 6 object(s):
    [16, 40) 'offset_y.i171.i.i' (line 1681) <== Memory access at offset 12 underflows this variable
    [80, 108) 'shifts.i172.i.i' (line 1685)
    [144, 168) 'offset_y.i.i.i' (line 1681)
    [208, 236) 'shifts.i.i.i' (line 1685)
    [272, 66144) 'rc.i25' (line 2367)
    [66400, 132272) 'rc.i' (line 2156)
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
      (longjmp, SEH and C++ exceptions *are* supported)
Thread T2 created by T0 here:
    #0 0x740abdbd in CreateThread /home/runner/work/llvm-mingw/llvm-mingw/llvm-project/compiler-rt/lib/asan/asan_win.cpp:158:3
    #1 0x76f27579  (C:\Windows\System32\ucrtbase.dll+0x10047579)
    #2 0x7775310 in pthread_create C:/code/ffmpeg/compat/w32pthreads.h:91:29
    #3 0x7775310 in av_executor_alloc C:/code/ffmpeg/libavutil/executor.c:164:13

SUMMARY: AddressSanitizer: stack-buffer-underflow C:/code/ffmpeg/libavcodec/vvc/vvc_cabac.c:1682:22 in last_significant_coeff_xy_prefix
Shadow bytes around the buggy address:
  0x1c6ee380: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1c6ee400: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1c6ee480: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1c6ee500: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1c6ee580: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x1c6ee600: 00 00 00 00 f1[f1]00 00 00 f2 f2 f2 f2 f2 f8 f8
  0x1c6ee680: f8 f8 f2 f2 f2 f2 f8 f8 f8 f2 f2 f2 f2 f2 f8 f8
  0x1c6ee700: f8 f8 f2 f2 f2 f2 00 00 00 00 00 00 00 00 00 00
  0x1c6ee780: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1c6ee800: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1c6ee880: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==3444==ABORTING
threads=1

@frankplow , is this related to the log2 patch? Actually, I can reproduce the crash. But after I fix the log2 things, I can't reproduce it anymore.

Yeah same for me. I think it might just be that log2 uses a lot of stack memory.

frankplow commented 5 months ago

@nuomi2021 Martin found the root cause and posted a fix here: https://patchwork.ffmpeg.org/project/ffmpeg/patch/20240126130427.2159537-1-martin@martin.st/

Though he says he still has stack overflows for his ARMv7 build, so I think we still need to move the IntraEdgeParams and other large objects from the stack to the heap. Rather than reusing edge_emu_buffer, is it better to allocate memory specifically for IntraEdgeParams?

frankplow commented 4 months ago

IntraEdgeParams.(filtered_)?(top|left)_array were too small in some cases. Resolved by 85e031d5bfa83c25e4b644e3453fe8073d959a4c.