google / oss-fuzz

OSS-Fuzz - continuous fuzzing for open source software.
https://google.github.io/oss-fuzz
Apache License 2.0
10.15k stars 2.16k forks source link

ffmpeg: Use latest builder #12007

Open maflcko opened 1 month ago

github-actions[bot] commented 1 month ago

maflcko has previously contributed to projects/ffmpeg. The previous PR was #11714

maflcko commented 1 month ago

cc @michaelni can you take a look here, please? When I tried this outside of OSS-Fuzz, it was also failing with clang-18, so it seems an issue unrelated to OSS-Fuzz. Let me know if I should submit a ticket to https://trac.ffmpeg.org/ instead.

maflcko commented 1 month ago

For reference, to reproduce the link error locally, I used ./configure --cc="clang-18" --cxx="clang++-18 -std=c++11" --ld="clang++-18 -std=c++11" --enable-ossfuzz && make clean && make.

Looks like disabling the asm works around it for now.

However, some build issues remain.

michaelni commented 1 month ago

cc @michaelni can you take a look here, please? When I tried this outside of OSS-Fuzz, it was also failing with clang-18, so it seems an issue unrelated to OSS-Fuzz. Let me know if I should submit a ticket to https://trac.ffmpeg.org/ instead.

My box here has no clang-18 nor is there a clang-18 package maybe you can post the errors you receive and what you did?

maflcko commented 1 month ago

So far there are two related (?) build issues.

clang-16 (and 17, 18) with --enable-ossfuzz

To reproduce:

./configure --cc="clang-16" --cxx="clang++-16 -std=c++11" --ld="clang++-16 -std=c++11" --enable-ossfuzz && make clean && make

Output (taken from https://github.com/google/oss-fuzz/actions/runs/9270497789/job/25503641673#step:7:16578)

libavfilter/libavfilter.a(avf_showcqt_init.o): in function `ff_showcqt_init_x86':
avf_showcqt_init.c:(.text.unlikely.ff_showcqt_init_x86[ff_showcqt_init_x86]+0x98): undefined reference to `ff_showcqt_cqt_calc_sse'
/usr/bin/ld: avf_showcqt_init.c:(.text.unlikely.ff_showcqt_init_x86[ff_showcqt_init_x86]+0x141): undefined reference to `ff_showcqt_cqt_calc_sse3'
/usr/bin/ld: avf_showcqt_init.c:(.text.unlikely.ff_showcqt_init_x86[ff_showcqt_init_x86]+0x214): undefined reference to `ff_showcqt_cqt_calc_fma4'
/usr/bin/ld: avf_showcqt_init.c:(.text.unlikely.ff_showcqt_init_x86[ff_showcqt_init_x86]+0x2c0): undefined reference to `ff_showcqt_cqt_calc_avx'
/usr/bin/ld: avf_showcqt_init.c:(.text.unlikely.ff_showcqt_init_x86[ff_showcqt_init_x86]+0x380): undefined reference to `ff_showcqt_cqt_calc_fma3'

Workaround:

Add --disable-inline-asm --disable-asm.

libc++ build fails to link the fuzzer

The current OSS-Fuzz build still fails in this pull request (https://github.com/google/oss-fuzz/actions/runs/9276126037/job/25522451321?pr=12007#step:7:24150):

LD  tools/target_dem_aa_fuzzer
/usr/bin/ld: /usr/bin/ld: DWARF error: invalid or unhandled FORM value: 0x25
libavutil/libavutil.a(cpu.o): in function `ff_get_cpu_flags_x86':
cpu.c:(.text.ff_get_cpu_flags_x86[ff_get_cpu_flags_x86]+0x1dd): undefined reference to `ff_cpu_cpuid'
/usr/bin/ld: cpu.c:(.text.ff_get_cpu_flags_x86[ff_get_cpu_flags_x86]+0x249): undefined reference to `ff_cpu_cpuid'
/usr/bin/ld: cpu.c:(.text.ff_get_cpu_flags_x86[ff_get_cpu_flags_x86]+0x4f4): undefined reference to `ff_cpu_xgetbv'
/usr/bin/ld: cpu.c:(.text.ff_get_cpu_flags_x86[ff_get_cpu_flags_x86]+0x5e5): undefined reference to `ff_cpu_cpuid'
/usr/bin/ld: cpu.c:(.text.ff_get_cpu_flags_x86[ff_get_cpu_flags_x86]+0x863): undefined reference to `ff_cpu_cpuid'
/usr/bin/ld: cpu.c:(.text.ff_get_cpu_flags_x86[ff_get_cpu_flags_x86]+0x8b5): undefined reference to `ff_cpu_cpuid'
/usr/bin/ld: /usr/bin/ld: DWARF error: invalid or unhandled FORM value: 0x25
libavutil/libavutil.a(imgutils_init.o): in function `ff_image_copy_plane_uc_from_x86':
imgutils_init.c:(.text.ff_image_copy_plane_uc_from_x86[ff_image_copy_plane_uc_from_x86]+0xc6): undefined reference to `ff_image_copy_plane_uc_from_sse4'
clang++: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [Makefile:62: tools/target_dem_aa_fuzzer] Error 1

I tried without libc++ outside of the OSS-Fuzz container and it worked, so as a next step one would have to debug inside the OSS-Fuzz container or recreate a libc++ build outside of it (which is tedious, as it requires building all ffmpeg dependencies with libc++).

michaelni commented 1 month ago

libavfilter/libavfilter.a(avf_showcqt_init.o): in function ff_showcqt_init_x86': avf_showcqt_init.c:(.text.unlikely.ff_showcqt_init_x86[ff_showcqt_init_x86]+0x98): undefined reference toff_showcqt_cqt_calc_sse' Workaround:

Add --disable-inline-asm --disable-asm.

you should disable asm IF thats what you intend. The fuzzer already should test with and without asm by switching at runtime. So disabling seems not correct.

The undefined reference error (and maybe some other errors too) seems to come from a wrong extern_prefix set by configure. I have not investigated this beyond that but the errors disappear if its commented out.

I dont need clang-16-18 with ossfuzz :) If you do, then please investigate and submit a clean patch to ffmpeg-devel. I assume some minor fix around the extern_prefix code could resolve this

thx

maflcko commented 3 weeks ago

I dont need clang-16-18 with ossfuzz :)

The motivation for this change is that for an OSS-Fuzz infra upgrade, all projects should be compiled with at least clang-18.

I took another look and it seems that the first error happens after clang enabled -fsanitize-address-use-odr-indicator by default. So the first build error is reproducible with any clang version, if -fsanitize-address-use-odr-indicator is set. And conversely, it does not happen when -fno-sanitize-address-use-odr-indicator is set.

Though, that still doesn't fix the second error.