tag 7.0 and master won't compile

[zoltak]

Building tag 7.0 or master results in the following error:

e.g. docker build --build-arg ENABLE_FDKAAC=1 . -t my-ffmpeg-static:latest

MAN doc/libavutil.3 MAN doc/libswscale.3 MAN doc/libswresample.3 MAN doc/libavcodec.3 MAN doc/libavformat.3 MAN doc/libavdevice.3 MAN doc/libavfilter.3 LD ffmpeg_g LD ffprobe_g collect2: fatal error: ld terminated with signal 9 [Killed] compilation terminated. make: *** [Makefile:136: ffmpeg_g] Error 1 make: *** Waiting for unfinished jobs.... The command '/bin/sh -c tar $TAR_OPTS ffmpeg.tar.bz2 && FDKAAC_FLAGS=$(if [[ -n "$ENABLE_FDKAAC" ]] ;then echo " --enable-libfdk-aac --enable-nonfree " ;else echo ""; fi) && cd ffmpeg-* && sed -i 's/add_ldexeflags -fPIE -pie/add_ldexeflags -fPIE -static-pie/' configure && ./configure --pkg-config-flags="--static" --extra-cflags="-fopenmp" --extra-ldflags="-fopenmp -Wl,-z,stack-size=2097152" --toolchain=hardened --disable-debug --disable-shared --disable-ffplay --enable-static --enable-gpl --enable-version3 $FDKAAC_FLAGS --enable-fontconfig --enable-gray --enable-iconv --enable-lcms2 --enable-libaom --enable-libaribb24 --enable-libass --enable-libbluray --enable-libdav1d --enable-libdavs2 --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libkvazaar --enable-libmodplug --enable-libmp3lame --enable-libmysofa --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-librabbitmq --enable-librav1e --enable-librtmp --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtheora --enable-libtwolame --enable-libuavs3d --enable-libvidstab --enable-libvmaf --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-openssl --enable-libjxl || (cat ffbuild/config.log ; false) && make -j$(nproc) install' returned a non-zero code: 2

[wader]

Hi, it seems like ld got killed collect2: fatal error: ld terminated with signal 9 [Killed]. Could it be that your running of of memory when linking and the OOM-killer starts killing things? if it's possible you can probably verify it by looking in dmesg on the docker host.

[danbo]

Mine won't compile either, but for a different reason:

...
=> ERROR [builder 130/132] RUN   EXPAT_VERSION=$(pkg-config --modversion expat)   FFTW_VERSION=$(pkg-config --modversion fftw3)   FONTCONFIG_VERSION=$(pkg-config --modversion fontcon  0.0s
...
Dockerfile:936
--------------------
 935 |     
 936 | >>> RUN \
 937 | >>>   EXPAT_VERSION=$(pkg-config --modversion expat) \
...
 1008 |     
--------------------
ERROR: failed to solve: failed to prepare 9xuybf77et3xm8whj83hws1k9 as r40c6z28tug8tevvj5tjkv7ke: max depth exceeded

Found this note about layers. Max layers appears to be 125.. but if you look at the builder log, it shows ERROR [builder 130/132] so it's looking like we are past that though not sure why it would error on 130 instead of 125.

In any case, when I collapsed the RUN echo commands into the the main commands, it reported 94 total layers in the builder, having shaved off 38, allowing the build to complete successfully.

 => [builder 92/94] RUN   EXPAT_VERSION=$(pkg-config --modversion expat)   FFTW_VERSION=$(pkg-config --modversion fftw3)   FONTCONFIG_VERSION=$(pkg-config --modversion fontconfig)      0.4s
 => [builder 93/94] COPY checkelf /                                                                                                                                                      0.1s
 => [builder 94/94] RUN   /checkelf /usr/local/bin/ffmpeg &&   /checkelf /usr/local/bin/ffprobe                                                                                          0.4s
 => [final1 1/9] COPY --from=builder /versions.json /usr/local/bin/ffmpeg /usr/local/bin/ffprobe /                                                                                       1.0s
 => [final1 2/9] COPY --from=builder /usr/local/share/doc/ffmpeg/* /doc/                                                                                                                 0.1s
 => [final1 3/9] COPY --from=builder /etc/ssl/cert.pem /etc/ssl/cert.pem                                                                                                                 0.0s
 => [final1 4/9] RUN ["/ffmpeg", "-version"]                                                                                                                                             0.3s
 => [final1 5/9] RUN ["/ffprobe", "-version"]                                                                                                                                            0.4s
 => [final1 6/9] RUN ["/ffmpeg", "-hide_banner", "-buildconf"]                                                                                                                           0.4s
 => [final1 7/9] RUN ["/ffmpeg", "-f", "lavfi", "-i", "testsrc", "-c:v", "libsvtav1", "-t", "100ms", "-f", "null", "-"]                                                                  0.7s
 => [final1 8/9] RUN ["/ffprobe", "-i", "https://github.com/favicon.ico"]                                                                                                                0.9s
 => [final1 9/9] RUN ["/ffprobe", "-tls_verify", "1", "-ca_file", "/etc/ssl/cert.pem", "-i", "https://github.com/favicon.ico"]                                                           0.9s
 => [final2 1/1] COPY --from=final1 / /                                                                                                                                                  1.2s
 => exporting to image                                                                                                                                                                   0.9s
 => => exporting layers                                                                                                                                                                  0.8s
 => => writing image sha256:8df070da74af2f540c0cc8cfb3ad69b787f5a91df0ffc3f84875e76454e28306
 => => naming to docker.io/library/my-static-ffmpeg-7

We could further collapse the commands, ie not just echo but also wget, in total this should bring the total number of layers down by about 75 in the builder.

Side note / something I'll also be investigating: https://github.com/goldmann/docker-squash

Docker version 26.0.2, build 3c863ff / Ubuntu / AMD64

[wader]

Hey, should be fixed by #434 the reason they were split up was to easy debugging when downloading or checksum fails, but i noticed that it's possible to make the sha sum thing more verbose and also exit with error code on failure.

I think this will hopefully save enough layers for new things for some years to come :)

[wader]

About squashing, i do this https://github.com/wader/static-ffmpeg/blob/master/Dockerfile#L1052-L1054 which results in the final image having just one layer, is that same as what docker-squash would do?

[zoltak]

Hi, it seems like ld got killed collect2: fatal error: ld terminated with signal 9 [Killed]. Could it be that your running of of memory when linking and the OOM-killer starts killing things? if it's possible you can probably verify it by looking in dmesg on the docker host.

you were spot on. Out of memory

~# dmesg [66187.115649] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/docker/cf5bd8414eabd23846cc43404f2ad8c6a1c2c4f0e9210ddd28f246366f92e2e6,task=ld,pid=17399,uid=0 [66187.115668] Out of memory: Killed process 17399 (ld) total-vm:1494112kB, anon-rss:1121652kB, file-rss:4kB, shmem-rss:0kB, UID:0 pgtables:2924kB oom_score_adj:0 [66189.442054] oom_reaper: reaped process 17399 (ld), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

[wader]

@zoltak 👍 interesting that ld runs out, think i've only seen rust using a lot and get OOM-killed:ed