Linking libav (ffmpeg) libraries in multilib x265-Asuna fails in MSYS2/MinGW64/GCC

[LigH-de]

I set up an MSYS2/MinGW environment using media-autobuild suite and compiled ffmpeg with a lot of included codecs.

Also I compiled and installed L-SMASH libraries in its interactive MinGW64 shell from the already existing sources, according to the locations in the suite:

cd /build/liblsmash-git/
./configure --prefix="/local64"
make install-lib

Then I retrieved a working directory of the Asuna branch:

git clone https://github.com/msg7086/x265-Yuuki-Asuna.git --branch Asuna --single-branch x265-Yuuki-Asuna-Asuna

I modified the multilib script provided by Multicoreware, based on its msys build samples. See makehdr10_w64_Asuna.sh in the attached ZIP archive.

The building process was successful for enabling support for AviSynth, VapourSynth, L-SMASH, MKV (although "deprecated"), ZIMG. Enabling support for LAVF also appears to find the required libraries. But the linker fails: Full console output of ccache'd buid in make_Asuna.txt in the attached ZIP archive

[ 96%] Linking CXX executable x265.exe
E:/MABS/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: CMakeFiles/cli.dir/objects.a(lavf.cpp.obj):lavf.cpp:(.text+0x11): undefined reference to `avcodec_free_context'
E:/MABS/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: CMakeFiles/cli.dir/objects.a(lavf.cpp.obj):lavf.cpp:(.text+0x1a): undefined reference to `avformat_close_input'
E:/MABS/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: CMakeFiles/cli.dir/objects.a(lavf.cpp.obj):lavf.cpp:(.text+0x19d): undefined reference to `av_init_packet'
E:/MABS/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: CMakeFiles/cli.dir/objects.a(lavf.cpp.obj):lavf.cpp:(.text+0x1c0): undefined reference to `avcodec_receive_frame'
E:/MABS/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: CMakeFiles/cli.dir/objects.a(lavf.cpp.obj):lavf.cpp:(.text+0x1d8): undefined reference to `av_read_frame'
E:/MABS/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: CMakeFiles/cli.dir/objects.a(lavf.cpp.obj):lavf.cpp:(.text+0x1f4): undefined reference to `av_packet_unref'
E:/MABS/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: CMakeFiles/cli.dir/objects.a(lavf.cpp.obj):lavf.cpp:(.text+0x204): undefined reference to `av_init_packet'
E:/MABS/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: CMakeFiles/cli.dir/objects.a(lavf.cpp.obj):lavf.cpp:(.text+0x226): undefined reference to `avcodec_send_packet'
E:/MABS/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: CMakeFiles/cli.dir/objects.a(lavf.cpp.obj):lavf.cpp:(.text+0x23d): undefined reference to `avcodec_receive_frame'
E:/MABS/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: CMakeFiles/cli.dir/objects.a(lavf.cpp.obj):lavf.cpp:(.text+0x2a4): undefined reference to `av_packet_unref'
E:/MABS/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: CMakeFiles/cli.dir/objects.a(lavf.cpp.obj):lavf.cpp:(.text+0x2b8): undefined reference to `av_packet_unref'
E:/MABS/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: CMakeFiles/cli.dir/objects.a(lavf.cpp.obj):lavf.cpp:(.text+0x4aa): undefined reference to `avcodec_find_decoder'
E:/MABS/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: CMakeFiles/cli.dir/objects.a(lavf.cpp.obj):lavf.cpp:(.text+0x4ba): undefined reference to `avcodec_alloc_context3'
E:/MABS/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: CMakeFiles/cli.dir/objects.a(lavf.cpp.obj):lavf.cpp:(.text+0x4d3): undefined reference to `avcodec_parameters_to_context'
E:/MABS/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: CMakeFiles/cli.dir/objects.a(lavf.cpp.obj):lavf.cpp:(.text+0x4e7): undefined reference to `avcodec_open2'
E:/MABS/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: CMakeFiles/cli.dir/objects.a(lavf.cpp.obj):lavf.cpp:(.text+0x5ad): undefined reference to `av_packet_unref'
E:/MABS/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: CMakeFiles/cli.dir/objects.a(lavf.cpp.obj):lavf.cpp:(.text+0x60c): undefined reference to `av_frame_alloc'
E:/MABS/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: CMakeFiles/cli.dir/objects.a(lavf.cpp.obj):lavf.cpp:(.text+0x62e): undefined reference to `avformat_open_input'
E:/MABS/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: CMakeFiles/cli.dir/objects.a(lavf.cpp.obj):lavf.cpp:(.text+0x64b): undefined reference to `avformat_find_stream_info'
E:/MABS/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: CMakeFiles/cli.dir/objects.a(lavf.cpp.obj):lavf.cpp:(.text+0x6c9): undefined reference to `avcodec_find_decoder'
E:/MABS/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: CMakeFiles/cli.dir/objects.a(lavf.cpp.obj):lavf.cpp:(.text+0x6d8): undefined reference to `avcodec_alloc_context3'
E:/MABS/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: CMakeFiles/cli.dir/objects.a(lavf.cpp.obj):lavf.cpp:(.text+0x6f4): undefined reference to `avcodec_parameters_to_context'
E:/MABS/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: CMakeFiles/cli.dir/objects.a(lavf.cpp.obj):lavf.cpp:(.text+0x707): undefined reference to `avcodec_open2'
E:/MABS/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: CMakeFiles/cli.dir/objects.a(lavf.cpp.obj):lavf.cpp:(.text+0x72c): undefined reference to `av_dict_set'
E:/MABS/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: CMakeFiles/cli.dir/objects.a(lavf.cpp.obj):lavf.cpp:(.text+0x73f): undefined reference to `avcodec_open2'
E:/MABS/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: CMakeFiles/cli.dir/objects.a(lavf.cpp.obj):lavf.cpp:(.text+0x761): undefined reference to `av_dict_free'
E:/MABS/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: CMakeFiles/cli.dir/objects.a(lavf.cpp.obj):lavf.cpp:(.text+0x82b): undefined reference to `av_pix_fmt_desc_get'
E:/MABS/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: CMakeFiles/cli.dir/objects.a(lavf.cpp.obj):lavf.cpp:(.text+0x2c): undefined reference to `av_frame_free'
collect2.exe: error: ld returned 1 exit status

Compiling x264 with a reduced ffmpeg (video decoders only) works in m-ab-s. But it uses its own local libraries in this case. The ffmpeg libraries installed in the system should be taken from the full compliation of ffmpeg,

If you need any more detail, please request.

Log, script, pkgconfig/libav*.pc and CMake files archive

[LigH-de]

Maybe @1480c1 can add an idea?

[1480c1]

Try setting the PKG_CONFIG environment variable with export PKG_CONFIG=$LOCALDESTDIR/bin/ab-pkg-config-static.bat. cmake doesn't handle multi-worded PKG_CONFIG variables nicely, so it breaks on mabs' default PKG_CONFIG and can't find the proper flags and libs is what I assume.

[LigH-de]

Almost...

-- Could NOT find PkgConfig (missing: PKG_CONFIG_EXECUTABLE) 
    Reason given by package: The command
      "E:/MABS/msys64/mingw64/bin/pkgconf --keep-system-libs --static" --version
    failed with output:

    stderr: 

    result: 
Das System kann die angegebene Datei nicht finden {{File not found}}

full make log

[1480c1]

seems like cmake didn't pickup ab-pkg-config-static.bat, going to see if I can repro locally once I have time

[LigH-de]

Ooooh - now that you mention it... I may have to purge the build directories to force cmake to create new scripts.

[LigH-de]

No, the cmake error messages are different now.

-- x265 version 3.4+-+54
-- Checking for module 'libavformat'
--   Package 'libavformat', required by 'virtual:world', not found
-- avformat: include F:/MABS/local64/include lib F:/MABS/local64/lib/libavformat.a
-- Checking for module 'libavcodec'
--   Package 'libavcodec', required by 'virtual:world', not found
-- avcodec: include F:/MABS/local64/include lib F:/MABS/local64/lib/libavcodec.a
-- Checking for module 'libavutil'
--   Package 'libavutil', required by 'virtual:world', not found
-- avutil: include F:/MABS/local64/include lib F:/MABS/local64/lib/libavutil.a
-- Checking for module 'libswresample'
--   Package 'libswresample', required by 'virtual:world', not found
-- swresample: include F:/MABS/local64/include lib F:/MABS/local64/lib/libswresample.a

Same linker errors.

[1480c1]

just making sure, if you run pkgconf --keep-system-libs --static --libs libavformat, does it return anything?

[LigH-de]

Yes.

$ pkgconf --keep-system-libs --static --libs libavformat
-LF:/MABS/local64/lib -lavformat -lbz2 -lgme -lmodplug -lopenmpt -lrpcrt4 -lchromaprint -lbluray -lxml2 -lmbedtls -lmbedx509 -lmbedcrypto -lrtmp -lwinmm -lgdi32 -lgnutls -latomic -lcrypt32 -ltasn1 -lhogweed -lgmp -lnettle -lzmq -liphlpapi -lsodium -lavcodec -lvpx -lwebpmux -lm -llzma -ldav1d -lopencore-amrwb -lzvbi -liconv -lpng -lsnappy -lz -laom -lcodec2 -lgsm -lilbc -lmp3lame -lopencore-amrnb -lopenjp2 -lopus -lrav1e -lws2_32 -luserenv -lshine -lspeex -ltheoraenc -ltheoradec -ltwolame -lvo-amrwbenc -lvorbisenc -lvorbis -logg -lwebp -lx264 -lx265 -lxavs -lm -lxvidcore -lkvazaar -lm -lswresample -lsoxr -lavutil -pthread -lm -lmfx -lole32 -luuid -lOpenCL -lvulkan -lstdc++ -lpthread -ladvapi32 -lshell32 -luser32 -lmingwthrd -lmingw32 -lgcc -lmoldname -lmingwex -lkernel32 -lcfgmgr32 -lbcrypt -ldl

[1480c1]

https://github.com/msg7086/x265-Yuuki-Asuna/blob/Asuna/source/cmake/FindFF.cmake#L16 oh

[1480c1]

@LigH-de can you try patching this?

diff --git a/source/cmake/FindFF.cmake b/source/cmake/FindFF.cmake
index 01d571965..5230ac191 100644
--- a/source/cmake/FindFF.cmake
+++ b/source/cmake/FindFF.cmake
@@ -13,16 +13,16 @@ find_package(PkgConfig)

 MACRO(FFMPEG_FIND varname shortname headername)

-  IF(NOT WIN32)
+  #IF(NOT WIN32)
     PKG_CHECK_MODULES(PC_${varname} "lib${shortname}")

     FIND_PATH(${varname}_INCLUDE_DIR "lib${shortname}/${headername}"
       HINTS ${PC_${varname}_INCLUDEDIR} ${PC_${varname}_INCLUDE_DIRS}
       NO_DEFAULT_PATH
       )
-  ELSE()
-    FIND_PATH(${varname}_INCLUDE_DIR "lib${shortname}/${headername}")
-  ENDIF()
+  #ELSE()
+    #FIND_PATH(${varname}_INCLUDE_DIR "lib${shortname}/${headername}")
+  #ENDIF()

   IF(${${varname}_INCLUDE_DIR} STREQUAL "${varname}_INCLUDE_DIR-NOTFOUND")
@@ -32,10 +32,10 @@ MACRO(FFMPEG_FIND varname shortname headername)

 #   GET_DIRECTORY_PROPERTY(FFMPEG_PARENT DIRECTORY ${${varname}_INCLUDE_DIR} PARENT_DIRECTORY)
     GET_FILENAME_COMPONENT(FFMPEG_PARENT ${${varname}_INCLUDE_DIR} PATH)
-    IF(WIN32)
-      SET(ENV{PKG_CONFIG_PATH} "${CMAKE_PREFIX_PATH}/lib/pkgconfig/")
-      PKG_CHECK_MODULES(PC_${varname} lib${shortname})
-    ENDIF()
+    #IF(WIN32)
+   #   SET(ENV{PKG_CONFIG_PATH} "${CMAKE_PREFIX_PATH}/lib/pkgconfig/")
+    #  PKG_CHECK_MODULES(PC_${varname} lib${shortname})
+    #ENDIF()
 #    MESSAGE(STATUS "Using FFMpeg dir parent as hint: ${FFMPEG_PARENT}")

     # NO_SYSTEM_ENVIRONMENT_PATH prevent accidently linking to dlls in $PATH

[LigH-de]

Maybe these exceptions shall rather mean "not for MSVC" than "not for any Windows compiler"?...

Testing...

Without the ab-pkg-config-static.bat export: No, that did not find FFMPEG. Trying again with it: Looks better...

-- Found PkgConfig: E:/MABS/local64/bin/ab-pkg-config-static.bat (found version "1.7.3") 
-- Checking for module 'libavformat'
--   Found libavformat, version 58.65.101
-- avformat: include E:/MABS/local64/include lib E:/MABS/local64/lib/libavformat.a
-- Checking for module 'libavcodec'
--   Found libavcodec, version 58.117.101
-- avcodec: include E:/MABS/local64/include lib E:/MABS/local64/lib/libavcodec.a
-- Checking for module 'libavutil'
--   Found libavutil, version 56.63.101
-- avutil: include E:/MABS/local64/include lib E:/MABS/local64/lib/libavutil.a
-- Checking for module 'libswresample'
--   Found libswresample, version 3.8.100
-- swresample: include E:/MABS/local64/include lib E:/MABS/local64/lib/libswresample.a

We just may need to limit the selection of codecs in libavcodec to be linked into x265 to video decoders only (means, may require building an own "ffmpeg light", similar to x264).

There is an x265 API mismatch between the older x265 code base of the Asuna branch and the x265 encoder in a most recent ffmpeg:

[ 96%] Linking CXX executable x265.exe
E:/MABS/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: E:/MABS/local64/lib/libavcodec.a(libx265.o):libx265.c:(.text.unlikely+0x339): undefined reference to `x265_api_get_198'
E:/MABS/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: E:/MABS/local64/lib/libavcodec.a(libx265.o):libx265.c:(.text.unlikely+0x355): undefined reference to `x265_api_get_198'
E:/MABS/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: E:/MABS/local64/lib/libavcodec.a(libx265.o):libx265.c:(.text.unlikely+0x371): undefined reference to `x265_api_get_198'
E:/MABS/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: E:/MABS/local64/lib/libavcodec.a(libx265.o):libx265.c:(.text.unlikely+0x4db): undefined reference to `x265_api_get_198'
E:/MABS/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: E:/MABS/local64/lib/libavcodec.a(libx265.o):libx265.c:(.text.unlikely+0x4eb): undefined reference to `x265_api_get_198'
collect2.exe: error: ld returned 1 exit status

:rage1: Without ccache, due to purging the build directory, NASM drops 9 MB of legacy macro parameter warnings...

logs of first run without ccache (zipped) logs of second run with ccache (plain)

[1480c1]

haha, due to nasm, I couldn't scroll back up my console buffer

[1480c1]

Maybe try a selective build and remove x265?

[LigH-de]

I have made fire! :fire:

Added the following export to my build script, to have the "video decoders only" light ffmpeg build for x264 (previously created by m-ab-s in configuration x2643=6) preferred in the package configuration:

export PKG_CONFIG_PATH=$LOCALDESTDIR/opt/lightffmpeg/lib/pkgconfig:$LOCALDESTDIR/lib/pkgconfig

Now it seems to be complete (except for the strange version number).

E:\MABS\x265-Yuuki-Asuna-Asuna\release\Win64>x265.exe -V
x265 [info]: HEVC encoder version 3.4+-+54
x265 [info]: build info [Windows][GCC 10.2.0][64 bit] Asuna 8bit+10bit+12bit
x265 [info]: (lsmash 2.16.1)
x265 [info]: (libavformat 58.65.101)
x265 [info]: (libavcodec  58.117.101)
x265 [info]: (libavutil   56.63.101)
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX XOP FMA4 FMA3 BMI1

Quick test was successful with Y4M, AVS and MP4 inputs, and MP4 output.

[msg7086]

3.4+-+54 means the remote repo is not named as origin.

Can you please try replacing source/version.rb with the following content?

light = ARGV.first == 'light'

nickname = `git rev-parse --abbrev-ref HEAD`.strip
vbase, vhead, vhcommit = `git describe --tags --first-parent HEAD`.strip.split('-')
branch = nickname == 'Yuuki' ? 'stable' : 'old-stable'
remote_branch = `git branch -r`[%r((\S+/#{branch})), 1]
_, vmaster, vmcommit = `git describe --tags --first-parent #{remote_branch}`.strip.split('-')

version = ''
tag = vbase.delete('M')

if vhcommit == vmcommit
  version = "#{vbase}+#{vmaster}-#{vmcommit}"
else
  vdiff = vhead.to_i - vmaster.to_i
  version = "#{vbase}+#{vmaster}-#{vmcommit}+#{vdiff}"
end
puts light ? tag : version

[LigH-de]

Looks good:

x265 [info]: HEVC encoder version 3.4+54-gf9c3b2805

[msg7086]

That ... looks wrong. How does it check out code? Correct version number relies on knowing the distance between base branch and current branch, so it needs to know how far old-stable is behind Asuna. The correct version is 3.4+13-g729a838d3+41.

On the other hand though, if you are happy with this version number then it's not a big deal.

[LigH-de]

Well, I can hold back to get it sorted. Your example looks indeed much more specific.

I used git commands to clone and update this branch a few days ago, within the era of the last commit.

[msg7086]

If you run git branch -r, do you see origin/old-stable or similar in the list?

If you run git describe --tags --first-parent origin/old-stable, is it 3.4-13-g729a838d3?

[LigH-de]

I checked it out using

git clone https://github.com/msg7086/x265-Yuuki-Asuna.git --branch Asuna --single-branch x265-Yuuki-Asuna-Asuna

so it returns:

$ git branch -r
  origin/Asuna
$ git describe --tags --first-parent origin/old-stable
fatal: Not a valid object name origin/old-stable

The --single-branch might be the culprit?

[msg7086]

Please try this then check version.

git remote set-branches --add origin old-stable
git fetch

Or you can remove --single-branch next time.

[LigH-de]

Now this:

$ git describe --tags --first-parent origin/old-stable
3.4-13-g729a838d3

[msg7086]

That looks correct. You should get the correct version in x265.

[LigH-de]

For the records: Building for Win32 may need CXXFLAGS=-fpermissive as workaround or it may stop (at VapourSynth's frameDoneCallback, I believe).

[DJATOM]

Probably it's just enough to add proper macross for gcc/clang.