search

m-ab-s / media-autobuild_suite

This Windows Batchscript helps setup a Mingw-w64 compiler environment for building ffmpeg and other media tools under Windows.
GNU General Public License v3.0
1.52k stars 263 forks source link

Compile of modules fail: Version mismatch error of aclocal.m4 based on wrong libtool #57

Closed MacharaStormwing closed 9 years ago

MacharaStormwing commented 9 years ago

Hi.

I ran your build build script and oveall it worked surprisingly well, but there were a few hickups that made the whole compile not work.

The first two are about what I mentioned in the title: Both libgnurx-2.5.1 and mediainfo (based on mediainfo_0.7.72_AllInclusive.7z) failed because of this error.

libtool: Version mismatch error. This is libtool 2.4.6, but the libtool: definition of this LT_INIT comes from libtool 2.4.5. libtool: You should recreate aclocal.m4 with macros from libtool 2.4.6 libtool: and run autoconf again.

More specific error lines below that have been (in case of libgnurx):

Makefile:456: recipe for target 'libgnurx_la-regex.lo' failed make: *\ [libgnurx_la-regex.lo] Error 63

and (in case of mediainfo)

Makefile:526: recipe for target 'Conf.lo' failed make: *\ [Conf.lo] Error 63

How to fix an issue like this (myself, if possible??

Also later in the process I get this:

Saving to: ‘FFmbc-0.7.2.tar.bz2’ FFmbc-0.7.2.tar.bz2 [ <=> ] 4.33M 2.35MB/s in 1.8s 2015-02-19 08:24:14 (2.35 MB/s) - ‘FFmbc-0.7.2.tar.bz2’ saved [4544811] ERROR: libfaac not found

ffmpeg will then not build because libfaac was not built, even though I did not actually notice an issue with libfaac. Are there log-files of the whole build-process kept to check up on?

Also mplayer-svn compile failed. The last lines where:

Makefile:731: recipe for target 'libmpcodecs/ad_spdif.o' failed make: *\ [libmpcodecs/ad_spdif.o] Error 1

Then mpg-git failed:

Unable to find development files for some of the required Libav libraries (libavutil >= 52.48.101 libavcodec >= 55.34.1 libavformat >= 55.12.0 libswscale >= 2.1.2). Aborting. sed: can't read ./build/c4che/_cache.py: No such file or directory sed: can't read ./build/c4che/_cache.py: No such file or directory sed: can't read ./build/c4che/_cache.py: No such file or directory The project was not configured: run "waf configure" first! The project was not configured: run "waf configure" first! mkdir: cannot create directory ‘/local32/bin-video/mpv/lua’: No such file or directory cp: target ‘/local32/bin-video/mpv/lua’ is not a directory

mkvtoolnix-git failed:

LINK src/mkvmerge.exe

C:/msys/msys64/mingw32/bin/../lib/gcc/i686-w64-mingw32/4.9.2/../../../../i686-w64-mingw32/bin/ld.exe: cannot find -lgnurx collect2.exe: error: ld returned 1 exit status LINK src/mkvmerge.exe C:/msys/msys64/mingw32/bin/../lib/gcc/i686-w64-mingw32/4.9.2/../../../../i686-w64-mingw32/bin/ld.exe: cannot find -lgnurx collect2.exe: error: ld returned 1 exit status mv: cannot stat ‘/local32/bin-video/mkvtoolnix/share/locale’: No such file or directory mv: cannot stat ‘/local32/bin-video/mkvtoolnix/share/doc/mkvtoolnix/guide’: No such file or directory

When it comes to taking hand yourself (even if I built ffmpeg and dependencies completely manually before), I still have somewhat of an issue: When I open the shell myself using "mintty" in the root folder msys in the entire availble file system I do not find the files media-suite_compile.sh or media-suite_update.sh.

How to best open the shell at the right place? I'm currently trying to reconstruct it from your .bat script, but there must be an easier way...

A more detailed wiki page to help novice users of your tool-set and msys2 help themselves would be appreciated!

MaPePeR commented 9 years ago

I think i found the first issue with libgnurx: The scripts downloads the Makefile.am and configure.ac from 2 different places, so they don't match anymore. I tried to download the source without overwriting this 2 files, but it did not work

Which makes no sense, because both files should not have changed. I think the autobuild suite tries to emulate this script from mingw.

jb-alvarado commented 9 years ago

Hi DragonMorkeleb, thank you for the report. Now the most things works again. There was a mismatch in msys2 and mingw-w64 the the developer from them have fix that. It can be that mplayer still not work, until now I had no time to look at it. When you had download the zip file from here then the media-suite_compile.sh or media-suite_update.sh have to be also in the root folder directly there where the batch file is and the mintty shortcut. For manual compiling the easiest is to open the mintty shortcut.

jb-alvarado commented 9 years ago

MaPePeR, libgnurx will compile now fine, because msys2 get an update :).

mediainfo runs here now also, so I guess ffmpeg and mpv will work to. With mplayer I am not sure.

MaPePeR commented 9 years ago

Worked like a charm for me again.

You and this project are amazing! I can't even estimate how much time it already saved me.

MacharaStormwing commented 9 years ago

It works for me now, too.

Mplayer still fails, though, see the error below,

Is there any chance to get a gcc 5 compiler set up and compile it with that? From what I've read, it can improve performance of some encoding like x264 around 3 percent (and peformance on 4.9.2 is down a percent compared to 4.8.3).

gcc -MMD -MP -Wundef -Wstrict-prototypes -Wmissing-prototypes -Wdisabled-optimization -Wno-pointer-sign -Wdeclaration-after-statement -std=gnu99 -D_ISOC99_SOURCE -I. -Iffmpeg -I/local64/include -mms-bitfields -mthreads -mtune=generic -pipe -fno-tree-vectorize -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -DPTW32_STATIC_LIB -O3 -std=gnu99 -DLIBTWOLAME_STATIC -DMODPLUG_STATIC -fno-common -DFRIBIDI_ENTRY="" -DPIC -I/local64/include -I/mingw64/include -I/local64/include -I/local64/include/freetype2 -I/local64/include -I/mingw64/include -I/local64/include/freetype2 -I/mingw64/include -I/mingw64/include/libpng16 -I/local64/include/freetype2 -I/mingw64/include -I/mingw64/include/libpng16 -DZLIB_CONST -I/local64/include/opus -I/local64/include -I/mingw64/include -I/mingw64/include/schroedinger-1.0 -I/mingw64/include/orc-0.4 -I/local64/include -c -o libmpcodecs/ad_spdif.o libmpcodecs/ad_spdif.c In file included from C:/msys/msys64/mingw64/x86_64-w64-mingw32/include/windows.h:71:0, from C:/msys/msys64/mingw64/x86_64-w64-mingw32/include/winsock2.h:23, from ffmpeg/libavformat/os_support.h:112, from ffmpeg/libavformat/internal.h:26, from libmpcodecs/ad_spdif.c:26: C:/msys/msys64/mingw64/x86_64-w64-mingw32/include/wingdi.h:603:5: error: conflicting types for 'BITMAPINFOHEADER' } BITMAPINFOHEADER,_LPBITMAPINFOHEADER,_PBITMAPINFOHEADER; ^ In file included from ./libmpdemux/stheader.h:24:0, from libmpcodecs/ad_internal.h:27, from libmpcodecs/ad_spdif.c:23: ./libmpdemux/ms_hdr.h:76:3: note: previous declaration of 'BITMAPINFOHEADER' was here } BITMAPINFOHEADER, _PBITMAPINFOHEADER, LPBITMAPINFOHEADER; ^ In file included from C:/msys/msys64/mingw64/x86_64-w64-mingw32/include/windows.h:71:0, from C:/msys/msys64/mingw64/x86_64-w64-mingw32/include/winsock2.h:23, from ffmpeg/libavformat/os_support.h:112, from ffmpeg/libavformat/internal.h:26, from libmpcodecs/ad_spdif.c:26: C:/msys/msys64/mingw64/x86_64-w64-mingw32/include/wingdi.h:603:23: error: conflicting types for 'LPBITMAPINFOHEADER' } BITMAPINFOHEADER,_LPBITMAPINFOHEADER,_PBITMAPINFOHEADER; ^ In file included from ./libmpdemux/stheader.h:24:0, from libmpcodecs/ad_internal.h:27, from libmpcodecs/ad_spdif.c:23: ./libmpdemux/ms_hdr.h:76:42: note: previous declaration of 'LPBITMAPINFOHEADER' was here } BITMAPINFOHEADER, PBITMAPINFOHEADER, LPBITMAPINFOHEADER; ^ In file included from C:/msys/msys64/mingw64/x86_64-w64-mingw32/include/windows.h:71:0, from C:/msys/msys64/mingw64/x86_64-w64-mingw32/include/winsock2.h:23, from ffmpeg/libavformat/os_support.h:112, from ffmpeg/libavformat/internal.h:26, from libmpcodecs/ad_spdif.c:26: C:/msys/msys64/mingw64/x86_64-w64-mingw32/include/wingdi.h:603:43: error: conflicting types for 'PBITMAPINFOHEADER' } BITMAPINFOHEADER,_LPBITMAPINFOHEADER,_PBITMAPINFOHEADER; ^ In file included from ./libmpdemux/stheader.h:24:0, from libmpcodecs/ad_internal.h:27, from libmpcodecs/ad_spdif.c:23: ./libmpdemux/ms_hdr.h:76:22: note: previous declaration of 'PBITMAPINFOHEADER' was here } BITMAPINFOHEADER, PBITMAPINFOHEADER, LPBITMAPINFOHEADER; ^ In file included from C:/msys/msys64/mingw64/x86_64-w64-mingw32/include/windows.h:71:0, from C:/msys/msys64/mingw64/x86_64-w64-mingw32/include/winsock2.h:23, from ffmpeg/libavformat/os_support.h:112, from ffmpeg/libavformat/internal.h:26, from libmpcodecs/ad_spdif.c:26: C:/msys/msys64/mingw64/x86_64-w64-mingw32/include/wingdi.h:672:5: error: conflicting types for 'BITMAPINFO' } BITMAPINFO,_LPBITMAPINFO,_PBITMAPINFO; ^ In file included from ./libmpdemux/stheader.h:24:0, from libmpcodecs/ad_internal.h:27, from libmpcodecs/ad_spdif.c:23: ./libmpdemux/ms_hdr.h:80:3: note: previous declaration of 'BITMAPINFO' was here } BITMAPINFO, LPBITMAPINFO; ^ In file included from C:/msys/msys64/mingw64/x86_64-w64-mingw32/include/windows.h:71:0, from C:/msys/msys64/mingw64/x86_64-w64-mingw32/include/winsock2.h:23, from ffmpeg/libavformat/os_support.h:112, from ffmpeg/libavformat/internal.h:26, from libmpcodecs/ad_spdif.c:26: C:/msys/msys64/mingw64/x86_64-w64-mingw32/include/wingdi.h:672:17: error: conflicting types for 'LPBITMAPINFO' } BITMAPINFO,_LPBITMAPINFO,PBITMAPINFO; ^ In file included from ./libmpdemux/stheader.h:24:0, from libmpcodecs/ad_internal.h:27, from libmpcodecs/ad_spdif.c:23: ./libmpdemux/ms_hdr.h:80:16: note: previous declaration of 'LPBITMAPINFO' was here } BITMAPINFO, LPBITMAPINFO; ^ Makefile:731: recipe for target 'libmpcodecs/ad_spdif.o' failed

make: *\ [libmpcodecs/ad_spdif.o] Error 1

Build mplayer-svn failed...

2015-02-25 13:41 GMT+01:00 MaPePeR notifications@github.com:

Worked like a charm for me again.

You and this project are amazing! I can't even estimate how much time it already saved me.

— Reply to this email directly or view it on GitHub https://github.com/jb-alvarado/media-autobuild_suite/issues/57#issuecomment-75953281 .

jb-alvarado commented 9 years ago

Yes I know, this problem with player still exist. Something in the Source Code from mplayer works not together with the actual mingw-w64 version.

mabreedlove commented 9 years ago

It's possible to build gcc within msys2 but the packaging system will generally just clobber whatever it is you have configured so it has to be built outside of the package management system which simply isn't worth it, to be honest. That 5% is more easily gained by using x264's own PGO system along with LTO and doesn't come with the same set of headaches.

To do this, get at least 1 or several .y4m files using something like ffmpeg:

ffmpeg -i <movie> -map 0:v out.y4m

You can then copy the files to the x264 build directory or wherever and using an already configured x264 build directory (doesn't need to be clean) run:

make VIDS="/path/to/video1.y4m /path/to/video2.y4m..." fprofiled

It will then perform a batch PGO build of itself instead of the default. Make sure you don't use too long of video clips or it will take forever and y4m files are uncompressed so are quite large. I'm still in the process of trying to figure out how to implement this and LTO within the package management system as an optional feature given that profiling is tied to each program so additional builds of dependent libraries would have to be done for each program (ffmpeg, mplayer, etc) though LTO doesn't have that issue. As most libraries come with some form of demos or testing programs, or are applications without dependencies (x264, x265 for instance), it's fairly easy to build and profile each individually. The below snippet is an example of an opus build taken from my own migrated scripts with some small modifications to allow pasting into a mingw64 terminal window without polluting the environment:

cd /build64/opus-1.1
_config=("CPPFLAGS=-D_FORTIFY_SOURCE=2 -D__USE_MINGW_ANSI_STDIO=1" "AR=gcc-ar" "NM=gcc-nm" "RANLIB=gcc-ranlib")
_config+=("--prefix=/local64" "--bindir=/local64/bin-audio" "--build=x86_64-w64-mingw32" "--host=x86_64-w64-mingw32")
_config+=("--enable-custom-modes" "--disable-shared")

./configure "${_config[@]}" \
    CFLAGS="-O2 -march=native -mfpmath=sse -flto -ffat-lto-objects -flto-partition=1to1 -fprofile-generate -pipe" \
    LDFLAGS="-fuse-linker-plugin -fprofile-generate -pipe"
make -j$(nproc)
make -j1 check

curl -O http://opus-codec.org/testvectors/opus_testvectors.tar.gz
tar -zxf opus_testvectors.tar.gz
./tests/run_vectors.sh ./ opus_testvectors 48000

make distclean

./configure "${_config[@]}" \
    CFLAGS="-O2 -march=native -mfpmath=sse -flto -ffat-lto-objects -flto-partition=1to1 -fprofile-use -pipe" \
    LDFLAGS="-fuse-linker-plugin -fprofile-use -pipe"
make -j$(nproc)
make check && make install

The following flags have been reliably tested to increase performance across the board as well:

-O2 -mfpmath=sse -fomit-frame-pointer -fschedule-insns -fsched-pressure -fira-loop-pressure -fgcse-sm -fgraphite-identity

Using -O3 improves performance in most instances (bzip2, zlib, x264, x265, ffmpeg, etc) but reduces performance on highly optimized libraries such as gmp. Besides "-fomit-frame-pointer" which reduces ability to debug, these flags successfully compile the Mingw64 CRT and winpthreads as well as GCC with all its dependencies in addition to the applications here. Most come from Intel and GCC optimization references which I then verified myself using hardware-profiling. I also use some additional param settings though their benefits are much less noticeable.

I usually compile programs for my own uses with performance in mind, so compiling with "-march=native -mfpmath=sse" or even "-march=x86-64 -mtune=native -mfpmath=sse" isn't an issue. One thing to note is that all x86-64 processors come with SSE support though early pentium4's only had single precision which is why GCC doesn't have "-mfpmath=sse" enabled by default. Unless you need to worry about this rare case, building with "-mfpmath=sse" is a large performance increase. This can and should be used when building for i686 if able but will otherwise simply be ignored by GCC if not supported by the target arch.

As for mplayer, try putting the following at the top of libmpdemux/ms_hdr.h:

#ifdef defined(__MINGW32__)
#include <windows.h>
#endif

The definition within mplayer uses signed ints within the BITMAPINFO struct despite Microsoft's reference using DWORDs for most fields so there is an obvious conflict anytime the mingw windows.h file doesn't get included before this file does.

MacharaStormwing commented 9 years ago

Thanx for the tips.

I tried with these options, now:

-mtune=native -pipe -Ofast -mfpmath=sse -fomit-frame-pointer -fschedule-insns -fsched-pressure -fira-loop-pressure -fgcse-sm -fgraphite-identity -march=core-avx2

When it comes to performance it is the overall performance in x264 or x265 encoding (with fdk_aac) that I care about, so I hope -Ofast is appropriate?

I noticed that the current source of x265 produces results with noticably higher bitrate as opposed to older versions that may be considered more stable. Is it possible to state a specific svn or git version to be used, or a specific release of a component to be used? If not how would I best go about it manually? (I originally though that vlc did not play some of my x265 encodes properly, but then found it played properly on one pc and with artefacts on another. It turns out if the machine is too slow (and that does not have to mean it maxes out the cpu, strangely) it apparently tries to compensate and that causes artefacts. Any ideas why? But I understand this is outside the topic of this discussion, so it's ok to ignore.)

Can you explain a bit more how this profiling with y4m file works? What does it to? Is once a profile is created, does it work for any video encoding or is it then optimized to certain kinds of videos (anime versus film, etc)? You mentioned multiple y4m files - what does more than one help, and with how many do I get best performance? Can you provide more step-by-step instructions on how to make use of this optimization?

MacharaStormwing commented 9 years ago

I just tried another compile-run and this time mvf-git failed:

[230/232] Compiling misc/bstr.c [231/232] Linking build/mpv.com [232/232] Linking build/mpv.exe Waf: Leaving directory /build64/mpv-git/build' 'build' finished successfully (14.072s) Waf: Entering directory/build64/mpv-git/build'

Waf: Leaving directory `/build64/mpv-git/build' Build failed Traceback (most recent call last): File "/build64/mpv-git/.waf-1.8.4-6442f4de997a9be97a56ebfe12e49ac5/waflib/Task.py", line 107, in process ret=self.run() File "/build64/mpv-git/.waf-1.8.4-6442f4de997a9be97a56ebfe12e49ac5/waflib/Build.py", line 457, in run return self.generator.exec_task() File "/build64/mpv-git/.waf-1.8.4-6442f4de997a9be97a56ebfe12e49ac5/waflib/Build.py", line 473, in exec_install_files self.generator.bld.do_install(y.abspath(),destfile,chmod=self.chmod,tsk=self) File "/build64/mpv-git/.waf-1.8.4-6442f4de997a9be97a56ebfe12e49ac5/waflib/Build.py", line 529, in do_install raise Errors.WafError('Could not install the file %r'%tgt) WafError: Could not install the file '/local64/bin-video/mpv/bin/mpv.exe'

Build mpv-git failed...

Might that have something to do with that my virus scanner recently detected mpv.exe as virus? Any idea how to avoid this?

mabreedlove commented 9 years ago

-Ofast enables all of -O3 as well as -ffast-math which will break some applications (the most notable that I've recognized being Opus which will fail its checks and the test binaries used for it will segfault). I would avoid using it and instead use -O3. Applications that can take advantage of -ffast-math safely will generally enable it regardless (x264 and/or x265 does this by default). Also, if you're targetting for core-avx2 and also running it natively, then specifying the -mtune is unnecessary. The -mtune option is used when you want to target a specific general architecture while tuning for a more specific one (such as -march=corei7 -mtune=core-avx2 for instance). The -march options tells GCC to output assembly that will typically only run on that particular chipset or possibly newer whereas -mtune allows the executable to run on the general arch you specify but tweaking performance for a specific chipset. Once you've told it only to use specific instructions for a particular chip, there isn't really any tuning that it has to worry about unless the -mtune you specify is more specific than the one specified in -march. Again, if building for only the system that will run the binary then -march=native will take care of all of this for you without needing to deal with it (native will be swapped out for core-avx2 if you are building on a core-avx2 arch).

As for your libfdk_aac and x264/x265 question, I would not enable "-ffast-math" for audio in general since some of the correct IEEE behavior may be relied upon. Also, keep in mind that x265 (as is currently configured within the script) will ignore any custom flags you choose to use and will be replaced with the "release flags" that are shipped with it which includes "-DNDEBUG -O3 -ffast-math" This is by x265's own design and must be overriden manually. This only applies to x265.

The following lines in the script should be changed from:

    cmake -G "MSYS Makefiles" \
        -DCMAKE_INSTALL_PREFIX:PATH=$LOCALDESTDIR \
        -DBIN_INSTALL_DIR=$LOCALDESTDIR/bin-video \
        -DENABLE_SHARED:BOOLEAN=OFF \
        -DHG_EXECUTABLE=/usr/bin/hg.bat \
        -DCMAKE_CXX_FLAGS="-static-libgcc -static-libstdc++" \
        -DCMAKE_C_FLAGS="-static-libgcc -static-libstdc++" \
        ../../source

to

    cmake -G "MSYS Makefiles" \
        -DCMAKE_INSTALL_PREFIX:PATH=$LOCALDESTDIR \
        -DBIN_INSTALL_DIR=$LOCALDESTDIR/bin-video \
        -DENABLE_SHARED:BOOLEAN=OFF \
        -DCMAKE_CXX_FLAGS_RELEASE:STRING="$CXXFLAGS -DNDEBUG -O3 -ffast-math" \
        -DCMAKE_EXE_LINKER_FLAGS_RELEASE:STRING="$LDFLAGS -Wl,-static" \
        ../../source

This is the only way of passing user-defined flags into the build system for it. As for pulling specific releases, the script pulls trunk and not any particular tags or old releases so if your issue is that older builds perform worse, that won't ever be an issue anyway. As for why VLC might be encountering issues, VLC is subject to whatever codecs you have setup as well which may include video-hardware decoding. You may want to check this.

Profiling works when GCC builds/links any code that has "-fprofile-generate" specified. At this point, "-lgcov" (gcov library) is linked in automatically along with some other flags to dump statistics to a directory every time the binary is run (in the form of .gcda files and by default, these are dumped within whatever directory the build was done). The next build will use "-fprofile-use" which will then use these files to optimize the resulting code. The x264 build system does this automatically when you tell it to "make VIDS='.....' fprofiled" for you, however it is the only one that will do this automatically. It essentially automates a build using -fprofile-generate, then runs the resulting binary against the files you specified in the make line, then rebuilds with -fprofile-use all without you having to intervene. When profiling, it is important that you run whatever binary you are profiling in the way in which you intend to use it (in this case, by passing in files similar to what you will be encoding). Using the same variety of files is best, however it likely won't hurt profiling terribly by passing in different types (make sure to at least profile an animated one along filmed video though since these are wildly different). The goal of profiling is to hit as many code paths as will likely be used heavily when you eventually use the tool on its own so its best to feed it a small variety that will match what you use it for. Don't, for instance, give it only an anime file when you intend to use it for encoding film video and vice-versa. When you run make, you will see it performing multiple encodings using these files. Also, profile files are invalid between different build versions. You cannot profile an application, then update the revision or change its build flags and expect the profiles to remain consistent. They won't and will almost always error out during the build or simply provide worse performance since they no longer match the code they were intended to profile.

I think your most recent error has more to do with installing an mpv.com file after an mpv.exe file. Msys2 (when attempting to execute files in a path) will go through a number of extensions (.exe, .com, *.bat, etc) in addition to files without extensions. This being a directory in your path, I'm wondering if that's not the case. Another build issue which is likely not the case here but worth mentioning, I will put in bold below. Either way, no one actually uses .com files anymore and this file may be hitting your anti-virus due to its file extension. Without temporarily disabling the anti-virus, it is hard to say for sure.

Specifying building with too high an amount of CPU's can cause build errors if files are either needed or in use when a subsequent rule hits them. If a build has failed, lowering the CPU count may resolve the issue.

mabreedlove commented 9 years ago

Issue regarding x265 not accepting build flags have been pushed to trunk.