Open LigH-de opened 1 year ago
I'm also running into this issue. Seems to be related to https://github.com/curl/curl/issues/10620 and https://github.com/curl/curl/issues/10694
I'm not sure which version of curl is cloned using the autobuild but my guess is that is was fixed with 8.0.0 as it's an issue with 7.88.1
libcurl was rebuilt today; build/curl-git/build-32bit/libcurl.pc reports Version: 8.3.0-DEV
and contains -lbrotlidec
twice.
msys64/var/log/pacman.log reports mingw-w64-[i686|x86_64]-brotli (1.0.9-5 -> 1.0.9-6)
as last update.
Same
This happened to me today as well
same
Try downgrading pkgconf. I don't see this link error, but I'm admittedly running an older pkgconf version due to an unrelated issue with pkgconf in my environment that I haven't tracked down yet (I'm currently using 1.9.5). I'm wondering if my local issue is an actual bug rather than just specific to my environment.
That's not easy because the suite keeps the MSYS2 environment up-to-date.
That's not easy because the suite keeps the MSYS2 environment up-to-date.
True, but it's worth a test and you can add it to an IgnorePkg line in /etc/pacman.conf for the duration of the test. I'm mostly just trying to figure out where the issue may be and am definitely not suggesting this as a permanent solution 😇
Same problem here
I also have this problem
I have no idea where the responsibility for this lies, but without it being fixed, m-ab-s is virtually useless, unfortunately.
Try downgrading pkgconf. I That's not easy because the suite keeps the MSYS2 environment up-to-date.
If one is to try this fix what steps do I have to perform?
C:\media-ab\local64\bin-global>curl --version
curl 8.3.0-DEV (x86_64-w64-mingw32) libcurl/8.3.0-DEV OpenSSL/3.1.2 zlib/1.3 brotli/1.0.9 zstd/1.5.5 libidn2/2.3.0 libpsl/0.21.0 (no IDNA support) nghttp2/1.55.1
Release-Date: [unreleased]
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM PSL SPNEGO SSL SSPI threadsafe TLS-SRP UnixSockets zstd
- - -
C:/media-ab/msys64/mingw64/lib/libbrotlidec.a(decode.c.obj):(.rdata$.refptr._kBrotliPrefixCodeRanges[.refptr._kBrotliPrefixCodeRanges]+0x0): undefined reference to `_kBrotliPrefixCodeRanges'
C:/media-ab/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/13.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/media-ab/msys64/mingw64/lib/libbrotlidec.a(state.c.obj):(.text+0xda): undefined reference to `BrotliGetDictionary'
C:/media-ab/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/13.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/media-ab/msys64/mingw64/lib/libbrotlidec.a(state.c.obj):(.text+0xe6): undefined reference to `BrotliGetTransforms'
C:/media-ab/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/13.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/media-ab/msys64/mingw64/lib/libbrotlidec.a(state.c.obj):(.text+0x103): undefined reference to `BrotliDefaultAllocFunc'
C:/media-ab/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/13.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/media-ab/msys64/mingw64/lib/libbrotlidec.a(state.c.obj):(.text+0x10a): undefined reference to `BrotliDefaultFreeFunc'
C:/media-ab/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/13.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/media-ab/msys64/mingw64/lib/libbrotlidec.a(state.c.obj):(.rdata$.refptr.BrotliDefaultFreeFunc[.refptr.BrotliDefaultFreeFunc]+0x0): undefined reference to `BrotliDefaultFreeFunc'
C:/media-ab/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/13.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/media-ab/msys64/mingw64/lib/libbrotlidec.a(state.c.obj):(.rdata$.refptr.BrotliDefaultAllocFunc[.refptr.BrotliDefaultAllocFunc]+0x0): undefined reference to `BrotliDefaultAllocFunc'
collect2.exe: error: ld returned 1 exit status
make: *** [Makefile:328: mediainfo.exe] Error 1
@Murmur
If you want to try it, insert the following line into line 3 of the build\media-suite_compile.sh file:
for 64-bit:
pacman -U --noconfirm https://repo.msys2.org/mingw/mingw64/mingw-w64-x86_64-pkgconf-1~1.9.5-1-any.pkg.tar.zst
for 32-bit:
pacman -U --noconfirm https://repo.msys2.org/mingw/mingw32/mingw-w64-i686-pkgconf-1~1.9.5-1-any.pkg.tar.zst
I have only tested it for 64-bit Clang
Don't get confused, every time you run the script the package is upgraded and then downgraded again. But this should only take seconds.
Haven't tried for 32-bit, but certainly works for 64-bit. Thanks a lot.
Works for 64bit script such as gpac.exe, MP4Box.exe, mediainfo.exe
binaries were compiled.
Override pkgconf binary:
media-suite_compile.sh (line 4): pacman -U --noconfirm "https://repo.msys2.org/mingw/mingw64/mingw-w64-x86_64-pkgconf-1~1.9.5-1-any.pkg.tar.zst"
However later vulkan-loader.git
project failed and mb script exits, no ffmpeg
binaries were written yet I was looking for. This may or may not be related to a same problem described in this topic.
( https://github.com/m-ab-s/media-autobuild_suite/issues/2507 vulkan-loader )
┌ vulkan-loader git ..................................... [Updates found]
├ Running uninstall...
0001-loader-cross-compile-static-linking-hacks.patch
Patch could not be applied with `git am`. Continuing without patching.
0003-loader-prefix-cjson-symbols-and-mark-most-as-static.patch
Patch could not be applied with `git am`. Continuing without patching.
├ Running dependencies...
├ Installing Vulkan-Headers...
├ Running uninstall...
├ Running cmake...
├ Running build...
├ Running install...
├ Building Vulkan-Loader...
├ Running cmake...
├ Running build...
Likely error (tail of the failed operation logfile):
C:/projects/media-ab/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/13.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: loader/libloader-opt.a(phys_dev_ext.c.obj):phys_dev_ext.c:(.rdata$.refptr.vkPhysDevExtTramp7[.refptr.vkPhysDevExtTramp7]+0x0): undefined reference to `vkPhysDevExtTramp7'
C:/projects/media-ab/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/13.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: loader/libloader-opt.a(phys_dev_ext.c.obj):phys_dev_ext.c:(.rdata$.refptr.vkPhysDevExtTramp6[.refptr.vkPhysDevExtTramp6]+0x0): undefined reference to `vkPhysDevExtTramp6'
C:/projects/media-ab/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/13.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: loader/libloader-opt.a(phys_dev_ext.c.obj):phys_dev_ext.c:(.rdata$.refptr.vkPhysDevExtTramp5[.refptr.vkPhysDevExtTramp5]+0x0): undefined reference to `vkPhysDevExtTramp5'
C:/projects/media-ab/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/13.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: loader/libloader-opt.a(phys_dev_ext.c.obj):phys_dev_ext.c:(.rdata$.refptr.vkPhysDevExtTramp4[.refptr.vkPhysDevExtTramp4]+0x0): undefined reference to `vkPhysDevExtTramp4'
C:/projects/media-ab/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/13.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: loader/libloader-opt.a(phys_dev_ext.c.obj):phys_dev_ext.c:(.rdata$.refptr.vkPhysDevExtTramp3[.refptr.vkPhysDevExtTramp3]+0x0): undefined reference to `vkPhysDevExtTramp3'
C:/projects/media-ab/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/13.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: loader/libloader-opt.a(phys_dev_ext.c.obj):phys_dev_ext.c:(.rdata$.refptr.vkPhysDevExtTramp2[.refptr.vkPhysDevExtTramp2]+0x0): undefined reference to `vkPhysDevExtTramp2'
C:/projects/media-ab/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/13.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: loader/libloader-opt.a(phys_dev_ext.c.obj):phys_dev_ext.c:(.rdata$.refptr.vkPhysDevExtTramp1[.refptr.vkPhysDevExtTramp1]+0x0): undefined reference to `vkPhysDevExtTramp1'
C:/projects/media-ab/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/13.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: loader/libloader-opt.a(phys_dev_ext.c.obj):phys_dev_ext.c:(.rdata$.refptr.vkPhysDevExtTramp0[.refptr.vkPhysDevExtTramp0]+0x0): undefined reference to `vkPhysDevExtTramp0'
collect2.exe: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.
build failed. Check C:/projects/media-ab/build/vulkan-loader-git/build-64bit/ab-suite.build.log
This is required for other packages, so this script will exit.
This successfully works around issue building mediainfo.exe
If you want to try it, insert the following line into line 3 of the build\media-suite_compile.sh file: for 64-bit:
pacman -U --noconfirm https://repo.msys2.org/mingw/mingw64/mingw-w64-x86_64-pkgconf-1~1.9.5-1-any.pkg.tar.zst
What do I insert where when I run both 32 and 64 bit compilations in one? I guess that requires a bitness switch?
@LigH-de like @H7-26 already wrote: there are 2 packages, one for 32 bit, one for 64 bit. If you have both 32 and 64 bit, well then, install both :-)
However, when downgrading manually (without changing media-suite_compile.sh): After downgrading, you should tell pacman to keep it's fingers from touching pkgconfig, otherwise it would update pkgconf at next run.
To do this add
IgnorePkg = pkgconf mingw-w64-x86_64-pkgconf
(64-bit)
or
IgnorePkg = pkgconf mingw-w64-i686-pkgconf
(32-bit)
to pacman.conf found in /etc
When editing while using windows pacman.conf can be found in
I have no idea where the responsibility for this lies, but without it being fixed, m-ab-s is virtually useless, unfortunately.
@1480c1 this is unresolved for weeks and stops the whole thing from working.
The fix from MurMur above works great.
If you want to try it, insert the following line into line 3 of the build\media-suite_compile.sh file: for 64-bit: pacman -U --noconfirm https://repo.msys2.org/mingw/mingw64/mingw-w64-x86_64-pkgconf-1~1.9.5-1-any.pkg.tar.zst
@Biswa96 wrote in MSYS2's IRC, it may be related to https://github.com/msys2/MINGW-packages/issues/18554
Adding
-lbrotlicommon
may workaround the issue.
Could this be a simple extra parameter to be added to the build script for all projects which need libbrotli?
Not sure I have much time to run this down further right now, but there is one behavioural difference between 1.9.5 and 2.0.x that may explain some of this (and explains one non-suite issue I had locally). Running pkgconf --path
against a package that is both suite-compiled and provided by mingw (libcurl, libxml2, etc) results in different path search/matching behaviour between the two pkgconf versions. With 1.9.5, the .pc file in /local64/lib/pkgconfig
was matched first, but 2.0.x matches the .pc files in /mingw64/lib/pkgconfig
first instead. In this case (and others), the two are likely to be different and I'd assume the suite-compiled versions should be used for compiling/linking, but that's not the case with 2.0.x. This may cause problems if cflags and libs differ between the two (like this one).
FYI, I think there's still an ordering bug filed against upstream pkgconf, so this could be an offshoot of that, but it might also be helpful to know which version's behaviour is the proper, intended path search behaviour as well.
Edit: It's basically not respecting PKG_CONFIG_PATH ordering. It always searches the last path in the envvar first for some reason.
some libbrotli object files were being renamed by the script via "do_hide_all_sharedlibs". manually copying them back during the libbrotli config allowed me to finish the compilation successfully
some libbrotli object files were being renamed by the script via "do_hide_all_sharedlibs". manually copying them back during the libbrotli config allowed me to finish the compilation successfully
a silly question as I can't remember if the script builds shared or static or both ... were you building static ?
a silly question as I can't remember if the script builds shared or static or both ... were you building static ?
yes i built static
Same – MABS recommends building static ffmpeg, so I select static only as well. Building both in parallel is supported but usually causes a higher chance of trouble, so I don't try that anymore.
@LigH-de Thanks a lot! Changed line 16 in media-suite_compile.sh to
FFMPEG_BASE_OPTS=("--pkg-config=pkgconf" --pkg-config-flags="--keep-system-libs --keep-system-cflags --static" "--cc=$CC" "--cxx=$CXX" "--ld=$CXX" "--extra-cxxflags=-fpermissive" "--extra-cflags=-Wno-int-conversion" --extra-ldflags=-lbrotlicommon )
Adding --extra-ldflags=-lbrotlicommon while using upstream msys pkgconf did the trick for me!
I still had some issues with glslang, shaderc and vulkan-loader I had to work out, but that might be unrelated to this issue.
So, forgive me the foreign language, to @LigH-de: "Danke, einmal mit Profis arbeiten!" :-D
I am not sure if that is a suitable workaround in general. Possibly just for your state of the suite.
I just ran mine again after several weeks. And fonfconfig has now very similar issues linking bzip2. So I'll rather wait for a fix in pkgconf.
Well, of course, it's just a very quick'n'dirty fix, which only masquerades the underlying root cause and might introduce other problems. So, surely this needs to be further investigated, but for those impatient folks like me, who want a binary now , it does the trick ;-)
I am not sure if that is a suitable workaround in general. Possibly just for your state of the suite.
I just ran mine again after several weeks. And fonfconfig has now very similar issues linking bzip2. So I'll rather wait for a fix in pkgconf.
Just wondering, are the pkgconf maintainers aware of the issue ? (a link to the issue would be handy if you know it)
As linked above - MSYS2 maintainers like @Biswa96 are, so I'm pretty sure they forwarded it.
Cool ! Thank you.
I not sure how to interpret that there seems to be no response to the issue(s) over there ? Also seems to be no commits for a few weeks. Perhaps on vacation ?
Ahhh... https://github.com/pkgconf/pkgconf/issues/322 led me to https://github.com/pkgconf/pkgconf/issues/268 Now I know why it felt like groundhog day :-D
I managed to get at least most of the suite compiling reliably here (minus the stuff I don't build, of course), but it took a couple of minor edits and downgrading pkgconf even further (to 1.8.0) until either an upstream fix arrives or I muster the willpower to fix it myself. I'm cleaning up my edits and will create some PRs once that's done to help solve some issues that I ran into. I don't compile vulkan-related stuff, so I probably won't be able to help with that (may just need the existing patches rebased). My edits are nothing major, though, just a fix for luajit's new rolling release scheme and changing the way shaderc's dependencies are handled, so it shouldn't take me too long to clean up/test.
@waldonnis thank you.
Just added PR #2517 to make shaderc compilation less prone to breaking and use mpv's 0.36 release tag until the PR to swap mpv from waf (now completely removed) to meson is worked out. If that's stalled, I may see what I can do in my spare time to further that along. I have a luajit patch as well which I'm probably just going to create a bug report for and paste it in there for folks to use until I can figure out how best to submit that otherwise (if needed).
Aside from those fixes, downgrading to pkgconf 1~1.8.0-2 should allow the suite to compile reliably (again, not including Vulkan-related stuff, since I didn't even try it).
All of my general changes have been merged at this point, fyi (along with my dovi_tool and hdr10plus_tool support stuff - yay!), so we should be back to near-normal now aside from the pkgconf issue.
@waldonnis thank you for the magic touch. Anyone interested this is how I was able to build ffmpeg+ffplay +mediainfo +mp4box+gpac +random other
binaries.
downgrade pkgconf
build/media-suite_compile.sh, line 5
pacman -U --noconfirm "https://repo.msys2.org/mingw/mingw64/mingw-w64-x86_64-pkgconf-1~1.8.0-2-any.pkg.tar.zst"
disable placebo+glsl+vulkan
build/ffmpeg_options.txt
list, you can remove a line or use #
to comment out.
--enable-libplacebo --enable-libglslang --enable-vulkan
Vulkan compilation fails on a patch merge due to a source code changes or something else black magic.
@waldon disable placebo+glsl+vulkan
- remove modules from
build/ffmpeg_options.txt
list, you can remove a line or use#
to comment out.--enable-libplacebo --enable-libglslang --enable-vulkan
Vulkan compilation fails on a patch merge due to a source code changes or something else black magic.
Sorry for this dummy's question: I rely on nvidia gpu encoding via ffmpeg's nvenc ... does disabling any of these affect that ? I have some vague uncertain recollection that placebo and/or glslang is needed to enable something good.
@waldon disable placebo+glsl+vulkan
- remove modules from
build/ffmpeg_options.txt
list, you can remove a line or use#
to comment out.--enable-libplacebo --enable-libglslang --enable-vulkan
Vulkan compilation fails on a patch merge due to a source code changes or something else black magic.
Sorry for this dummy's question: I rely on nvidia gpu encoding via ffmpeg's nvenc ... does disabling any of these affect that ? I have some vague uncertain recollection that placebo and/or glslang is needed to enable something good.
Yes, but it may not matter for some uses. For starters, you'd lose the Vulkan filters (doc link), which can be useful.
Obviously, you also lose libplacebo (doc link) which is basically mpv's rendering engine, so it can use a Vulkan interface to do some video processing (scaling, tone-mapping, etc) more efficiently on the gpu. If you're not using libplacebo in your filterchain, though, I doubt you'd miss it. I don't use it, even though I do a few operations that would really benefit from it...mostlly because it breaks compilation so often.
If you're not sure if you're using it now, test an ffmpeg binary without it when doing your normal operations to see if losing it would affect you in time/heat/system resources/power. If not, you save yourself having to deal with the compilation problems. If you want/need it, then just keep working binaries around until you or others can solve the compilation problems (usually, rebasing the patches solves it). When I get sufficiently annoyed at not having it, I'll probably see if anything can be done to make building with it a tad more resilient. Unfortunately, rebasing patches can be thorny at times, so the effort may not yield anything worthwhile.
OK, thanks. I'll do that.
I recall having many issues over the years building Khronos stuff, it'd often break with new commits and require attention. So much so for example I gave up on their ICD Loader and went another route.
@Biswa96 wrote in MSYS2's IRC, it may be related to msys2/MINGW-packages#18554
Adding
-lbrotlicommon
may workaround the issue.Could this be a simple extra parameter to be added to the build script for all projects which need libbrotli?
Well, I just compiled brotli in another build system and noticed brotli needed patching (a) CMakeLists.txt before the configure and (b) the .pc.in files
'run_post_patch': [
'sed -i.bak "/docs/ s/^/#/" "../CMakeLists.txt"',
'sed -i.bak "/man3/ s/^/#/" "../CMakeLists.txt"',
#'cat "../CMakeLists.txt"',
'sed -i.bak "s/-lbrotlienc/-lbrotlienc -lbrotlicommon/" "../scripts/libbrotlienc.pc.in"',
'sed -i.bak "s/-lbrotlidec/-lbrotlidec -lbrotlicommon/" "../scripts/libbrotlidec.pc.in"',
],
I guess a pull request with patches could be created for CMakeLists.txt, scripts/libbrotlidec.pc.in, libbrotlienc.pc.in
Or perhaps equivalent sed's.
I'm not sure how to create a pull request ... I really should look it up :)
EDIT:
Ah. Oh.
MABS doesn't build brotli.
It installs via do_pacman_install brotli
, so the above will not work.
I suppose there could be a libbrotlienc.pc and libbrotlidec.pc somewhere which could be edited via sed
instead.
Perhaps immediately after
do_pacman_install brotli
something like this (untested as yet):
grep_and_sed '-lbrotlidec' "$MINGW_PREFIX"/lib/pkgconfig/libbrotlidec.pc \
's|-lbrotlidec|-lbrotlidec -lbrotlicommon|g' "$MINGW_PREFIX"/lib/pkgconfig/libbrotlidec.pc
grep_and_sed '-lbrotlienc' "$MINGW_PREFIX"/lib/pkgconfig/libbrotlienc.pc \
's|-lbrotlienc|-lbrotlienc -lbrotlicommon|g' "$MINGW_PREFIX"/lib/pkgconfig/libbrotlienc.pc
edit: AH, NO, a newbie-to-linux error, should be 'or' and the check different.
Latest MediaBuild worked fine for the apps I need most ffmpeg, mp4box, gpac, mediainfo
. Few things to note for ffmpeg compilation.
build\media-suite_compile.sh
file anymore.--disable-doc
flag to build\ffmpeg_options.txt
to overcome some problems with doc Makefile.--enable-fontconfig, --enable-libfreetype, --enable-libharfbuzz, --enable-libfribidi
flags to build\ffmpeg_options.txt
if you need drawtext
overlay text filter in an encoding pipelines.--enable-libglslang, --enable-vulkan
flags to build\ffmpeg_options.txt
--enable-libplacebo
flag from build\ffmpeg_options.txt
(still broken)C:\projects\media-ab\local64\bin-video\ffmpeg.exe -filters | grep -i "drawtext"
logs.zip