OpenCV fails to compile with -Wl,--hash-style=gnu set

[elsandosgrande]

Well, it is as title says. If I set media-libs/opencv *FLAGS-=-Wl,--hash-style=gnu* in /etc/portage/packages.cflags/opencv.conf, everything is dandy again.

Do pardon me if this is redundant, however, I found no references to opencv or the hash-style recommendation amongst the issue tickets, therefore I present this ticket to you now.

Since the compiler flag in question is presented in the examples and somewhat recommended, though stated as optional, I would prefer it if the workaround in question were added to ltoworkarounds.conf, but do fully understand if doing so is outside the scope of the overlay, as it is stated as optional.

Have a nice day!

[funghetto]

Can't reproduce:

media-libs/opencv-3.4.1-r5::gentoo was built with the following:
USE="eigen ffmpeg gtk jpeg opencl opengl openmp png python qt5 tiff webp xine (-contrib) (-contrib_cvv) (-contrib_dnn) (-contrib_hdf) (-contrib_sfm) (-contrib_xfeatures2d) (-cuda) -debug -dnn_samples -examples -gdal (-gflags) (-glog) -gphoto2 -gstreamer -ieee1394 -java -jpeg2k -lapack -libav -openexr -pch (-tesseract) -testprograms -threads -v4l -vaapi (-vtk)" ABI_X86="32 (64) (-x32)" CPU_FLAGS_X86="avx avx2 fma3 popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3" PYTHON_TARGETS="python2_7 python3_6 -python3_5"
CFLAGS="-march=native -O3 -fgraphite-identity -floop-nest-optimize -fdevirtualize-at-ltrans -fipa-pta -fno-semantic-interposition -flto=4 -fuse-linker-plugin -falign-functions=32 -pipe -Wl,--as-needed -Wl,--hash-style=gnu"
CXXFLAGS="-march=native -O3 -fgraphite-identity -floop-nest-optimize -fdevirtualize-at-ltrans -fipa-pta -fno-semantic-interposition -flto=4 -fuse-linker-plugin -falign-functions=32 -pipe -Wl,--as-needed -Wl,--hash-style=gnu"
LDFLAGS="-Wl,--as-needed -Wl,--hash-style=gnu -march=native -O3 -fgraphite-identity -floop-nest-optimize -fdevirtualize-at-ltrans -fipa-pta -fno-semantic-interposition -flto=4 -fuse-linker-plugin -falign-functions=32 -pipe"

[elsandosgrande]

I do not have -Wl,--as-needed set.

Here: opencv.txt

As I am currently compiling opencv, the log in /var is inadequate, but I do think that the terminal output will be of help.

Also, GitHub does not accept .conf files, so here is a renamed make.conf: make.txt

[ionenwks]

That sounds strange given hash-style=gnu is a default on binutils with gentoo (--enable-default-hash-style=gnu is used except on mips) and should be enabled whether it's set or not. This does lead me to wonder why it's (still) even in the LDFLAGS for GentooLTO though, it's default unless using old binutils.

I don't really know anything about opencv though, maybe it's doing something strange with specific configurations.

[barolo]

I think that it's related to one of your global useflags, maybe ffmpeg? Mine builds just fine with any combo of ldflags and: USE="eigen jpeg opengl openmp png python qt5 vaapi" on opencv

[elsandosgrande]

Well, regarding the use flags, here they are (emerge --pretend --verbose [package name]): [ebuild R ] media-video/ffmpeg-4.1.3:0/56.58.58::gentoo USE="X alsa bzip2 encode fontconfig gme gpl hardcoded-tables iconv jpeg2k libass libcaca libxml2 lzma mp3 network openal opencl opengl openh264 opus postproc pulseaudio sdl speex svg theora threads truetype twolame v4l vaapi vdpau vorbis vpx webp x264 x265 xcb xvid zlib zvbi (-altivec) -amr -amrenc (-appkit) -bluray -bs2b -cdio -chromaprint -chromium -codec2 -cpudetection -debug -doc -fdk -flite -frei0r -fribidi -gcrypt -gmp -gnutls -gsm -iec61883 -ieee1394 -jack -kvazaar -ladspa -libaom -libdrm -libilbc -libressl -librtmp -libsoxr -libv4l -lv2 (-mipsdspr1) (-mipsdspr2) (-mipsfpu) (-mmal) -modplug -openssl -oss -pic -rubberband -samba -snappy -srt -ssh -static-libs -test -wavpack -zeromq -zimg" ABI_X86="(64) -32 -x32" CPU_FLAGS_X86="aes avx avx2 fma4 mmx mmxext sse sse2 sse3 sse4_1 sse4_2 ssse3 xop -3dnow -3dnowext -fma3" FFTOOLS="aviocat cws2fws ffescape ffeval ffhash fourcc2pixfmt graph2dot ismindex pktdumper qt-faststart sidxindex trasher" VIDEO_CARDS="-nvidia" 0 KiB

[ebuild R ] media-libs/opencv-3.4.1-r5:0/3.4.1::gentoo USE="eigen examples ffmpeg gstreamer gtk java jpeg jpeg2k opencl openexr opengl openmp pch png python qt5 threads tiff v4l vaapi webp -contrib -contrib_cvv -contrib_dnn -contrib_hdf -contrib_sfm -contrib_xfeatures2d -cuda -debug -dnn_samples -gdal -gflags -glog -gphoto2 -ieee1394 -lapack -libav -tesseract -testprograms -vtk -xine" ABI_X86="(64) -32 -x32" CPU_FLAGS_X86="avx avx2 popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3 -fma3" PYTHON_TARGETS="python2_7 python3_6 -python3_5" 0 KiB

I ended up stumbling upon my solution by adding -o $@ in a vain attempt at resolution ( https://stackoverflow.com/a/20514664 ; as you can see, I am rather naive), which removed all common flags after parsing. Thereafter, seeing as removing GentooLTO flags did not result in resolution previously, I proceeded to manipulate some, at least in my prior opinion, unrelated flags (the error flags), as I saw no other weak links. I first removed all -W flags. It continued past the fifteenth element. Tried again with -Wall, no issues were encountered. Appened -Wextra thereafter, no change from the previous attempt. Then I proceeded to replace those two flags with -Wl,--hash-style=gnu and, as if on cue, the exact same error was encountered. After uncommenting my regular common flags, I saw it again. After that, I had the idea to create my own cflags configuration file and here we are now.

EDIT: @barolo After looking at your use flags and mine side-by-side, it seems that something is amiss with the additional ones on my end.

[barolo]

@elsandosgrande I've seen your global useflags, the ones you have over mine come from your globals, clean up there first. In general using flags globally is to be avoided, with the exception of most obvious ones. [ I'm taking about the "USE=" in your make.conf ]

[elsandosgrande]

@elsandosgrande I've seen your global useflags, the ones you have over mine come from your globals, clean up there first.

Yes, I understand. That was a conscious decision, since it saves me time having to add those to packages individually and helps for mobility. I know that having all of my eggs in one basket is not good, but the result is exactly the same as if I had added the use flags to all packages to which they can be applied individually.

In general using flags globally is to be avoided, with the exception of most obvious ones.

Yes, but, in cases like with experimental, I do add them to packages specifically (I would like to have the kernel sources' experimental features, like that patch that enables -march=bdver4, but would like to avoid experimental Bluetooth features, so this one is in package.use only).

Here, I will share package.use: package.txt

EDIT: Silly old me (even though I am merely 17), I forgot about how commenting works in markup. EDIT 2: Correction, quoting.

[barolo]

@elsandosgrande see above, package.use, per package use flags are fine, check ones in your make.conf, ones from USE=

[elsandosgrande]

@elsandosgrande see above, package.use, per package use flags are fine, check ones in your make.conf, ones from USE=

I know, I added those personally. The flags, once parsed, are all as I would like them to be. If I were to instead add them in package.use, it would not change the outcome. That, and package.use would become huge.

[elsandosgrande]

Since I still have updates and Python use flag changes to complete and I have to go out to buy water, or else mom will not be happy, I will try to determine which use flag makes -Wl,--hash-style=gnu break opencv later, probably today.

[barolo]

I'll try to explain, you should probably read up on USE flags too. First try removing them temporarily and recompile opencv to check if one of them is the culprit. Hopefully you'll get something from this... Global use flags apply to everything and you can't really predict what they'll do and when they''ll be applied. That's why we use package.use to make sure that it's used just per package, yes it's usually huge, that's why many use it as directory with per package files for example ./package.use/opencv << [ you put your use flags here ] to make it manageable.

[elsandosgrande]

I understand that and was going to do that process for checking them. I would create a new line in package.use for OpenCV and then proceed to remove the flags in the delta one by one (the delta being between your flags and mine).

And regarding predictability, I look into what the flags do and what scope they have (package.gentoo.com/useflags). So far, the only use flag that I could not afford to have set globally is experimental.

Post Scriptum

I do understand that having all of the use flags declared globally in make.conf is in bad form, but I have not had any issues so far and it is easier to manage these cases individually rather that individually managing use flags for, basically, half of all packages installed, roughly at least.

[elsandosgrande]

So, I tried to send the build log after the rebuild failed, but GitHub is not letting it pass, even when the extension is .txt. I guess I will go on Pastebin later.

Edit

Here: https://pastebin.com/kgvzeMbk

[InBetweenNames]

@ionenwks This thread gave me a bit of caution:

https://patchwork.ozlabs.org/patch/849420/

I haven't delved too deep into it. My understanding was that both would be used unless otherwise specified. However, I might have been wrong about that.

[elsandosgrande]

@ionenwks This thread gave me a bit of caution:

https://patchwork.ozlabs.org/patch/849420/

Wait, what is this thread even for?

[barolo]

@elsandosgrande Could you try building with the same useflags as me?

[elsandosgrande]

I intended to, but got carried away with setting up Visual Studio Code for the first time on Gentoo.

Yeah, I will copy your flags and see if they work for me, but I fully intend to find the culprit at a later time, as it is 00:48 here in Sarajevo, Bosnia and Herzegovina.

Edit

It just built element 66 out of 800 and something, so it is a success (as is plainly visible, it fails at or after element 15 with my setup). Now to either hunt the mischievous flag down, or to go to bed.

[ionenwks]

@InBetweenNames Well, a quick test is simple. I compiled 3 binaries, one without flags, one with -Wl,--hash-style=gnu, and one with -Wl,--hash-style=both. The two first were identical, the last one differed. So it seems to be redundant (at least on binutils 2.32+ which this overlay enforces) Doesn't harm to keep it but I'd say less flags and explanations is better for the general user.

[elsandosgrande]

All right, I believe that I have successfully tracked the root of our little conundrum down to pch, as, after commenting USE=... out in make.conf, =media-libs/opencv-3.4.1-r5 examples ffmpeg gstreamer gtk java jpeg2k opencl openexr -pch threads tiff v4l webp builds past the problematic fifteenth element. It enables pre-compiled headers, which, allegedly, improve compilation time ( https://packages.gentoo.org/useflags/pch ).

So, after copying the workaround line for LibreOffice and modifying it accordingly, I ended up with media-libs/opencv "has pch ${IUSE//+} && use pch && FlagSubAllFlags -Wl,--hash-style=gnu*" in package.cflags/opencv.conf. It works, by the way.

Here are some files: package use.txt https://pastebin.com/vWZm03xY

After it passed the twentieth element, I just went ctrl+c and pasted the log contents.

As it is now 02:30, I will proceed to go to bed now. Do notify me if I have missed something!

@ionenwks Since you have replied as I am typing this out, I shall respond to your reply here. I, sadly, do not have a single C or C++ file to compile, nor do I have a way of comparing the binaries (or, at least, the knowledge to do so), therefore I apologize in advance if this is a redundant ticket.

[InBetweenNames]

@ionenwks thank you -- I will reproduce that, update my setup, and update the docs to reflect this.

[elsandosgrande]

... and update the docs to reflect this.

Does that render this ticket irrelevant beyond that reply?

[InBetweenNames]

After I reproduce it, so long as removing this flag fixes the problem, then I'd say it's fixed :)

I got the original advice from the Gentoo Wiki, but I realize now that it may not be up to date.

[elsandosgrande]

I got the original advice from the Gentoo Wiki, but I realize now that it may not be up to date.

Yeah, I get that feeling with a good portion of it. Exempli gratia, the real-time section of the wiki (PulseAudio) talks about the 2.6 kernel. Yikes!

I have the following procedure:

• check Gentoo Wiki
• check Arch Wiki
• when in conflict, Arch Wiki supersedes, within reason
• adapt Arch Wiki to suit needs

Well then, I am off to fight LibreOffice compilation.

Edit

It appears that LibreOffice hates -Wextra, its C++11/14/17 check to be precise, as it returns a no when -Wextra is present. Terrific! checking whether x86_64-pc-linux-gnu-g++ supports C++17, C++14, or C++11... configure: error: no

[InBetweenNames]

Tested and reproduced. I will update the docs to reflect this. Thanks again @ionenwks -- anything that makes this stuff simpler, the better.

[InBetweenNames]

Just one last thing @elsandosgrande -- try to make sure your profile's LDFLAGS are respected in your make.conf, unless you really need to override them. For example, when modifying your LDFLAGS, do the following:

LDFLAGS="${LDFLAGS} -Wl,--foo=bar"

Instead of:

LDFLAGS="-Wl,--foo=bar"

This will ensure you get your -Wl,-O1 -Wl,--as-needed as your profile probably contains.

Note also that portage-bashrc-mv automatically takes care of adding your CFLAGS to your LDFLAGS, so there's no need to manually do so yourself with the COMMON_FLAGS variable.

[elsandosgrande]

Just one last thing @elsandosgrande -- try to make sure your profile's LDFLAGS are respected in your make.conf, unless you really need to override them.

Wait, the selected profile has its own LDFLAGS? Also, would they not be applied automatically, not being able to be overwritten?

For example, when modifying your LDFLAGS, do the following:

LDFLAGS="${LDFLAGS} -Wl,--foo=bar"

Instead of:

LDFLAGS="-Wl,--foo=bar"

This will ensure you get your -Wl,-O1 -Wl,--as-needed as your profile probably contains.

So, that is how to preserve the LDFLAGS from the system profile. All right.

Note also that portage-bashrc-mv automatically takes care of adding your CFLAGS to your LDFLAGS, so there's no need to manually do so yourself with the COMMON_FLAGS variable.

Eh, as long as it does not give me any trouble, I will happily just wrap everything in COMMON_FLAGS, as I would end up simply applying all flags for every variable either way. Plus, regarding those automatic additions, I have added some flags next to the LTO-and-company ones. Also, it makes my life much easier when messing around with the flags during troubleshooting.

Well then, off to add ${LDFLAGS} to the LDFLAGS, finish setting up Visual Studio Code and then have Icedtea, OpenJDK and LibreOffice compile while I sleep (LibreOffice because I am trying out Python 3.7 and the Java implementations to check if pch is going to be a problem there as well).

Have a nice day/good night/ whatever time of day it is for you! It is 22:16 here in Sarajevo, by the way. I should probably start doing this earlier in the day.

[elsandosgrande]

So, with the conclusion that -Wl,--hash-style=gnu is redundant and even dangerous and, therefore, to be removed from all flag variables, has this ticket reached its end?

Aside from that, are there any methods of communication other than GitHub issue tickets? For example, Olive has a Discord server for discussions and other things that would not warrant a ticket ( https://github.com/olive-editor/olive/issues/543 ; I set the server up, though we had disagreements with regards to the scope of the server). I think that this overlay could benefit from proper discourse, let alone the less initiated amongst us (like me, for example, when it comes to setting up an overlay).

Edit

One more thing. How do I find the list of available linker options for a given linker? Exempli gratia, I could not find hash-style anywhere in the GCC man page (man gcc).

[InBetweenNames]

Redundant yes, but I don't think there's any danger in including it. It should be removed unless there's good reason to include it, I think.

Regarding other means of communication, we had an irc channel at one point, but I wasn't online enough to make real use of it. I'm open to suggestions.

[InBetweenNames]

Hash style is in the manpages for ld

[InBetweenNames]

Closing issue as it was resolved via the documentation update.

[elsandosgrande]

@InBetweenNames

1. My suggestion, a Discord server.

2. For some reason, I get unterminated character set at position 1 when trying to re-add your overlay (git was complaining about file modifications and whatnot, so layman --remove lto-overlay, layman --add lto-overlay and here we are):

   * Adding overlay...
   * Overlay "lto-overlay" is not official. Continue installing? [y/n]: y
   
   * CLI: Errors occurred processing action add
   * Exception caught installing repository "lto-overlay":
   * unterminated character set at position 1

Post Scriptum

I guess this is a good example of what I had in mind when I wrote "discourse". Pardon me if it is in bad taste to reply to a closed ticket in this manner.

[InBetweenNames]

Don't worry about it -- we're not too formal around here.

My only problem with a discord server is that it's not super accessible. What about Matrix?

[elsandosgrande]

What is Matrix? Also, is it safe to add the overlay now?

Edit

1. Thank you for telling me about man ld!
2. Could you explain to me how Discord might not be accessible? I mean, Olive is a FOSS project, but Discord is still good for them (there are actually quite a few people on the server). Please do share your point of view with me.

[InBetweenNames]

Discord is pretty closed off -- then again, I suppose so is GitHub. Well, at least with GitHub we get our own .git which could be moved around. It's fine for gaming communities, but we have a number of people here from different backgrounds that might not want Electron running on their desktop, etc. I suppose you could use the web client, but I find that gets a little tedious.

Matrix is an open protocol, by contrast. The most popular client is Riot (which is Electron based) but there's nothing stopping anyone from implementing the protocol on other devices. You can do that with Discord too, but it'll get you banned eventually.

I liked IRC, but the lack of message history without a bouncer was kind of a dealbreaker for me, who might only pop on a few times a week. Makes moderation very difficult.

[elsandosgrande]

Oh. Well, I guess it would be best to decide democratically then, with a ticket. Now, why am I getting that error with your overlay and yours alone?

[InBetweenNames]

No idea -- I've had that issue before in other repos due to an outdated layman. Try using eselect repository, it may work better for you.

[InBetweenNames]

I created a Gitter for GentooLTO. It's in the README. https://gitter.im/gentooLTO/community

[elsandosgrande]

Well, eselect worked, but I still have the issue with layman. Oh well, a problem for later, I guess.