search

faithfracture / Apple-Boost-BuildScript

Script for building Boost for Apple platforms (iOS, iOS Simulator, tvOS, tvOS Simulator, OS X)
279 stars 111 forks source link

Add in support for building with hidden visibility #36

Closed mobileben closed 5 years ago

mobileben commented 5 years ago

There can be times when hidden visibility needs to be enabled. I've run across this situation where you can end up with linker warnings such as

global weak symbol 'typeinfo for boost::system::error_category::std_category' from file 'boost/lib/libboost_system.a(error_code.o)' means the weak symbol cannot be overridden at runtime. This was likely caused by different translation units being compiled with different visibility settings.

Note I already create a branch in my fork and will create a PR for this once I've run run my build a bit more.

The addition is straight-forward. The main portion is

        --hidden-visibility)
            CXX_FLAGS="$CXX_FLAGS -fvisibility=hidden -fvisibility-inlines-hidden"
            ;;
faithfracture commented 5 years ago

I’ve seen this before too. I think mine was the other way around, or I was getting it from another another of my libraries that I was linking with Boost and that's where the warning was... Something like that, and that’s why I originally added that flag, but it’s been so long now I can’t remember.

I do also recall there being something with one of those flags either being deprecated or unused or it caused another warning or something. If you have the time, could you test this by linking Boost into an intermediate library & mess around with those flags on each build, just to be thorough?

E.g. Application -> intermediate lib -> Boost

Where intermediate lib does not have visibility hidden & Boost does, etc.?

On Mar 7, 2019, at 20:59, mobileben notifications@github.com wrote:

There can be times when hidden visibility needs to be enabled. I've run across this situation where you can end up with linker warnings such as

global weak symbol 'typeinfo for boost::system::error_category::std_category' from file 'boost/lib/libboost_system.a(error_code.o)' means the weak symbol cannot be overridden at runtime. This was likely caused by different translation units being compiled with different visibility settings.

Note I already create a branch in my fork and will create a PR for this once I've run run my build a bit more.

The addition is straight-forward. The main portion is

    --hidden-visibility)
        CXX_FLAGS="$CXX_FLAGS -fvisibility=hidden -fvisibility-inlines-hidden"
        ;;

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

mobileben commented 5 years ago

What is the objective here from testing more? The only reason I'm asking is this situation is a bit dependent on the compilation settings for share libraries as well as the project.

So for example, if you inherit a pre-built shared lib that uses boost and has been compiled one way, you may need this enabled (or disabled). Supporting this provides the flexibility to enable it if needed. It only turns it on if explicitly built using the --hidden-visibility flag.

Those flags are not deprecated.

faithfracture commented 5 years ago

Only to make sure that adding / changing both of those flags didn't cause any other problems. I just remember their being something weird when I was trying to get that different visibility settings warning to go away in my project. I thought it was that one of them wasn't valid with more recent versions of clang or something, but it very well could have been something else. If you're pretty confident that it doesn't cause any other problems then I'm not that worried about it.

mobileben commented 5 years ago

Those flags are perfectly safe and are dependent on use case. Visibility relates to whether or not the declaration is visible to other modules or not (corresponding to "external linkage").

While perhaps not overly "common", at least for me ... common enough where I've run into this professionally a few times. It can happen when you are integrating pre-built libraries or even if you pre-build yourself.

https://developer.apple.com/library/archive/documentation/DeveloperTools/Conceptual/CppRuntimeEnv/Articles/SymbolVisibility.html

I should also note that I've run for quite some time with warnings like that. So on the other hand, I don't fully know if there are bad side effects from the warning. But given the number of external libs I use, I can end up with close to 100 warnings related to this type of issue (not from boost, but other libs as well).

faithfracture commented 5 years ago

I merged in your PR. Would you please close this if that solves the issue?

mobileben commented 5 years ago

Thanks, I'm reviewing right now by building it. I have been using this to build:

./boost.sh -ios --boost-version 1.67.0 --min-ios-version 9.3 --ios-archs "armv7 armv7s arm64 arm64e" --hidden-visibility --universal --no-framework

I'm noticing right now it won't build. I'm seeing this

Invent missing headers cp: /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator12.2.sdk/usr/include/crt_externs.h: No such file or directory

(BTW, I just saw https://github.com/faithfracture/Apple-Boost-BuildScript/issues/21 where this file is missing and you indicate there is no issue with the file missing)

and inevitably

Building Boost for iPhoneSimulator Error staging iPhoneSimulator. Check log.

This could be an Xcode thing (I'm on 10.2.1).

Currently reviewing this.

mobileben commented 5 years ago

Okay, so the culprit looks like this:

darwin.compile.c++ iphonesim-build/boost/bin.v2/libs/thread/build/darwin-12.2~iphonesim/release/architecture-x86/link-static/macosx-version-iphonesim-12.2/target-os-iphone/threadapi-pthread/threading-multi/pthread/thread.o In file included from libs/thread/src/pthread/thread.cpp:17: In file included from ./boost/thread/once.hpp:26: In file included from ./boost/thread/pthread/once_atomic.hpp:20: In file included from ./boost/atomic.hpp:12: In file included from ./boost/atomic/atomic.hpp:20: In file included from ./boost/atomic/fences.hpp:21: In file included from ./boost/atomic/detail/operations.hpp:17: In file included from ./boost/atomic/detail/operations_lockfree.hpp:21: In file included from ./boost/atomic/detail/ops_gcc_atomic.hpp:24: ./boost/atomic/detail/ops_gcc_x86_dcas.hpp:408:16: error: address argument to atomic builtin cannot be const-qualified ('const volatile boost::atomics::detail::gcc_dcas_x86_64::storage_type ' (aka 'const volatile unsigned __int128 ') invalid) return __sync_val_compare_and_swap(&storage, value, value); ^ ~~~~ 1 error generated.

https://github.com/boostorg/atomic/issues/15

I've tried building for 1.69.

But now get this:

fatal error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/lipo: x86_64/libboost_atomic.a and i386/libboost_atomic.a have the same architectures (x86_64) and can't be in the same fat output file

Aborted: Lipo failed

What I am noticing is that if I lipo -info one of the .a (eg. libboost_atomic.a) in build/x86_64 and build/i386

I get, respectively

racerx:ben:x86_64 $ lipo -info libboost_atomic.a 
Architectures in the fat file: libboost_atomic.a are: x86_64 i386 
racerx:ben:i386 $ lipo -info libboost_atomic.a 
Architectures in the fat file: libboost_atomic.a are: x86_64 i386

Any thoughts on this one?

faithfracture commented 5 years ago

Bah. Yeah, that probably has to do with the change I just pushed for #39. Poop. SO, that is coming from trying to lipo the iOS Simulator binaries. I only build with armv7 & arm64, so I'd bet your additional armv7s and arm64e archs the simulator binaries are probably getting double built or something like that. I tried to be smart about choosing the simulator architectures based on the iOS ones, but that obviously didn't work. I'll see if I can figure it out in the next day or so.

Unless you really need the armv7s and arm64e archs right now you could probably get away with excluding them for the time being.

mobileben commented 5 years ago

Let me test your theory later today about omitting armv7s and arm64e

. Right now I'm still on 1.67 and have a stable checked in binary for that version. So I'm not impacted by this right now. Although I'll probably move to 1.69 soon. I would just need to test it up against some other libs I'm using that require boost.

I'll keep you posted once I run that test of omitting those slices.

faithfracture commented 5 years ago

I can run ./boost.sh -ios --boost-version 1.69.0 on my machine (works for tvOS & macOS as well), so definitely let me know if it doesn't work for you. There could be some other conflict with the other flags that I'll have to sort out, which would be a pain 👎

mobileben commented 5 years ago

I can indeed build using your default 1.69.0 build as you specified.

It looks like I narrowed down the culprit. I was using my theoretical "optimal" flags which would include --universal, --no-framework, --hidden-visibility, --min-ios-version, and --ios-archs.

I've gone through permutations of setting the flags.

There seem to be 2 issues floating around.

  1. Setting --min-ios-version to 9.3.
  2. Using armv7 or armv7s

At least in my case, I can go to 10.0. However even this

./boost.sh -ios --boost-version 1.69.0 --hidden-visibility --universal --no-framework --min-ios-version 10.0

Won't build. Curiously enough, even though I'm using the default ios archs, it complains about armv7 in this output

fatal error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/lipo: armv7/libboost_atomic.a and arm64/libboost_atomic.a have the same architectures (armv7) and can't be in the same fat output file

Even this fails

./boost.sh -ios --boost-version 1.69.0 --min-ios-version 10.0

Explicitly setting --min-ios-version to 11.0 works of course, as that's the default.

The main one I'm testing with is

./boost.sh -ios --boost-version 1.69.0 --hidden-visibility --universal --no-framework --min-ios-version 10.0 --ios-archs "armv7 armv7s arm64 arm64e"

It always results with

fatal error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/lipo: x86_64/libboost_atomic.a and i386/libboost_atomic.a have the same architectures (x86_64) and can't be in the same fat output file

Also curious from the output are these snippets

Decomposing libboost_atomic.a Unpacking /Users/ben/Development/3rdParty/Apple-Boost-BuildScript/build/boost/1.69.0/ios/release/build/armv7/obj/atomic Unpacking /Users/ben/Development/3rdParty/Apple-Boost-BuildScript/build/boost/1.69.0/ios/release/build/armv7s/obj/atomic Unpacking /Users/ben/Development/3rdParty/Apple-Boost-BuildScript/build/boost/1.69.0/ios/release/build/arm64/obj/atomic Unpacking /Users/ben/Development/3rdParty/Apple-Boost-BuildScript/build/boost/1.69.0/ios/release/build/arm64e/obj/atomic Unpacking /Users/ben/Development/3rdParty/Apple-Boost-BuildScript/build/boost/1.69.0/ios/release/build/x86_64/obj/atomic ar: ../../libboost_atomic.a is a fat file (use libtool(1) or lipo(1) and ar(1) on it) ar: ../../libboostatomic.a: Inappropriate file type or format mv: rename *.o to atomic*.o: No such file or directory

Archiving atomic ...ios-armv7 ...ios-armv7s ...ios-arm64 ...ios-arm64e ...ios-sim-x86_64 ar: obj/atomic/.o: No such file or directory ...ios-sim-i386 ar: obj/atomic/.o: No such file or directory

thomasdao commented 5 years ago

Hi, I was building Boost with --hidden-visibility flag for both iOS and Mac, and while it builds ok, somehow all warnings are still there. I'm not so sure if this happens for you all. My build commands are below:

Build for iOS:
./boost.sh -ios --boost-version 1.70.0 --boost-libs "date_time regex system thread" --hidden-visibility

Build for Mac:
./boost.sh -macos --boost-version 1.70.0 --boost-libs "date_time regex system thread" --min-macos-version 10.11 --no-framework --hidden-visibility
faithfracture commented 5 years ago

Well lame. Looks like I totally broke some stuff.

This is on my radar, but I'm way in the middle of something else for work so I probably won't have time to fix this until next week. Hopefully that doesn't mess any of y'all up too much.

velthune commented 5 years ago

@faithfracture have you solved something? Script fails when passing --ios-archs "armv7" ):

faithfracture commented 5 years ago

Sorry, no :( . I haven't had time yet. Been working on something critical for the real job (since that kinda takes priority). I don't remember if it worked before the recent change. You could try checking out the previous commit & just going with that in the meantime.

faithfracture commented 5 years ago

@velthune What is the full command you are using? The current script doesn't support simply running ./boost.sh -ios --ios-archs "armv7". In the default configuration, the min iOS version is 11, which only supports 64-bit architectures. If you want to build with --ios-archs "armv7" you'll need to run ./boost.sh -ios --min-ios-version 10.0 --ios-archs "armv7", or whatever minimum iOS version you're targeting. iOS 10 is the last version that supported 32-bit architectures.

faithfracture commented 5 years ago

Wow. Apparently I don't know what I'm doing, lol. Should be fixed now. Sorry about that.

@thomasdao Yeah, I still see those warnings regardless of whether I pass the --hidden-visibility flag. I don't know if it's my project configuration or if it's the Boost build. If it works for @mobileben I don't really care. Thus far it's just been an irritating warning & isn't causing me any problems.