Strange "undefined reference" Android shared library build issue when linking to godot-cpp (on macOS host)

[Sakari369]

Hey, I've been pounding my head against this for many days. What I'm trying to do, is to compile a pretty straightforward GDNative project against godot-cpp for Android. The build works perfectly when compiled natively on macOS and with macOS as the target.

But when building for Android, I was really confused for many days as the same CMakeLists.txt could not be used to produce a working shared library (.so) built for Android arm64-v8 (Oculus Quest 2 target).

I finally managed to figure out that for some freaking reason, initially when I run my make command for my project, with an out of source build made with CMake, first time running the compiler and linker will almost 100% fail.

But get this, when I subsequently run the command again random number of time, it magically manages to work and produce a shared library 🧐

To produce a working shared library, I've noticed that if I run:

make VERBOSE=1 2>&1 | less

Two times, this will result in the compilation to work. But if I run with only make, it will almost always fail .. 😑😬

I have no idea why this might be, as I have never seen this error while building for macOS and testing on macOS purely. I was wondering if anyone here had any idea or had run into the same issue ?

Check out this video describing my process:

https://user-images.githubusercontent.com/1212726/103136659-eab4fd00-46ca-11eb-9aed-ad0dc4bbaea3.mp4

Kinda frustrating! I have no idea why suddenly the build succeeds, as first runs always seem to fail on undefined references. I dont think this should happen, as I'm creating a .so shared library so those symbols would have to be resolved on runtime, not when linking a shared library.

Here is the specific error (cut short):

Scanning dependencies of target gd_omg
[ 22%] Building CXX object CMakeFiles/gd_omg.dir/src/GDPoly.cpp.o
[ 22%] Building CXX object CMakeFiles/gd_omg.dir/src/GDPolyModifier.cpp.o
[ 44%] Building CXX object CMakeFiles/gd_omg.dir/src/GDMath.cpp.o
[ 44%] Building CXX object CMakeFiles/gd_omg.dir/src/GDPolyAnimator.cpp.o
[ 55%] Building CXX object CMakeFiles/gd_omg.dir/src/GDSceneLoader.cpp.o
[ 66%] Building CXX object CMakeFiles/gd_omg.dir/src/gdlibrary.cpp.o
[ 77%] Linking CXX shared library lib/libgd_omg.so
CMakeFiles/gd_omg.dir/src/gdlibrary.cpp.o: In function `godot_gdnative_init':
/Users/sakari/dvl/omg/omg3d/godot/src/gdlibrary.cpp:50: undefined reference to `godot::Godot::gdnative_init(godot_gdnative_init_options*)'
CMakeFiles/gd_omg.dir/src/gdlibrary.cpp.o: In function `godot_gdnative_terminate':
/Users/sakari/dvl/omg/omg3d/godot/src/gdlibrary.cpp:54: undefined reference to `godot::Godot::gdnative_terminate(godot_gdnative_terminate_options*)'
CMakeFiles/gd_omg.dir/src/gdlibrary.cpp.o: In function `godot_nativescript_init':
/Users/sakari/dvl/omg/omg3d/godot/src/gdlibrary.cpp:58: undefined reference to `godot::Godot::nativescript_init(void*)'
CMakeFiles/gd_omg.dir/src/gdlibrary.cpp.o: In function `void godot::register_class<godot::GDPoly>()':
/Users/sakari/dvl/omg/omg3d/godot/godot-cpp/include/core/Godot.hpp:176: undefined reference to `godot::_TagDB::register_type(unsigned long, unsigned long)'
/Users/sakari/dvl/omg/omg3d/godot/godot-cpp/include/core/Godot.hpp:178: undefined reference to `nativescript_api'
/Users/sakari/dvl/omg/omg3d/godot/godot-cpp/include/core/Godot.hpp:178: undefined reference to `nativescript_api'
/Users/sakari/dvl/omg/omg3d/godot/godot-cpp/include/core/Godot.hpp:178: undefined reference to `godot::_RegisterState::nativescript_handle'
/Users/sakari/dvl/omg/omg3d/godot/godot-cpp/include/core/Godot.hpp:178: undefined reference to `godot::_RegisterState::nativescript_handle'
/Users/sakari/dvl/omg/omg3d/godot/godot-cpp/include/core/Godot.hpp:181: undefined reference to `nativescript_1_1_api'
/Users/sakari/dvl/omg/omg3d/godot/godot-cpp/include/core/Godot.hpp:181: undefined reference to `nativescript_1_1_api'
CMakeFiles/gd_omg.dir/src/gdlibrary.cpp.o: In function `void godot::register_class<godot::GDPolyModifier>()':
/Users/sakari/dvl/omg/omg3d/godot/godot-cpp/include/core/Godot.hpp:176: undefined reference to `godot::_TagDB::register_type(unsigned long, unsigned long)'
/Users/sakari/dvl/omg/omg3d/godot/godot-cpp/include/core/Godot.hpp:178: undefined reference to `nativescript_api'
/Users/sakari/dvl/omg/omg3d/godot/godot-cpp/include/core/Godot.hpp:178: undefined reference to `nativescript_api'
/Users/sakari/dvl/omg/omg3d/godot/godot-cpp/include/core/Godot.hpp:178: undefined reference to `godot::_RegisterState::nativescript_handle'
/Users/sakari/dvl/omg/omg3d/godot/godot-cpp/include/core/Godot.hpp:178: undefined reference to `godot::_RegisterState::nativescript_handle'
...

This goes on for many pages, listing all the undefined references.

I have built the godot-cpp in godot-cpp/bin/libgodot-cpp.android.release.arm64v8.a, using the following scons command:

scons target=release generate_bindings=yes platform=android android_api_level=29 android_arch=arm64v8 -j8

I have the android-ndk-r21d installed for macOS for building and I am using the prebuilt toolchain for macOS.

Here is also my CMakeFiles.txt used to create the buildfiles:

project (gd_omg)
set(CMAKE_BUILD_TYPE Debug)
#set(CMAKE_BUILD_TYPE Release)
set(gd_omg_VERSION 0.5)

cmake_minimum_required (VERSION 3.1)

message("PROJECT_BINARY_DIR = " ${PROJECT_BINARY_DIR})
message("PROJECT_SOURCE_DIR = " ${PROJECT_SOURCE_DIR})
message("CMAKE_BINARY_DIR = " ${CMAKE_BINARY_DIR})

include_directories("${PROJECT_SOURCE_DIR}/src")

# Setup conan.
include(${CMAKE_BINARY_DIR}/conanbuildinfo.cmake)
conan_basic_setup()

# Rpath setting, disables warning about this on macOS.
cmake_policy(SET CMP0068 NEW)

list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/cmake")

set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -std=c++17 -fno-exceptions")
set(CMAKE_CXX_FLAGS_RELEASE "-Os")
set(CMAKE_CXX_FLAGS_DEBUG "-g -Og")

set(SOURCES
    src/GDGlobals.cpp
    src/GDMath.cpp
    src/GDPoly.cpp
    src/GDPolyModifier.cpp
    src/GDPolyAnimator.cpp
    src/GDSceneLoader.cpp
    src/gdlibrary.cpp
    ../omg/src/PSILog.cpp
)

set(HEADERS
    src/GDOmg.h
    src/GDGlobals.h
    src/GDMath.h
    src/GDPoly.h
    src/GDPolyModifier.h
    src/GDPolyAnimator.h
    src/GDSceneLoader.h
)

# Build target achitecture dependent stuff
set(OMG_ARCH "macos_x86_64")

set(GODOT_OS "osx")
set(GODOT_ARCH "64")
set(GODOT_RELEASE_TYPE "release")

if (ANDROID)
    set(OMG_ARCH "android_arm8")
    set(GODOT_OS "android")
    set(GODOT_ARCH "arm64v8")

    # Required for android toolchain to find library.
    set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY BOTH)

    message("Building for android")
endif()

# Find OMG library.
set(OMG_ROOT "${CMAKE_CURRENT_SOURCE_DIR}/../omg")
set(OMG_LIB_DIR "${OMG_ROOT}/lib")

find_library(OMG_LIBRARY
    NAME omg
    PATHS ${OMG_LIB_DIR}
    PATH_SUFFIXES ${OMG_ARCH}
    NO_DEFAULT_PATH)

# Find Godot CPP binding libraries.
set(GODOT_CPP_DIR "${CMAKE_CURRENT_SOURCE_DIR}/godot-cpp")

set(GODOT_CPP_LIB_NAME "godot-cpp.${GODOT_OS}.${GODOT_RELEASE_TYPE}.${GODOT_ARCH}")
find_library(GODOT_CPP_LIBRARY
    NAME ${GODOT_CPP_LIB_NAME}
    PATHS ${GODOT_CPP_DIR}
    PATH_SUFFIXES "bin"
    NO_DEFAULT_PATH)

add_library(${PROJECT_NAME} SHARED ${SOURCES})

target_include_directories(${PROJECT_NAME} PUBLIC
    ${OMG_ROOT}/includes
    ${OMG_ROOT}/includes/ext

    ${GODOT_CPP_DIR}/godot_headers
    ${GODOT_CPP_DIR}/include
    ${GODOT_CPP_DIR}/include/core
    ${GODOT_CPP_DIR}/include/gen
)

target_link_libraries(${PROJECT_NAME} ${GODOT_CPP_LIBRARY} ${OMG_LIBRARY} ${CONAN_LIBS})

set(CMAKE_LIBRARY_OUTPUT_DIRECTORY ${PROJECT_SOURCE_DIR}/lib/${OMG_ARCH})

install (TARGETS ${PROJECT_NAME} 
     ARCHIVE DESTINATION ${CMAKE_LIBRARY_OUTPUT_DIRECTORY}
     LIBRARY DESTINATION ${CMAKE_LIBRARY_OUTPUT_DIRECTORY})

Any idea why this could happen? 🙂

[Calinou]

Does it work better if you use the Ninja generator (cmake -GNinja) instead of the default UNIX Makefiles? Make sure to install Ninja on your system beforehand.

[Sakari369]

Does it work better if you use the Ninja generator (cmake -GNinja) instead of the default UNIX Makefiles? Make sure to install Ninja on your system beforehand.

Hey @Calinou, thank you for the quick reply! 🙂 Testing with Ninja, generating the ninja.build with:

cmake -DCMAKE_TOOLCHAIN_FILE={$ANDROID_NDK_ROOT}/build/cmake/android.toolchain.cmake -DANDROID_NDK={$ANDROID_NDK_ROOT} -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-29 -DANDROID_STL=c++_shared -GNinja ..

Results in this error consistently:

[1/1] Linking CXX shared library lib/libgd_omg.so
FAILED: lib/libgd_omg.so
: && /Users/sakari/dvl/android-ndk/android-ndk-r21d/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang++ --target=aarch64-none-linux-android29 --gcc-toolchain=/Users/sakari/dvl/android-ndk/android-ndk-r21d/toolchains/llvm/prebuilt/darwin-x86_64 --sysroot=/Users/sakari/dvl/android-ndk/android-ndk-r21d/toolchains/llvm/prebuilt/darwin-x86_64/sysroot -fPIC -g -DANDROID -fdata-sections -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security     -Wall -std=c++17 -fno-exceptions -g -Og  -Wl,--exclude-libs,libgcc.a -Wl,--exclude-libs,libgcc_real.a -Wl,--exclude-libs,libatomic.a -Wl,--build-id -Wl,--fatal-warnings -Wl,--no-undefined -Qunused-arguments -shared -Wl,-soname,libgd_omg.so -o lib/libgd_omg.so CMakeFiles/gd_omg.dir/src/GDGlobals.cpp.o CMakeFiles/gd_omg.dir/src/GDMath.cpp.o CMakeFiles/gd_omg.dir/src/GDPoly.cpp.o CMakeFiles/gd_omg.dir/src/GDPolyModifier.cpp.o CMakeFiles/gd_omg.dir/src/GDPolyAnimator.cpp.o CMakeFiles/gd_omg.dir/src/GDSceneLoader.cpp.o CMakeFiles/gd_omg.dir/src/gdlibrary.cpp.o CMakeFiles/gd_omg.dir/Users/sakari/dvl/omg/omg3d/omg/src/PSILog.cpp.o  ../godot-cpp/bin/libgodot-cpp.android.release.arm64v8.a  /Users/sakari/dvl/omg/omg3d/omg/lib/android_arm8/libomg.a  -latomic -lm && :
CMakeFiles/gd_omg.dir/src/gdlibrary.cpp.o: In function `godot_gdnative_init':
/Users/sakari/dvl/omg/omg3d/godot/build-android/../src/gdlibrary.cpp:50: undefined reference to `godot::Godot::gdnative_init(godot_gdnative_init_options*)'
CMakeFiles/gd_omg.dir/src/gdlibrary.cpp.o: In function `godot_gdnative_terminate':
/Users/sakari/dvl/omg/omg3d/godot/build-android/../src/gdlibrary.cpp:54: undefined reference to `godot::Godot::gdnative_terminate(godot_gdnative_terminate_options*)'
CMakeFiles/gd_omg.dir/src/gdlibrary.cpp.o: In function `godot_nativescript_init':
/Users/sakari/dvl/omg/omg3d/godot/build-android/../src/gdlibrary.cpp:58: undefined reference to `godot::Godot::nativescript_init(void*)'

With Ninja I cannot get it to build at all. Makes me think that maybe I am doing something wrong here, but make compiling it in some situations make me think otherwise.

Any idea what I could be doing wrong ?

Note: I have not yet had the opportunity to even test this on Android, as I had problem getting the .so included in the Android Template Export APK, so I don't actually know if the shared library built by make sometimes is correct or not.

Basically what I'm wondering here is why does this linking even need to check the references as it's gonna be a .so shared library, shouldn't those be resolved at runtime ?

[Sakari369]

Okay I found out the reason this was happening. I noticed that building the godot-cpp library for Android still used the macOS ranlib command to generate the indexes to the resulting .a archive, resulting in the linking phase not finding the .o files inside the archive.

The SConstruct was missing this line:

env['RANLIB'] = toolchain + "/bin/" + arch_info['tool_path'] + "-ranlib"

After the line 345 in the SConstruct file. After adding this the linking phase works correctly.

Not sure how anyone has been able to build android version of godot-cpp with this, but might be a macOS only issue though ? Not sure.

@Calinou Want me to create pull request for this oneliner fix, or you want to add this to the SConstruct ? Of course would probably need testing on Windows & Linux or at least confirming if it was working there or not before this.

Anyway, happy to finally get this compiling, spent many days trying to figure this out .. at least learned more on this subject.

[Sakari369]

Here is the diff just to make sure:

diff --git a/SConstruct b/SConstruct
index d22c9a3..6b2a487 100644
--- a/SConstruct
+++ b/SConstruct
@@ -343,6 +343,7 @@ elif env['platform'] == 'android':
     env['CC'] = toolchain + "/bin/clang"
     env['CXX'] = toolchain + "/bin/clang++"
     env['AR'] = toolchain + "/bin/" + arch_info['tool_path'] + "-ar"
+    env['RANLIB'] = toolchain + "/bin/" + arch_info['tool_path'] + "-ranlib"

     env.Append(CCFLAGS=['--target=' + arch_info['target'] + env['android_api_level'], '-march=' + arch_info['march'], '-fPIC'])#, '-fPIE', '-fno-addrsig', '-Oz'])
     env.Append(CCFLAGS=arch_info['ccflags'])

[CedNaru]

There is a pending PR already right now which includes this change and more.

330

[Sakari369]

Closing this as fixed in #330