search

godotengine / godot

Godot Engine – Multi-platform 2D and 3D game engine
https://godotengine.org
MIT License
90.89k stars 21.14k forks source link

Godot build with summator shared library failed. #56887

Open Devilsparta opened 2 years ago

Devilsparta commented 2 years ago

Godot version

3.4.dev(dfdc2b3)

System information

MacOS

Issue description

My godot build with summator failed when I'm building it a shared library.

I'm using the 3.4 branch. The building enviroment is MacOS. I can build it to static library, but cannot build it a shared library.

My summator is exactly the same with the tutrial.

modules/summator
├── SCsub
├── __pycache__
│   └── config.cpython-310.pyc
├── config.py
├── register_types.cpp
├── register_types.h
├── register_types.os
├── register_types.osx.opt.tools.64.o
├── summator.cpp
├── summator.h
├── summator.os
└── summator.osx.opt.tools.64.o

and my SCsub file is:

Import('env')

sources = [
    "register_types.cpp",
    "summator.cpp"
]

# First, create a custom env for the shared library.
module_env = env.Clone()

# Position-independent code is required for a shared library.
module_env.Append(CCFLAGS=['-fPIC'])

# Don't inject Godot's dependencies into our shared library.
module_env['LIBS'] = []

# Define the shared library. By default, it would be built in the module's
# folder, however it's better to output it into `bin` next to the
# Godot binary.
shared_lib = module_env.SharedLibrary(target='#bin/summator', source=sources)

# Finally, notify the main build environment it now has our shared library
# as a new dependency.

# LIBPATH and LIBS need to be set on the real "env" (not the clone)
# to link the specified libraries to the Godot executable.

env.Append(LIBPATH=['#bin'])

# SCons wants the name of the library with it custom suffixes
# (e.g. ".x11.tools.64") but without the final ".so".
shared_lib_shim = shared_lib[0].name.rsplit('.', 1)[0]
env.Append(LIBS=[shared_lib_shim])

My terminal of scons command output is:

➜  godot git:(3.4) scons target=release_debug platform=osx -j6
scons: Reading SConscript files ...
Building for macOS 10.12+, platform x86-64.
Checking for C header file mntent.h... (cached) no
scons: done reading SConscript files.
scons: Building targets ...
[ 77%] Linking Shared Library ==> bin/libsummator.osx.opt.tools.64.dylib
[ 80%] Undefined symbols for architecture x86_64:
  "_global_lock()", referenced from:
      void ClassDB::register_class<Summator>() in register_types.os
  "_global_unlock()", referenced from:
      void ClassDB::register_class<Summator>() in register_types.os
  "_err_print_error(char const*, char const*, int, char const*, ErrorHandlerType)", referenced from:
      void ClassDB::register_class<Summator>() in register_types.os
      MethodBind1<int>::call(Object*, Variant const**, int, Variant::CallError&) in summator.os
      MethodBind0::call(Object*, Variant const**, int, Variant::CallError&) in summator.os
      MethodBind0RC<int>::call(Object*, Variant const**, int, Variant::CallError&) in summator.os
  "postinitialize_handler(Object*)", referenced from:
      Object* ClassDB::creator<Summator>() in register_types.os
  "D_METHOD(char const*)", referenced from:
      Summator::_bind_methods() in summator.os
  "D_METHOD(char const*, char const*)", referenced from:
      Summator::_bind_methods() in summator.os
  "MethodBind::_set_const(bool)", referenced from:
      MethodBind* create_method_bind<Summator, int>(void (Summator::*)(int)) in summator.os
      MethodBind* create_method_bind<Summator>(void (Summator::*)()) in summator.os
      MethodBind0RC<int>::MethodBind0RC() in summator.os
  "MethodBind::_set_returns(bool)", referenced from:
      MethodBind0RC<int>::MethodBind0RC() in summator.os
  "MethodBind::_generate_argument_types(int)", referenced from:
      MethodBind* create_method_bind<Summator, int>(void (Summator::*)(int)) in summator.os
      MethodBind* create_method_bind<Summator>(void (Summator::*)()) in summator.os
      MethodBind0RC<int>::MethodBind0RC() in summator.os
  "MethodBind::MethodBind()", referenced from:
      MethodBind* create_method_bind<Summator, int>(void (Summator::*)(int)) in summator.os
      MethodBind* create_method_bind<Summator>(void (Summator::*)()) in summator.os
      MethodBind0RC<int>::MethodBind0RC() in summator.os
  "MethodBind::~MethodBind()", referenced from:
      MethodBind* create_method_bind<Summator, int>(void (Summator::*)(int)) in summator.os
      MethodBind1<int>::~MethodBind1() in summator.os
      MethodBind1<int>::~MethodBind1() in summator.os
      MethodBind* create_method_bind<Summator>(void (Summator::*)()) in summator.os
      MethodBind0::~MethodBind0() in summator.os
      MethodBind0::~MethodBind0() in summator.os
      MethodBind0RC<int>::MethodBind0RC() in summator.os
      ...
  "StringName::StringName(char const*)", referenced from:
      Summator::_get_property_listv(List<PropertyInfo, DefaultAllocator>*, bool) const in summator.os
      Reference::_get_property_listv(List<PropertyInfo, DefaultAllocator>*, bool) const in summator.os
  "StringName::StringName(String const&)", referenced from:
      void ClassDB::register_class<Summator>() in register_types.os
      void ClassDB::_add_class<Summator>() in register_types.os
      void ClassDB::_add_class<Reference>() in register_types.os
      Summator::_get_class_namev() const in summator.os
      MethodBind* create_method_bind<Summator, int>(void (Summator::*)(int)) in summator.os
      PropertyInfo::PropertyInfo(Variant::Type, String, PropertyHint, String const&, unsigned int, StringName const&) in summator.os
      MethodBind* create_method_bind<Summator>(void (Summator::*)()) in summator.os
      ...
  "StringName::StringName()", referenced from:
      Summator::Summator() in summator.os
      Summator::Summator() in summator.os
      Summator::_get_property_listv(List<PropertyInfo, DefaultAllocator>*, bool) const in summator.os
      MethodBind* create_method_bind<Summator, int>(void (Summator::*)(int)) in summator.os
      MethodBind1<int>::_gen_argument_type_info(int) const in summator.os
      GetTypeInfo<int, void>::get_class_info() in summator.os
      PropertyInfo::PropertyInfo(Variant::Type, String, PropertyHint, String const&, unsigned int, StringName const&) in summator.os
      ...
  "StringName::~StringName()", referenced from:
      void ClassDB::register_class<Summator>() in register_types.os
      void ClassDB::_add_class<Summator>() in register_types.os
      void ClassDB::_add_class<Reference>() in register_types.os
      Summator::_bind_methods() in summator.os
      Summator::_get_property_listv(List<PropertyInfo, DefaultAllocator>*, bool) const in summator.os
      Summator::_get_class_namev() const in summator.os
      Summator::~Summator() in summator.os
      ...
  "StringName::operator=(StringName const&)", referenced from:
      Summator::_get_class_namev() const in summator.os
      MethodBind* create_method_bind<Summator, int>(void (Summator::*)(int)) in summator.os
      PropertyInfo::PropertyInfo(Variant::Type, String, PropertyHint, String const&, unsigned int, StringName const&) in summator.os
      MethodBind* create_method_bind<Summator>(void (Summator::*)()) in summator.os
      MethodBind* create_method_bind<Summator, int>(int (Summator::*)() const) in summator.os
      PropertyInfo::operator=(PropertyInfo const&) in summator.os
  "Memory::free_static(void*, bool)", referenced from:
      void ClassDB::register_class<Summator>() in register_types.os
      void ClassDB::_add_class<Summator>() in register_types.os
      void ClassDB::_add_class<Reference>() in register_types.os
      Summator::_bind_methods() in summator.os
      Summator::_get_property_listv(List<PropertyInfo, DefaultAllocator>*, bool) const in summator.os
      Summator::_get_class_namev() const in summator.os
      MethodBind* create_method_bind<Summator, int>(void (Summator::*)(int)) in summator.os
      ...
  "Memory::alloc_static(unsigned long, bool)", referenced from:
      DefaultAllocator::alloc(unsigned long) in summator.os
  "Object::call_multilevel(StringName const&, Variant const**, int)", referenced from:
      vtable for Summator in summator.os
  "Object::initialize_class()", referenced from:
      void ClassDB::register_class<Summator>() in register_types.os
      Summator::_initialize_classv() in summator.os
  "Object::_changed_callback(Object*, char const*)", referenced from:
      vtable for Summator in summator.os
  "Object::call_multilevel_reversed(StringName const&, Variant const**, int)", referenced from:
      vtable for Summator in summator.os
  "Object::call(StringName const&, Variant const**, int, Variant::CallError&)", referenced from:
      vtable for Summator in summator.os
  "Object::setvar(Variant const&, Variant const&, bool*)", referenced from:
      vtable for Summator in summator.os
  "Object::to_string()", referenced from:
      vtable for Summator in summator.os
  "String::String(char const*)", referenced from:
      void ClassDB::register_class<Summator>() in register_types.os
      void ClassDB::_add_class<Summator>() in register_types.os
      void ClassDB::_add_class<Reference>() in register_types.os
      Summator::_get_property_listv(List<PropertyInfo, DefaultAllocator>*, bool) const in summator.os
      Summator::_get_class_namev() const in summator.os
      Summator::get_class() const in summator.os
      MethodBind* create_method_bind<Summator, int>(void (Summator::*)(int)) in summator.os
      ...
  "ClassDB::_add_class2(StringName const&, StringName const&)", referenced from:
      void ClassDB::_add_class<Summator>() in register_types.os
      void ClassDB::_add_class<Reference>() in register_types.os
      void ClassDB::_add_class<Summator>() in summator.os
      void ClassDB::_add_class<Reference>() in summator.os
  "ClassDB::bind_methodfi(unsigned int, MethodBind*, MethodDefinition const&, Variant const**, int)", referenced from:
      Summator::_bind_methods() in summator.os
  "ClassDB::get_property_list(StringName, List<PropertyInfo, DefaultAllocator>*, bool, Object const*)", referenced from:
      Summator::_get_property_listv(List<PropertyInfo, DefaultAllocator>*, bool) const in summator.os
      Reference::_get_property_listv(List<PropertyInfo, DefaultAllocator>*, bool) const in summator.os
  "ClassDB::classes", referenced from:
      void ClassDB::register_class<Summator>() in register_types.os
  "Variant::can_convert_strict(Variant::Type, Variant::Type)", referenced from:
      MethodBind1<int>::call(Object*, Variant const**, int, Variant::CallError&) in summator.os
  "Variant::clear()", referenced from:
      MethodBind1<int>::call(Object*, Variant const**, int, Variant::CallError&) in summator.os
      MethodBind0RC<int>::call(Object*, Variant const**, int, Variant::CallError&) in summator.os
  "Variant::Variant(Variant const&)", referenced from:
      MethodBind1<int>::call(Object*, Variant const**, int, Variant::CallError&) in summator.os
      MethodBind0RC<int>::call(Object*, Variant const**, int, Variant::CallError&) in summator.os
  "Variant::Variant(int)", referenced from:
      MethodBind0RC<int>::call(Object*, Variant const**, int, Variant::CallError&) in summator.os
  "Reference::_bind_methods()", referenced from:
      void ClassDB::register_class<Summator>() in register_types.os
      Summator::_initialize_classv() in summator.os
  "Reference::Reference()", referenced from:
      Summator::Summator() in summator.os
      Summator::Summator() in summator.os
  "Reference::~Reference()", referenced from:
      Summator::Summator() in summator.os
      Summator::Summator() in summator.os
      Summator::~Summator() in summator.os
      Summator::~Summator() in summator.os
  "Object::_validate_property(PropertyInfo&) const", referenced from:
      vtable for Summator in summator.os
  "Object::get_argument_options(StringName const&, int, List<String, DefaultAllocator>*) const", referenced from:
      vtable for Summator in summator.os
  "Object::get_translatable_strings(List<String, DefaultAllocator>*) const", referenced from:
      vtable for Summator in summator.os
  "Object::getvar(Variant const&, bool*) const", referenced from:
      vtable for Summator in summator.os
  "String::operator==(char const*) const", referenced from:
      Summator::is_class(String const&) const in summator.os
  "Variant::operator int() const", referenced from:
      MethodBind1<int>::call(Object*, Variant const**, int, Variant::CallError&) in summator.os
  "typeinfo for MethodBind", referenced from:
      typeinfo for MethodBind1<int> in summator.os
      typeinfo for MethodBind0 in summator.os
      typeinfo for MethodBind0RC<int> in summator.os
  "typeinfo for Reference", referenced from:
      typeinfo for Summator in summator.os
  "operator new(unsigned long, void* (*)(unsigned long))", referenced from:
      List<PropertyInfo, DefaultAllocator>::push_back(PropertyInfo const&) in summator.os
  "operator new(unsigned long, char const*)", referenced from:
      Object* ClassDB::creator<Summator>() in register_types.os
      MethodBind* create_method_bind<Summator, int>(void (Summator::*)(int)) in summator.os
      MethodBind* create_method_bind<Summator>(void (Summator::*)()) in summator.os
      MethodBind* create_method_bind<Summator, int>(int (Summator::*)() const) in summator.os
ld: symbol(s) not found for architecture x86_64
[ 80%] clang: error: linker command failed with exit code 1 (use -v to see invocation)
[ 80%] scons: *** [bin/libsummator.osx.opt.tools.64.dylib] Error 1
scons: building terminated because of errors.
[Time elapsed: 00:00:53.897]

Please help me!! If you need any detail information, just replay and I will edit here.

Steps to reproduce

Put project downbelow in godot/modules and run scons target=release_debug platform=osx -j6

Minimal reproduction project

summator.zip

Devilsparta commented 2 years ago

Anyone?

Calinou commented 2 years ago

Anyone?

C++ modules aren't used often (and macOS isn't that popular in the Godot community), so it may take a while to find a solution to this.

cc @gramps

Gramps commented 2 years ago

I've never had any issues with GodotSteam but this seems to be something OSX-specific. I also have no Mac to test it with. @SapphireMH handles that stuff for GodotSteam so he might have an idea why this happens.

SapphireMH commented 2 years ago

We don't run into this particular issue with our own module, so I'd recommend you to compare your config.py and scsub file to ours and see if something there is strange or requires changing.

On first glance I can't see what's wrong but I don't have the time to deep dive into it.

Also please mention the version of MacOS, version of Scons, etc, that you are using, sometimes all it takes is an update to the buildtools to fix confusing errors, I've been there.

https://github.com/Gramps/GodotSteam/tree/master/godotsteam

kiran-kp commented 1 year ago

FYI, I had a similar issue on Linux but the only error I was seeing was .../projects/godot/bin/godot.linuxbsd.editor.x86_64.llvm: symbol lookup error: .../projects/godot/bin/liblisp.linuxbsd.editor.x86_64.llvm.so: undefined symbol: _ZN10MethodBind12_set_returnsEb

My understanding is that this has to do with use of the templated MethodBind class and the template not being instantiated for whatever reason. I resolved my issue by making my module statically linked but having all my actual implementation details in a shared library.

mikejohnstn commented 1 year ago

I'm hitting this too. A few clarifications:

  1. "summator" is the example / tutorial in the Godot docs themselves, so I'd really expect it to work!
  2. In particular it's from the "Improving the build system for development" section that explains how to get a module to work as a shared library for development purposes.
  3. The end goal is to not stare at your screen for 30 seconds every time you compile your module changes (while scons figures out that nothing else has changed).

Latest macOS Ventura, 4.0.3-stable, scons v4.5.2.

mikejohnstn commented 1 year ago

Just tried the summator tutorial on my Windows PC and ran into the same linker errors. I don't have a Linux install to try, but either this tutorial for modules only works on Linux, or something has changed in Godot 4 to prevent this tutorial from working. Or we're all doing something wrong. :)

I looked at Godot's SCsubs for its built-in modules and unfortunately none of them seem to support this shared library approach. Do engine module devs all wait for 30s each time they make a small change?

I also looked at @SapphireMH's SCsub and godotsteam doesn't seem to support this approach either.

Lastly I looked through @Zylann's SCsub and some Discord history, and while I found fellow developers complaining about slow SCons build times for modules, I couldn't find any trace of people who have successfully tried the shared library approach as described in the godot docs.

Unfortunately I'm new to SCons and I'm not sure how to debug this any further. It's truly impressive that something like godot_voxel exists at all with 30s build times for even small module changes.

Calinou commented 1 year ago

The code to build Summator as a shared library wasn't extensively tested on all platforms and is likely broken in its current state in 2023. It was added years ago by @touilleMan in https://github.com/godotengine/godot-docs/commit/927ef4576ccfc9a184728620fe16ae9a46944f50.

I looked at Godot's SCsubs for its built-in modules and unfortunately none of them seem to support this shared library approach.

This is planned to be implemented at some point. See https://github.com/godotengine/godot-proposals/issues/1796.

Do engine module devs all wait for 30s each time they make a small change?

You can optimize the linking setup significantly on Linux by using a different linker such as mold, or get a faster CPU. It can pay for itself over time if you're a professional developer :slightly_smiling_face:

macOS has a commercial version of the mold linker available as sold. You could try its evaluation version for free.

You can speed up SCons itself on macOS using pyston_lite_autoload (see https://www.pyston.org/). That said, SCons is usually not the largest bottleneck in small incremental changes – it's the linking, which is done by the linker that's written in C/C++.

mikejohnstn commented 1 year ago

Thanks for your reply and suggestions. Good to know support for shared libraries is planned.

SCons is usually not the largest bottleneck in small incremental changes – it's the linking

On an M1 I'm seeing about 10s spent just in the scons: Building targets ... phase where you see a percentage number going up, and then ~5s linking. This is after an incremental change to a small module. So your pyston suggestion could be a good one in my case.

Calinou commented 1 year ago

On an M1 I'm seeing about 10s spent just in the scons: Building targets ... phase where you see a percentage number going up, and then ~5s linking

Does it improve if you use the progress=no SCons option? I've found that progress reporting can slightly slow down the build process, especially on Windows with its slow console I/O.

mikejohnstn commented 1 year ago

Does it improve if you use the progress=no SCons option?

Unfortunately there's no improvement by hiding progress.

I'm leaning toward just trying to use GDExtension if at all possible even though it's still a little raw, since a shared library workflow would be valuable to me if I can figure out how to make it do most of what I need to do.

Thanks again for your suggestions.

Redwarx008 commented 1 year ago 
scons: Reading SConscript files ...
Auto-detected 16 CPU cores available for build parallelism. Using 15 cores by default. You can override it with the -j argument.
Found MSVC version 14.3, arch x86_64
Building for platform "windows", architecture "x86_64", target "editor".
NOTE: Developer build, with debug optimization level and debug symbols (unless overridden).
Checking for C header file mntent.h... (cached) no
scons: done reading SConscript files.
scons: Building targets ...
[ 50%] Linking Static Library modules\module_summator.windows.editor.dev.x86_64.lib ...
[ 96%] Linking Shared Library bin\summator.windows.editor.dev.x86_64.dll ...
[100%] progress_finish(["progress_finish"], [])
[100%] Linking Program bin\godot.windows.editor.dev.x86_64.exe ...
[100%] Linking Program bin\godot.windows.editor.dev.x86_64.console.exe ...
register_types.windows.editor.dev.x86_64.obj : error LNK2019: unresolved external symbol "void __cdecl _global_lock(void)" (?_global_lock@@YAXXZ) referenced in function "public: static void __cdecl ClassDB::register_class<class Summator>(bool)" (??$register_class@VSummator@@@ClassDB@@SAX_N@Z) 
register_types.windows.editor.dev.x86_64.obj : error LNK2019: unresolved external symbol "void __cdecl _global_unlock(void)" (?_global_unlock@@YAXXZ) referenced in function "public: static void __cdecl ClassDB::register_class<class Summator>(bool)" (??$register_class@VSummator@@@ClassDB@@SAX_N@Z)
register_types.windows.editor.dev.x86_64.obj : error LNK2019: unresolved external symbol "void __cdecl _err_print_error(char const *,char const *,int,char const *,bool,enum ErrorHandlerType)" (?_err_print_error@@YAXPEBD0H0_NW4ErrorHandlerType@@@Z) referenced in function "public: static void __cdecl ClassDB::register_class<class Summator>(bool)" (??$register_class@VSummator@@@ClassDB@@SAX_N@Z)
register_types.windows.editor.dev.x86_64.obj : error LNK2019: unresolved external symbol "public: static void __cdecl Memory::free_static(void *,bool)" (?free_static@Memory@@SAXPEAX_N@Z) referenced in function "public: static void __cdecl ClassDB::_add_class<class RefCounted>(void)" (??$_add_class@VRefCounted@@@ClassDB@@SAXXZ)

I just tried it on windows and it also failed, maybe we should mark this feature on the documentation as deprecated so people don't waste their time.