Plugin does not work under Apple Silicon (M1-ARM64) due to a lack of correct binaries

[cocona20xx]

Godot Engine v3.3.2.stable.official - https://godotengine.org
OpenGL ES 3.0 Renderer: Apple M1
OpenGL ES Batching: ON

--- Debugging process stopped ---
Toggle Visible
Toggle Visible
Set map_file
 Can't open dynamic library: /Users/cocona20xx/Desktop/Qodot-testing/addons/qodot/bin/osx/libqodot.dylib, error: dlopen(/Users/cocona20xx/Desktop/Qodot-testing/addons/qodot/bin/osx/libqodot.dylib, 2): no suitable image found.  Did find:
    /Users/cocona20xx/Desktop/Qodot-testing/addons/qodot/bin/osx/libqodot.dylib: mach-o, but wrong architecture
    /Users/cocona20xx/Desktop/Qodot-testing/addons/qodot/bin/osx/libqodot.dylib: stat() failed with errno=60.
 modules/gdnative/gdnative.cpp:502 - No valid library handle, can't get symbol from GDNative object
 modules/gdnative/nativescript/nativescript.cpp:1482 - No nativescript_init in "res://addons/qodot/bin/osx/libqodot.dylib" found
 res://addons/qodot/src/nodes/qodot_map.gd:144 - Attempt to call function 'new' in base 'NativeScript' on a null instance.
 modules/gdnative/gdnative.cpp:407 - No valid library handle, can't terminate GDNative object

Qodot ver: 1.6.4 Godot Engine ver already listed in log macOS version: macOS Big Sur 11.3 This is a pretty common issue plaguing a lot of plugins that use gdnative-related tools. i'd be willing to help test a mac ARM build, fyi.

[Shfty]

I suppose that figures. Wouldn't be Apple without breaking things every few years.

This is a pretty common issue plaguing a lot of plugins that use gdnative-related tools.

This is expected. Given the present novelty factor and price point for Apple Silicon, only a fraction of those projects are likely to have maintainers with access to it.

I've attempted to update the Travis CI builds for both libmap and libqodot to XCode 12.5 to support Intel + ARM64 universal binaries, but some part of the process is breaking under this configuration. The thrown error originates inside a deeply-nested python function that appears to relate to the brew / scons setup process rather than GDNative, making it a non-trivial breakage.

i'd be willing to help test a mac ARM build, fyi.

I'd hope that you're willing to compile one as well. I'm not able to test locally as I don't own any M1 hardware, and iterating on an Apple-tooled build environment using nothing but remote CI is a slow and painful process that I'm not particularly eager to undertake.

It would be better if someone with direct access to the affected hardware could figure out the appropriate setup steps and submit a pull request against the .travis.yml files inside the libqodot and libmap repos.

[cocona20xx]

I can look into it if you can give me the steps used normally for compiling it.

[benbeshara]

I've just opened PR's in the relevant repos - the way Travis does the brew/python/scons setup means it now takes 15-20 minutes(!) to build for macOS regardless of if you just do the x86 build or the fat binary. Switching to something like CMake would probably improve this a lot, but if we're sticking close to Godot it's probably not the best idea to throw a different build system into the mix.

[benbeshara]

Just bumping to see if you had the time to look at the PR re: this yet (#134) @Shfty

[benbeshara]

I have the current libraries built for Intel and Apple Silicon mac's available here if anyone's looking for them, I'll try to keep the plugin bundle up to date

[TrueSgtMonkey]

@benbeshara Dude this worked. Thank you so much. Just used it to build a map and it is working swimmingly

[benbeshara]

@TrueSgtMonkey no problem :) glad I could help!

[steezeburger]

@benbeshara can you explain a bit how you built them so we can build ourselves and you don't have to keep them up to date?

I'm having trouble building libmap even, but I don't have a ton of experience with C and its tooling.

[benbeshara]

@steezeburger Since it's all written in C it's portable - you just need to update the build architecture in the SConstruct files in libmap and libqodot (so rather than having the macOS compile flags be env.Append(CCFLAGS = ['-g','-O2','-arch', 'x86_64']) you'd have env.Append(CCFLAGS = ['-g','-O2','-arch', 'x86_64','-arch','arm64']) for a fat binary, or env.Append(CCFLAGS = ['-g','-O2','-arch','arm64']) for just an apple silicon build. I hope that helps!

[steezeburger]

@benbeshara thanks! This lets me know I'm heading in the right direction. I actually got that far, but when I tried to build libmap, I get an error related to clang:

qodot-plugin/libqodot/libmap (●)$  scons -c platform=osx arm=yes; scons platform=osx arm=yes                                                                                                   [11:39:48]
scons: Reading SConscript files ...
*********** Building for arm ***********
scons: done reading SConscript files.
scons: Cleaning targets ...
Removed build/geo_generator.os
Removed build/map_data.os
Removed build/map_parser.os
Removed build/matrix.os
Removed build/surface_gatherer.os
Removed build/vector.os
scons: done cleaning targets.
scons: Reading SConscript files ...
*********** Building for arm ***********
scons: done reading SConscript files.
scons: Building targets ...
gcc -o build/geo_generator.os -c -g -O2 -arch arm64 -fPIC -Isrc/h src/c/geo_generator.c
gcc -o build/map_data.os -c -g -O2 -arch arm64 -fPIC -Isrc/h src/c/map_data.c
gcc -o build/map_parser.os -c -g -O2 -arch arm64 -fPIC -Isrc/h src/c/map_parser.c
gcc -o build/matrix.os -c -g -O2 -arch arm64 -fPIC -Isrc/h src/c/matrix.c
gcc -o build/surface_gatherer.os -c -g -O2 -arch arm64 -fPIC -Isrc/h src/c/surface_gatherer.c
gcc -o build/vector.os -c -g -O2 -arch arm64 -fPIC -Isrc/h src/c/vector.c
gcc -o build/libmap.dylib -arch arm64 -install_name @loader_path/libmap.dylib -dynamiclib build/geo_generator.os build/map_data.os build/map_parser.os build/matrix.os build/surface_gatherer.os build/vector.os
Undefined symbols for architecture arm64:
  "_entities", referenced from:
      _geo_generator_run in geo_generator.os
      _generate_brush_vertices in geo_generator.os
      _geo_generator_print_entities in geo_generator.os
      _geo_generator_get_brush_vertex_count in geo_generator.os
      _geo_generator_get_brush_index_count in geo_generator.os
      _map_data_reset in map_data.os
      _map_data_set_spawn_type_by_classname in map_data.os
      ...
     (maybe you meant: _map_data_print_entities, _map_data_get_entities , _geo_generator_get_entities , _geo_generator_print_entities )
  "_entity_count", referenced from:
      _geo_generator_run in geo_generator.os
      _geo_generator_print_entities in geo_generator.os
      _map_data_reset in map_data.os
      _map_data_set_spawn_type_by_classname in map_data.os
      _map_data_get_entity_count in map_data.os
      _map_data_print_entities in map_data.os
      _commit_entity in map_parser.os
      ...
     (maybe you meant: _map_data_get_entity_count)
  "_entity_geo", referenced from:
      _geo_generator_run in geo_generator.os
      _generate_brush_vertices in geo_generator.os
      _geo_generator_print_entities in geo_generator.os
      _geo_generator_get_entities in geo_generator.os
      _geo_generator_get_brush_vertex_count in geo_generator.os
      _geo_generator_get_brush_index_count in geo_generator.os
      _map_data_reset in map_data.os
      ...
  "_texture_count", referenced from:
      _map_data_reset in map_data.os
      _map_data_register_worldspawn_layer in map_data.os
      _map_data_find_texture in map_data.os
      _map_data_register_texture in map_data.os
      _map_data_set_texture_size in map_data.os
      _map_data_get_texture_count in map_data.os
      _map_data_get_texture in map_data.os
      ...
     (maybe you meant: _map_data_get_texture_count)
  "_textures", referenced from:
      _map_data_reset in map_data.os
      _map_data_register_worldspawn_layer in map_data.os
      _map_data_find_texture in map_data.os
      _map_data_register_texture in map_data.os
      _map_data_set_texture_size in map_data.os
      _map_data_get_textures in map_data.os
      _map_data_get_texture in map_data.os
      ...
     (maybe you meant: _map_data_get_textures)
  "_worldspawn_layer_count", referenced from:
      _map_data_reset in map_data.os
      _map_data_register_worldspawn_layer in map_data.os
      _map_data_find_worldspawn_layer in map_data.os
      _map_data_get_worldspawn_layer_count in map_data.os
      _surface_gatherer_filter_brush in surface_gatherer.os
      _surface_gatherer_run in surface_gatherer.os
     (maybe you meant: _map_data_get_worldspawn_layer_count)
  "_worldspawn_layers", referenced from:
      _map_data_reset in map_data.os
      _map_data_register_worldspawn_layer in map_data.os
      _map_data_find_worldspawn_layer in map_data.os
      _map_data_get_worldspawn_layers in map_data.os
      _surface_gatherer_filter_brush in surface_gatherer.os
      _surface_gatherer_run in surface_gatherer.os
     (maybe you meant: _map_data_get_worldspawn_layers, _filter_worldspawn_layers )
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
scons: *** [build/libmap.dylib] Error 1
scons: building terminated because of errors.

Here are my changes to libmap/SConstruct:

opts.Add(BoolVariable('arm', "Build for mac m1 arm architecture.", 'no'))

...

if env['platform'] == "osx":
    if env['arm']:
        print('*********** Building for arm ***********')
        archFlags = ['-arch', 'arm64']
        env.Append(CCFLAGS = ['-g','-O2'] + archFlags)
        env.Append(LINKFLAGS = archFlags)
        # env.Append(CXX = 'clang++')
    else:
        # previous code

I also tried messing with the CXX flag based on some googling, but to no avail.

It's definitely not a big deal. I have been using your built version successfully, even in Godot 3.5! So thank you very much for that.

[benbeshara]

@steezeburger no problem :) I have some free time this week, I'll update my branches and see if I can get a new build up - it's been a while so I don't know if anything much has changed, but I'll comment back here if I get it working

[tinogoehlert]

Hi!, i ran into the same problem (Macbook air M1 2020) and changed multiple SConstruct files:

Sconstruct
libqodot/Sconstruct
libqodot/libmap/Sconstruct

i added the following flag to CCFLAGS and LINKFLAGS for osx:

    env.Append(CCFLAGS=['-g', '-O2', '-arch', 'x86_64', '-arch', 'arm64'])
    env.Append(LINKFLAGS=['-arch', 'x86_64', '-arch', 'arm64'])

this generated fat (universal) libs that worked for me. the root Sconstruct file might not be important, but i simply added the flags on every place where i found arch related buildflags ^^

oh, i also checked out tag v1.7.3 because i had problems resolving the submodules from master.

hope this helps :-)