Support MacOS - Githubissues

clemens-tolboom commented 3 years ago

Trying to compile on MacOS I still fail due to lack of C++ routine. Help welcome

Prepare

checkout PR

git clone git@github.com:boku-ilen/geodot-plugin.git
cd geodot-plugin
git fetch origin pull/33/head:feature/support-macos
git checkout feature/support-macos

Install GDal using homebrew

brew install gdal

Get path of GDAL install

brew info gdal | grep '/usr/local' | cut -d ' ' -f 1

'normal' compile prepare steps

git submodule update --init --recursive

Build GDNative support

cd godot-cpp
time scons generate_bindings=yes --jobs=$(sysctl -n hw.logicalcpu)
cd ..

Build final code

Run scons using osgeo_path and either debug or release scons target=debug osgeo_path=/usr/local/Cellar/gdal/3.2.1 scons target=release osgeo_path=/usr/local/Cellar/gdal/3.2.1

Check the result as addons must be filled

git status
find demo/addons/geodot/osx/

Run test scene

Assume you have shell access just type

cd demo
godot GeopackageDemo.tscn

Cleanup

To test rebuilding remove all files generated.

git clean -dfX
git reset --hard
git clean -fd demo/

clemens-tolboom commented 3 years ago

Running

scons --platform=osx

results in

scons: Reading SConscript files ...
note: scons_compiledb could not be imported. To be able to use the `--compiledb=` flag, please run `pip3 install scons-compiledb`.
scons: Reading SConscript files ...
OSGeo paths are invalid!
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
g++ -o Feature.os -c -fPIC -I/usr/local/Cellar/gdal/3.2.1/include -I. -I/Users/clemens/Downloads/godot/geodot-plugin/src/global Feature.cpp
In file included from Feature.cpp:1:
./Feature.h:18:26: warning: defaulted function definitions are a C++11 extension [-Wc++11-extensions]
    virtual ~Feature() = default;
                         ^
./Feature.h:32:32: warning: in-class initialization of non-static data member is a C++11 extension [-Wc++11-extensions]
    GeometryType geometry_type = NONE;
                               ^
Feature.cpp:14:10: warning: 'auto' type specifier is a C++11 extension [-Wc++11-extensions]
    for (auto &&oField : *feature) {
         ^
Feature.cpp:14:15: warning: rvalue references are a C++11 extension [-Wc++11-extensions]
    for (auto &&oField : *feature) {
              ^
Feature.cpp:14:24: warning: range-based for loop is a C++11 extension [-Wc++11-extensions]
    for (auto &&oField : *feature) {
                       ^
Feature.cpp:14:26: error: cannot use incomplete type 'OGRFeature' as a range
    for (auto &&oField : *feature) {
                         ^
./Feature.h:8:7: note: forward declaration of 'OGRFeature'
class OGRFeature;
      ^
Feature.cpp:22:19: error: member access into incomplete type 'OGRFeature'
    return feature->GetFieldAsString(name);
                  ^
./Feature.h:8:7: note: forward declaration of 'OGRFeature'
class OGRFeature;
      ^
5 warnings and 2 errors generated.
scons: *** [Feature.os] Error 1
scons: building terminated because of errors.
FileNotFoundError: [Errno 2] No such file or directory: 'src/raster-tile-extractor/build/libRasterTileExtractor.dylib':
  File "/Users/clemens/Downloads/godot/geodot-plugin/SConstruct", line 168:
    copyfile(os.path.join(rte_libpath, rte_library) + lib_file_ending, os.path.join(env['target_path'], rte_library) + lib_file_ending)
  File "/Users/clemens/lib/miniconda3/lib/python3.7/shutil.py", line 120:
    with open(src, 'rb') as fsrc:

kb173 commented 3 years ago

Thanks again for your work on this! In addition to defines.h, you'll need to add an #elif __APPLE__ branch to the GDAL includes in these files:

GeoRaster.cpp
RasterTileExtractor.cpp
Feature.cpp
LineFeature.cpp
PointFeature.cpp
PolygonFeature.cpp
VectorExtractor.cpp

Unfortunately I don't know how the #include should look on Mac, but I'm guessing it might be the same as on Linux. Could you try that and report back?

Once that is done, it should compile, unless there's hidden OS-specific code which only causes problems on Mac, not between Windows and Linux.

clemens-tolboom commented 3 years ago

With currrent PR I can build some code. But src/raster-tile-extractor/build/libRasterTileExtractor.dylib is not build.

scons --platform=osx --osgeo_path=/usr/local/Cellar/gdal/3.2.1

gives

scons: Reading SConscript files ...
note: scons_compiledb could not be imported. To be able to use the `--compiledb=` flag, please run `pip3 install scons-compiledb`.
scons: Reading SConscript files ...
OSGeo paths are invalid!
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
scons: `.' is up to date.
scons: done building targets.

of which

FileNotFoundError: [Errno 2] No such file or directory: 'src/raster-tile-extractor/build/libRasterTileExtractor.dylib':

puzzles me

  File "/Users/clemens/Downloads/godot/geodot-plugin/SConstruct", line 168:
    copyfile(os.path.join(rte_libpath, rte_library) + lib_file_ending, os.path.join(env['target_path'], rte_library) + lib_file_ending)
  File "/Users/clemens/lib/miniconda3/lib/python3.7/shutil.py", line 120:
    with open(src, 'rb') as fsrc:

clemens-tolboom commented 3 years ago

I can now compile.

[x] #38 Why are both SConstruct files different for windows? diff src/vector-extractor/SConstruct src/raster-tile-extractor/SConstruct
[x] #39 The linux includes all <gdal/... ... can't we make this uniform? grep --recursive --after=2 __unix__ .
[x] #40 What is env.Append(LINKFLAGS=['/WHOLEARCHIVE']) and should osx have that too?
[ ] More?

clemens-tolboom commented 3 years ago

Following the steps above I see one warning

g++ -o VectorExtractor.os -c -std=c++17 -fPIC -I/usr/local/Cellar/gdal/3.2.1/include -I. -I/Users/clemens/Downloads/godot/geodot-plugin/src/global VectorExtractor.cpp
VectorExtractor.cpp:208:5: warning: ignoring return value of function declared with 'warn_unused_result' attribute [-Wunused-result]
    square_layer->CreateFeature(square_feature);
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~
1 warning generated.

and 1 warning which should be a fat error

g++ -o demo/addons/geodot/osx/libgeodot.dylib -arch x86_64 -dynamiclib src/gdlibrary.os src/geodata.os src/geodot.os src/geofeatures.os src/geoimage.os -Lgodot-cpp/bin -Lsrc/raster-tile-extractor/build -Lsrc/vector-extractor/build -Llib -lgodot-cpp.osx.debug.64 -lRasterTileExtractor -lVectorExtractor -lgdal
ld: warning: directory not found for option '-Llib'

clemens-tolboom commented 3 years ago

Running scons target=d osgeo_path=/usr/local/Cellar/gdal/3.2.1 solved building.

Godot

ERROR: open_dynamic_library:
  Can't open dynamic library:
    /Users/clemens/Downloads/godot/geodot-plugin/demo/addons/geodot/osx/libgeodot.dylib
  , error: dlopen(/Users/clemens/Downloads/godot/geodot-plugin/demo/addons/geodot/osx/libgeodot.dylib, 2):
 Library not loaded: build/libRasterTileExtractor.dylib
  Referenced from: /Users/clemens/Downloads/godot/geodot-plugin/demo/addons/geodot/osx/libgeodot.dylib
  Reason: image not found.
   At: platform/osx/os_osx.mm:1937.
ERROR: get_symbol: No valid library handle, can't get symbol from GDNative object
   At: modules/gdnative/gdnative.cpp:501.
ERROR: init_library: No nativescript_init in "res://addons/geodot/osx/libgeodot.dylib" found
   At: modules/gdnative/nativescript/nativescript.cpp:1506.
ERROR: _create_autoload: Script does not inherit a Node: res://addons/geodot/geodot.gdns.
   At: editor/editor_autoload_settings.cpp:357.

kb173 commented 3 years ago

Very nice! Regarding the Windows builds and env.Append(LINKFLAGS=['/WHOLEARCHIVE']), I'll ping @MathiasBaumgartinger - can you try removing that line (line 42) from the SConstruct script in raster-tile-extractor and check if it still compiles on Windows?

The whole Windows build pipeline was a result of a lot of trial and error, so it's possible that the script includes things that we ended up not actually needing.

Regarding the second point, yeah, we could put all those GDAL includes into one file which we include everywhere else. That way we have all the ugly OS guards in one place.

Is the call producing ld: warning: directory not found for option '-Llib' generated by SCons? If so, could it be because of lines 151 to 153 in the SConstruct script? These:

elif env['platform'] == 'osx':
    env.Append(LIBS=['libgdal.dylib'])
    env.Append(LIBPATH=[os.path.join(env['osgeo_path'], "lib")])

I think the lines above were a pretty Windows-specific fix, and the "lib" parameter seems like it could produce the option '-Llib' mentioned in the error.

Regarding your last comment: Does build/libRasterTileExtractor.dylib exist, and did it build without warnings?

kb173 commented 3 years ago

Just a guess: Could it be that it's expecting libRasterTileExtractor.dylib to be in a directory called build for some reason, but it's actually in the same directory as libgeodot.dylib?

MathiasBaumgartinger commented 3 years ago

@kb173 Does compile without that line for me too! An important note however is that for me to work i need to build for release or i end up having lots of errors. Done via an extra CL-arg: target=release

clemens-tolboom commented 3 years ago

I guess I should have created new issues for my remarks ... not sure what comment is what now :( See new issues #38, #39,#40

clemens-tolboom commented 3 years ago

Just a guess: Could it be that it's expecting libRasterTileExtractor.dylib to be in a directory called build for some reason, but it's actually in the same directory as libgeodot.dylib?

So we should use the copied files?

https://stackoverflow.com/questions/59687933/linking-a-dynamic-library-that-links-in-symbols-from-a-static-library-macos-vs

Moving the files did not help :-(

clemens-tolboom commented 3 years ago

Hmmm ... moving in the demo root makes running the scene crash :-)

Registered camera FaceTime HD Camera (Display) with id 4 position 0 at index 3
handle_crash: Program crashed with signal 11
Dumping the backtrace. Please include this when reporting the bug on https://github.com/godotengine/godot/issues
[1] 1   libsystem_platform.dylib            0x00007fff203abd7d _sigtramp + 29
[2] OGRODSDriverIdentify(GDALOpenInfo*)
[3] RasterTileExtractor::get_tile_from_dataset(GDALDataset*, double, double, double, int, int)
[4] godot::GeoRasterLayer::get_image(double, double, double, int, int)
[5] void godot::_WrappedMethod<godot::GeoRasterLayer, godot::Ref<godot::GeoImage>, double, double, double, int, int>::apply<0, 1, 2, 3, 4>(godot::Variant*, godot::GeoRasterLayer*, godot::Variant**, godot::__Sequence<0, 1, 2, 3, 4>)
[6] godot_variant godot::__wrapped_method<godot::GeoRasterLayer, godot::Ref<godot::GeoImage>, double, double, double, int, int>(void*, void*, void*, int, godot_variant**)
[7] NativeScriptInstance::call(StringName const&, Variant const**, int, Variant::CallError&) (in Godot) + 401
[8] Object::call(StringName const&, Variant const**, int, Variant::CallError&) (in Godot) + 144
[9] Variant::call_ptr(StringName const&, Variant const**, int, Variant*, Variant::CallError&) (in Godot) + 111
[10] GDScriptFunction::call(GDScriptInstance*, Variant const**, int, Variant::CallError&, GDScriptFunction::CallState*) (in Godot) + 148207
[11] GDScriptInstance::call_multilevel(StringName const&, Variant const**, int) (in Godot) + 183
[12] Node::_notification(int) (in Godot) + 1578
[13] Spatial::_notificationv(int, bool) (in Godot) + 49
[14] Object::notification(int, bool) (in Godot) + 19
[15] SceneTree::_notify_group_pause(StringName const&, int) (in Godot) + 704
[16] SceneTree::idle(float) (in Godot) + 599
[17] Main::iteration() (in Godot) + 800
[18] OS_OSX::run() (in Godot) + 169
[19] main (in Godot) + 529
[20] 20  libdyld.dylib                       0x00007fff20382621 start + 1
[21] 21  ???                                 0x000000000000000a 0x0 + 10
-- END OF BACKTRACE --

clemens-tolboom commented 3 years ago

Comparing Windows/Linux/Mac directory content only Linux has https://github.com/boku-ilen/geodot-plugin/blob/master/demo/addons/geodot/x11/libgdal.so

So I guess we cannot build a game and deploy it on Windows or Mac unless that machine has GDAL installed themselves?

clemens-tolboom commented 3 years ago

We could try manual copy the library from /usr/local/Cellar/gdal/3.2.1/lib/libgdal.dylib

kb173 commented 3 years ago

Yeah, I just copied it manually for the Linux deployment as well (this is described to some extent in the Packaging steps in the README). We didn't push the DLLs for Windows, since we weren't sure which are needed and they're fairly huge altogether. We should probably revisit this, so I opened up an issue: #42

Are you still stuck on the handle_crash: Program crashed with signal 11 crash above atm?

clemens-tolboom commented 3 years ago

I'm still stuck but are now rebuilding to double check steps taken.

clemens-tolboom commented 3 years ago

I tried various locations. Not I have a script to run Godot 3.2 beta5

cd demo/
gd 5 --editor # wrong file locations
mkdir build
mv addons/geodot/osx/libRasterTileExtractor.dylib addons/geodot/osx/libVectorExtractor.dylib build/
gd 5 --editor # file locations OK
gd 5 # main scene crashes
cp /usr/local/Cellar/gdal/3.2.1/lib/libgdal.dylib .
gd 5 # main scene crashes
cp libgdal.dylib addons/geodot/osx/
gd 5 # main scene crashes
cp addons/geodot/osx/libgdal.dylib build/
gd 5 # main scene crashes

I have no other clues

kb173 commented 3 years ago

Is the 'main scene crash' the same in all those variants? What exactly is the error message?

I just realized that it's possible that one of them is actually mostly working, and the reason for the crash is that the paths aren't pointing to valid geodata. I'll try to find something simple to test with, one moment -

kb173 commented 3 years ago

Okay, please try downloading this small sample: http://www.geopackage.org/data/sample1_2.gpkg

And open the GeopackageDemo.tscn scene. There, you just have to adapt the "Path" to point to this file - when you've done that and run the scene, you should some things printed into the Godot logs.

clemens-tolboom commented 3 years ago

    - 'VirtualXPath'    [XML Path Language - XPath]
handle_crash: Program crashed with signal 11
ERROR: notification: NativeScriptInstance detected crash on method: get_dataset
   At: modules/gdnative/nativescript/nativescript.cpp:749.
Dumping the backtrace. Please include this when reporting the bug on https://github.com/godotengine/godot/issues
[1] 1   libsystem_platform.dylib            0x00007fff203abd7d _sigtramp + 29
[2] 2   libsystem_malloc.dylib              0x00007fff201927aa tiny_free_no_lock + 1116
[3] 3   libsqlite3.0.dylib                  0x0000000126c936c7 sqlite3VdbeMemSetStr + 109
[4] 4   libsqlite3.0.dylib                  0x0000000126cdae27 sqlite3Select + 992
[5] 5   libsqlite3.0.dylib                  0x0000000126cd1f25 yy_reduce + 3218
[6] 6   libsqlite3.0.dylib                  0x0000000126c99c5d sqlite3RunParser + 720
[7] 7   libsqlite3.0.dylib                  0x0000000126cccc04 sqlite3Prepare + 380
[8] 8   libsqlite3.0.dylib                  0x0000000126c98f9f sqlite3LockAndPrepare + 176
[9] 9   libsqlite3.0.dylib                  0x0000000126c982e2 sqlite3_exec + 176
[10] 10  libsqlite3.0.dylib                  0x0000000126c9933a sqlite3_get_table + 209
[11] SQLQuery(sqlite3*, char const*, SQLResult*)
[12] GDALGeoPackageDataset::CheckUnknownExtensions(bool)
[13] GDALGeoPackageDataset::Open(GDALOpenInfo*)
[14] OGRGeoPackageDriverOpen(GDALOpenInfo*)
[15] 15  libgdal.28.dylib                    0x0000000124b72f92 GDALOpenEx + 1690
[16] NativeDataset::NativeDataset(char const*)
[17] NativeDataset::NativeDataset(char const*)
[18] VectorExtractor::open_dataset(char const*)
[19] godot::GeoDataset::load_from_file(godot::String)
[20] godot::Geodot::get_dataset(godot::String)
[21] void godot::_WrappedMethod<godot::Geodot, godot::Ref<godot::GeoDataset>, godot::String>::apply<0>(godot::Variant*, godot::Geodot*, godot::Variant**, godot::__Sequence<0>)
[22] godot_variant godot::__wrapped_method<godot::Geodot, godot::Ref<godot::GeoDataset>, godot::String>(void*, void*, void*, int, godot_variant**)
[23] NativeScriptInstance::call(StringName const&, Variant const**, int, Variant::CallError&) (in Godot) + 401
[24] Object::call(StringName const&, Variant const**, int, Variant::CallError&) (in Godot) + 144
[25] Variant::call_ptr(StringName const&, Variant const**, int, Variant*, Variant::CallError&) (in Godot) + 111
[26] GDScriptFunction::call(GDScriptInstance*, Variant const**, int, Variant::CallError&, GDScriptFunction::CallState*) (in Godot) + 148207
[27] GDScriptInstance::_ml_call_reversed(GDScript*, StringName const&, Variant const**, int) (in Godot) + 183
[28] Object::notification(int, bool) (in Godot) + 19
[29] Node::_propagate_ready() (in Godot) + 171
[30] Node::_propagate_ready() (in Godot) + 107
[31] Node::_set_tree(SceneTree*) (in Godot) + 95
[32] SceneTree::init() (in Godot) + 31
[33] OS_OSX::run() (in Godot) + 48
[34] main (in Godot) + 529
[35] 35  libdyld.dylib                       0x00007fff20382621 start + 1
-- END OF BACKTRACE --

kb173 commented 3 years ago

Oh this is interesting! It seems like all the functions are found and called correctly! It's reporting sqlite3 issues because GeoPackages are essentially sqlite databases. Perhaps on Mac, there's another dependency for GDAL to have GeoPackage support?

clemens-tolboom commented 3 years ago

Note to self: find correct location

find demo -name libgdal.dylib
demo/addons/geodot/osx/libgdal.dylib
demo/libgdal.dylib
demo/build/libgdal.dylib

kb173 commented 3 years ago

Actually it does seem to find the sqlite3 library. Are you sure the Path to the GeoPackage is correct in the demo scene?

clemens-tolboom commented 3 years ago

brew info gdal

gives lots of dependencies ... guess we want a static build?! See #42?

gdal: stable 3.2.1 (bottled), HEAD
Geospatial Data Abstraction Library
https://www.gdal.org/
Conflicts with:
  cpl (because both install cpl_error.h)
/usr/local/Cellar/gdal/3.2.1 (296 files, 57.7MB) *
  Poured from bottle on 2021-01-23 at 14:10:21
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/gdal.rb
License: MIT
==> Dependencies
Build: pkg-config ✔
Required: cfitsio ✔, epsilon ✔, expat ✔, freexl ✔, geos ✔, giflib ✔, hdf5 ✔, jpeg ✔, json-c ✔, libdap ✔, libgeotiff ✔, libpng ✔, libpq ✔, libspatialite ✔, libtiff ✔, libxml2 ✔, netcdf ✔, numpy ✔, openjpeg ✔, pcre ✔, poppler ✔, proj ✔, python@3.9 ✔, sqlite ✔, unixodbc ✔, webp ✔, xerces-c ✔, xz ✔, zstd ✔

clemens-tolboom commented 3 years ago

File exists

GeoDot info
 path          : /Users/clemens/Downloads/godot/sample1_2.gpkg
 layer_name    : countiestbl
 attribute_name: NAME

kb173 commented 3 years ago

FYI, I bundled a small sample geopackage with the GeopackageDemo now. You should now be able to just run that scene without downloading anything or changing any path. Let me know if that helps with your error (unlikely, but it's still one less thing which can go wrong).

clemens-tolboom commented 3 years ago

Look like another codeflow crash

handle_crash: Program crashed with signal 11
ERROR: notification: NativeScriptInstance detected crash on method: get_dataset
   At: modules/gdnative/nativescript/nativescript.cpp:749.
Dumping the backtrace. Please include this when reporting the bug on https://github.com/godotengine/godot/issues
[1] 1   libsystem_platform.dylib            0x00007fff203abd7d _sigtramp + 29
[2] 2   ???                                 0x0000000110aa5080 0x0 + 4574564480
[3] 3   libsqlite3.0.dylib                  0x000000012785f6c7 sqlite3VdbeMemSetStr + 109
[4] 4   libsqlite3.0.dylib                  0x00000001278a6e27 sqlite3Select + 992
[5] 5   libsqlite3.0.dylib                  0x000000012789df25 yy_reduce + 3218
[6] 6   libsqlite3.0.dylib                  0x0000000127865c5d sqlite3RunParser + 720
[7] 7   libsqlite3.0.dylib                  0x0000000127898c04 sqlite3Prepare + 380
[8] 8   libsqlite3.0.dylib                  0x0000000127864f9f sqlite3LockAndPrepare + 176
[9] 9   libsqlite3.0.dylib                  0x00000001278642e2 sqlite3_exec + 176
[10] 10  libsqlite3.0.dylib                  0x000000012786533a sqlite3_get_table + 209
[11] SQLQuery(sqlite3*, char const*, SQLResult*)
[12] GDALGeoPackageDataset::Open(GDALOpenInfo*)
[13] OGRGeoPackageDriverOpen(GDALOpenInfo*)
[14] 14  libgdal.28.dylib                    0x000000012573ef92 GDALOpenEx + 1690
[15] NativeDataset::NativeDataset(char const*)
[16] NativeDataset::NativeDataset(char const*)
[17] VectorExtractor::open_dataset(char const*)
[18] godot::GeoDataset::load_from_file(godot::String)
[19] godot::Geodot::get_dataset(godot::String)
[20] void godot::_WrappedMethod<godot::Geodot, godot::Ref<godot::GeoDataset>, godot::String>::apply<0>(godot::Variant*, godot::Geodot*, godot::Variant**, godot::__Sequence<0>)
[21] godot_variant godot::__wrapped_method<godot::Geodot, godot::Ref<godot::GeoDataset>, godot::String>(void*, void*, void*, int, godot_variant**)
[22] NativeScriptInstance::call(StringName const&, Variant const**, int, Variant::CallError&) (in Godot) + 401
[23] Object::call(StringName const&, Variant const**, int, Variant::CallError&) (in Godot) + 144
[24] Variant::call_ptr(StringName const&, Variant const**, int, Variant*, Variant::CallError&) (in Godot) + 111
[25] GDScriptFunction::call(GDScriptInstance*, Variant const**, int, Variant::CallError&, GDScriptFunction::CallState*) (in Godot) + 147951
[26] GDScriptInstance::_ml_call_reversed(GDScript*, StringName const&, Variant const**, int) (in Godot) + 183
[27] Object::notification(int, bool) (in Godot) + 19
[28] Node::_propagate_ready() (in Godot) + 171
[29] Node::_propagate_ready() (in Godot) + 107
[30] Node::_set_tree(SceneTree*) (in Godot) + 95
[31] SceneTree::init() (in Godot) + 31
[32] OS_OSX::run() (in Godot) + 48
[33] main (in Godot) + 529
[34] 34  libdyld.dylib                       0x00007fff20382621 start + 1
[35] 35  ???                                 0x000000000000000a 0x0 + 10
-- END OF BACKTRACE --

clemens-tolboom commented 3 years ago

https://github.com/boku-ilen/geodot-plugin/pull/33#issuecomment-769214031 looks similar to https://github.com/boku-ilen/geodot-plugin/pull/33#issuecomment-768260810

Weird line is [2] 2 ???

clemens-tolboom commented 3 years ago

Looking for MacOS issues on https://github.com/OSGeo/gdal/issues related to SQLite

clemens-tolboom commented 3 years ago

GDAL code is https://github.com/OSGeo/gdal/blob/69aebed453b7520c3f5a7fd2eed8594f34903077/gdal/ogr/ogrsf_frmts/gpkg/ogrgeopackagedatasource.cpp#L6138

    SQLResult oResultTable;
    OGRErr err = SQLQuery(GetDB(), pszSQL, &oResultTable);
    sqlite3_free(pszSQL);

kb173 commented 3 years ago

Ah nice! It's interesting that you're getting a segfault though (signal 11), that's not something that should happen in GDAL either way... maybe this'll turn into a GDAL bug report, we'll see.

Can you try running the new self-contained RasterDemo? That one works without SQL, so it'd be interesting to know whether that runs without issues - if so, we could really turn to GDAL about this (and maybe build a minimal reproduction project).

clemens-tolboom commented 3 years ago

The code is not touched for years there ... so not sure it's GDAL. It could be a data structure for SQLite.

I did

git log ..258d73a17b1f7e93c9cf9b60a2411545d51879de

to see some memory leaks.

Anyways I was able to compile and got

gdal clemens$ ls -l .libs/
total 1107872
-rwxr-xr-x  1 clemens  staff   21607344 Jan 28 21:05 libgdal.28.dylib
-rw-r--r--  1 clemens  staff  539077656 Jan 28 21:05 libgdal.a
lrwxr-xr-x  1 clemens  staff         16 Jan 28 21:05 libgdal.dylib -> libgdal.28.dylib
lrwxr-xr-x  1 clemens  staff         13 Jan 28 21:05 libgdal.la -> ../libgdal.la
-rw-r--r--  1 clemens  staff       1559 Jan 28 21:05 libgdal.lai

Hope to test that tomorrow with godot :)

kb173 commented 3 years ago

Yeah, probably not... but still, even the possibility of a segfault coming from GDAL or SQLite is somewhat worrying. (Or it just seems that way and it's actually an issue on our side, which I suppose is more likely :D)

Nice, looking forward to testing that!

clemens-tolboom commented 3 years ago

Code is not in mentioned location but in line 1163 GDALGeoPackageDataset::Open and 1461 SQLQuery

clemens-tolboom commented 3 years ago

GDAL location

Trying to cleanup the locations for GDAL as mentioned in https://github.com/boku-ilen/geodot-plugin/pull/33#issuecomment-768262738 lead to the file mentioned in crash [14] 14 libgdal.28.dylib is NOT mine but located /usr/local/Cellar/gdal/3.2.1.

Similar for [10] 10 libsqlite3.0.dylib located /usr/local/Cellar/sqlite/3.34.0 which supports my question about how to distribute a game!?!

Remove files

Just in case some follow along

rm libgdal.dylib 
rm build/libgdal.dylib
rm addons/geodot/osx/libgdal.dylib

clemens-tolboom commented 3 years ago

SQLite

I can just query our database.

sqlite3 demo/geodata/geopackage_sample.gpkg

gives

SQLite version 3.33.0 2020-08-14 13:23:32
Enter ".help" for usage hints.

and

sqlite> SELECT * FROM gpkg_contents WHERE data_type = 'attributes' LIMIT 1;
geopackage_sample|attributes|geopackage_sample||2021-01-27T14:38:18.113Z|||||0

Rechecking (as I still duno what I'm doing) there are more SQLQuery ... between Lines 1163 - 1688 ... which required more GDAL knowlegde I don't have ... OTOH why would it run on linux / windows and not on Mac regarding SQLite!

clemens-tolboom commented 3 years ago

I give up :-(

clemens-tolboom commented 3 years ago

RasterDemo runs ok

clemens-tolboom commented 3 years ago

Are we missing features on homebrew/MacOS disabled

clemens-tolboom commented 3 years ago

Note to self:

gdalinfo --formats
ogrinfo --formats

clemens-tolboom commented 3 years ago

Running gdalinfo GPKG:sample1_2.gpkg:1 gives

gdalinfo GPKG:sample1_2.gpkg:2
    - 'VirtualXPath'    [XML Path Language - XPath]
ERROR 1: Cannot find table '2' in GeoPackage dataset
gdalinfo failed - unable to open 'GPKG:sample1_2.gpkg:2'.

clemens-tolboom commented 3 years ago

Above suggest it's OK but we can also add some probing within Godot later on

gdalinfo --formats | grep GPKG || echo "missing format"

clemens-tolboom commented 3 years ago

Trying to install using brew the master version of gdal

brew search gdal@
==> Formulae
gdal

seems futile as there is only one version somehow while

brew search postgresql@
==> Formulae
postgresql                    postgresql@10                 postgresql@11                 postgresql@12                 postgresql@9.4                postgresql@9.5                postgresql@9.6

--HEAD

This needs the XCode CLT first so run xcode-select --install when needed. If not done gcc is downloaded and compiled (on my machine) which takes ages.

But still Aaargh

brew install gdal --HEAD
==> Downloading https://homebrew.bintray.com/bottles/gcc-10.2.0_3.big_sur.bottle.tar.gz
==> Downloading from https://d29vzk4ow07wi7.cloudfront.net/f02c7193b10ea81529c0246bc196b725b1072e92d383e099d78c22956d29347e?response-content-disposition=attachment%3Bfilename%3D%22gcc-10.2.0_3.big_sur.bottle
######################################################################## 100.0%
==> Cloning https://github.com/OSGeo/gdal.git
Updating /Users/clemens/Library/Caches/Homebrew/gdal--git
==> Checking out branch master
Already on 'master'
Your branch is up to date with 'origin/master'.
HEAD is now at 9339586 Remove too strict CPLAssert checking global hCOAMutex (#3405)
==> Installing dependencies for gdal: gcc
==> Installing gdal dependency: gcc
==> Pouring gcc-10.2.0_3.big_sur.bottle.tar.gz
🍺  /usr/local/Cellar/gcc/10.2.0_3: 1,465 files, 339.5MB
==> Installing gdal --HEAD
==> ./configure --prefix=/usr/local/Cellar/gdal/HEAD-9339586 --mandir=/usr/local/Cellar/gdal/HEAD-9339586/share/man --with-libtool --with-local=/usr/local/Cellar/gdal/HEAD-9339586 --with-opencl --with-thread
Last 15 lines from /Users/clemens/Library/Logs/Homebrew/gdal/01.configure:
--without-idb
--without-ingres
--without-jp2mrsid
--without-kakadu
--without-mrsid
--without-mrsid_lidar
--without-msg
--without-oci
--without-ogdi
--without-podofo
--without-rasdaman
--without-sde
--without-sosi
--with-curl=/usr/bin/curl-config

READ THIS: https://docs.brew.sh/Troubleshooting

Please create pull requests instead of asking for help on Homebrew's GitHub,
Twitter or any other official channels.

clemens-tolboom commented 3 years ago

I will edit all comments later but mweh ... it's getting a mesh on my brew trying to reinstall stable gdal

library (libstdc++ from clang). This may cause problems at runtime.

brew install gdal
==> Downloading https://homebrew.bintray.com/bottles/gdal-3.2.1.big_sur.bottle.1.tar.gz
Already downloaded: /Users/clemens/Library/Caches/Homebrew/downloads/20b10e5dd2665cfdae7bc59caeafb4f3caf01c2a183e965da1256a45b2bdc917--gdal-3.2.1.big_sur.bottle.1.tar.gz
==> Pouring gdal-3.2.1.big_sur.bottle.1.tar.gz
Warning: gdal dependency gcc was built with a different C++ standard
library (libstdc++ from clang). This may cause problems at runtime.
🍺  /usr/local/Cellar/gdal/3.2.1: 296 files, 57.7MB

boku-ilen / geodot-plugin

Support MacOS #33

Prepare

checkout PR

Install GDal using homebrew

Get path of GDAL install

'normal' compile prepare steps

Build GDNative support

Build final code

Check the result as addons must be filled

Run test scene

Cleanup

Godot

GDAL location

Remove files

SQLite

--HEAD