Adding MacOS M1 support.

[metasim]

This adds building libpdaljni.dylib for arm64 architectures.

I have successfully tested this up to the publish workflow:

https://github.com/s22s/PDAL-Java/actions/runs/8282743168

...but I need advice on how to test that stage.

Closes https://github.com/PDAL/java/issues/61

[pomadchin]

Yo Sim! Btw trying it in your fork is the ez way; But I'll click buttons here to assist you as well haha.

[pomadchin]

That's sick! Thanks 🎉

[pomadchin]

I'm going to merge it, and its gonna trigger a snapshot publish.

[pomadchin]

Could you test it with 2.6.2+2-fd27b0cc-SNAPSHOT?

[pomadchin]

Ha, it does not really work on my mac 🤔

scala> System.load("/Users/.../Downloads/test/native/arm64-darwin/libpdaljni.2.6.dylib")
java.lang.UnsatisfiedLinkError: Can't load library: /Users/.../Downloads/test/native/arm64-darwin/libpdaljni.2.6.dylib

That's a dylib from the downloaded natives jar.

Wondering what's wrong with it.

$ file libpdaljni.2.6.dylib
libpdaljni.2.6.dylib: Mach-O 64-bit dynamically linked shared library arm64
$ otool -L libpdaljni.2.6.dylib
libpdaljni.2.6.dylib:
    @rpath/libpdaljni.2.6.dylib (compatibility version 0.0.0, current version 0.0.0)
    @rpath/libpdalcpp.16.dylib (compatibility version 16.0.0, current version 16.3.0)
    /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1600.151.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1336.0.0)

Which maches my locally built dylib as well 🤔

[metasim]

Why would it be referencing itself? @rpath/libpdaljni.2.6.dylib? or is that expected?

[pomadchin]

@metasim the first line is the ID string of the dynamic library, it is expected yea

[pomadchin]

My local otool diff (red - local, green - published via CI)

[metasim]

Output on my machine, which is macos 14.3.1

/Users/sfitch/Downloads/libpdaljni.2.6.dylib:
    @rpath/libpdaljni.2.6.dylib (compatibility version 0.0.0, current version 0.0.0)
    @rpath/libpdalcpp.16.dylib (compatibility version 16.0.0, current version 16.3.0)
    /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1600.151.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1336.0.0)

[pomadchin]

ha, does it work for you?

[pomadchin]

the only diff I see is /usr/lib/libc++.1.dylib 🤔

[metasim]

ha, does it work for you?

I need to convince macos it's safe first 😆

scala> System.load("/Users/sfitch/Downloads/libpdaljni.2.6.dylib")
java.lang.UnsatisfiedLinkError: /Users/sfitch/Downloads/libpdaljni.2.6.dylib: dlopen(/Users/sfitch/Downloads/libpdaljni.2.6.dylib, 0x0001): tried: '/Users/sfitch/Downloads/libpdaljni.2.6.dylib' (code signature in <4D31D938-9728-3052-ADBC-9F374E36F4D9> '/Users/sfitch/Downloads/libpdaljni.2.6.dylib' not valid for use in process: library load disallowed by system policy), '/System/Volumes/Preboot/Cryptexes/OS/Users/sfitch/Downloads/libpdaljni.2.6.dylib' (no such file), '/Users/sfitch/Downloads/libpdaljni.2.6.dylib' (code signature in <4D31D938-9728-3052-ADBC-9F374E36F4D9> '/Users/sfitch/Downloads/libpdaljni.2.6.dylib' not valid for use in process: library load disallowed by system policy)
  at java.base/jdk.internal.loader.NativeLibraries.load(Native Method)
  at java.base/jdk.internal.loader.NativeLibraries$NativeLibraryImpl.open(NativeLibraries.java:388)
  at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:232)
  at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:174)
  at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2389)
  at java.base/java.lang.Runtime.load0(Runtime.java:755)
  at java.base/java.lang.System.load(System.java:1953)
  ... 35 elided

Edit: use sudo xattr -d com.apple.quarantine ./libpdaljni.2.6.dylib

[pomadchin]

Oh man that's it; ha; weird so weird; I guess it's macos version diff

[metasim]

@pomadchin It's been a number of years since I've done RPATH wrangling.... any thoughts on why this wouldn't work?

❯ ls /opt/homebrew/lib/libpdalcpp.*
/opt/homebrew/lib/libpdalcpp.16.3.0.dylib /opt/homebrew/lib/libpdalcpp.16.dylib     /opt/homebrew/lib/libpdalcpp.dylib

❯ DYLD_LIBRARY_PATH=/opt/homebrew/lib scala

...

scala> System.load("/Users/sfitch/Downloads/libpdaljni.2.6.dylib")
java.lang.UnsatisfiedLinkError: /Users/sfitch/Downloads/libpdaljni.2.6.dylib: dlopen(/Users/sfitch/Downloads/libpdaljni.2.6.dylib, 0x0001): Library not loaded: @rpath/libpdalcpp.16.dylib

  Referenced from: <4D31D938-9728-3052-ADBC-9F374E36F4D9> /Users/sfitch/Downloads/libpdaljni.2.6.dylib

  Reason: tried: '/Users/runner/miniconda3/envs/pdal-java/lib/libpdalcpp.16.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/runner/miniconda3/envs/pdal-java/lib/libpdalcpp.16.dylib' (no such file), '/Users/runner/miniconda3/envs/pdal-java/lib/libpdalcpp.16.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/runner/miniconda3/envs/pdal-java/lib/libpdalcpp.16.dylib' (no such file), '/Users/sfitch/.sdkman/candidates/java/17.0.6-tem/lib/server/./libpdalcpp.16.dylib' (no such file), '/Users/sfitch/.sdkman/candidates/java/17.0.6-tem/lib/server/../libpdalcpp.16.dylib' (no such file), '/Users/sfitch/.sdkman/candidates/java/17.0.6-tem/bin/./libpdalcpp.16.dylib' (no such file), '/Users/sfitch/.sdkman/candidates/java/17.0.6-tem/bin/

[metasim]

Oh man that's it; ha; weird so weird; I guess it's macos version diff

They only have M1 builders for MacOS 14. You must be on 13?

[pomadchin]

Nah, I meant more like some new security stuff; wondering why x86 builds are fine and arm is not.

About libpdalcpp.16.dylib mb its not there? I'd reinstall pdal / try the new conda env with pdal 2.6.3 😃

[metasim]

Looks like the bound RPATH is to the runner's conda environment?

❯ otool -l libpdaljni.2.6.dylib | grep -A2 LC_RPATH
          cmd LC_RPATH
      cmdsize 56
         path /Users/runner/miniconda3/envs/pdal-java/lib (offset 12)

I'm assuming there's a way to get it to link without using @rpath.

[pomadchin]

yea, so it should be good; most likely you just really don't have a file its trying to find / or its somehwere else.

[metasim]

Not sure why it worked on x86 (older macos?), but I think I need to figure out how to not have it expect the library to be installed in /Users/runner/miniconda3/envs/pdal-java/lib. I'll work on it.

[pomadchin]

ha really?

just install pdal into the conda env; btw I think that's how conda envs work on mac; TBH locally I use PDAL via conda only 🤷

[pomadchin]

ok google for a bit, we need to sign apple binaries; that could be the arm thing, but ideally should be done with x86 binaries as well.

[metasim]

Note to self: https://gitlab.kitware.com/cmake/community/-/wikis/doc/cmake/RPATH-handling#default-rpath-settings

[pomadchin]

btw smth is not loading for me locally even with xattr -d com.apple.quarantine; I'll look into it as well. I originally thought it works but eh ):

Another thing is that signature is $$ (dev account). ): Hopefully will be able to find some alternative solution that works.

[metasim]

btw smth is not loading for me locally

What version of MacOS are you on? If it's < 14, I think it's because the M1 builder runs on 14, and I need to find a setting to be backward compatible.

[pomadchin]

14.1.2

[metasim]

Tangent: https://gregoryszorc.com/blog/2022/08/08/achieving-a-completely-open-source-implementation-of-apple-code-signing-and-notarization/ via https://news.ycombinator.com/item?id=32386762

[pomadchin]

Moved signature stuff into https://github.com/PDAL/java/issues/90