search

PDAL / java

Java extension and bindings for PDAL
https://pdal.io/java.html
Other
8 stars 10 forks source link

MacOs ARM compile-build problems #79

Closed WhiteSte closed 10 months ago

WhiteSte commented 10 months ago

I'm trying to publish my local natives inside a jar, is not my first time in doing that, but this is going to be difficult.

I've found some problems:

Both solution goes into the next error

I'm currently investigating for a solution. Any advice is appreciated

pomadchin commented 10 months ago

Hi @WhiteSte I’ll update these scripts, thanks for noticing it!

Usually there is no need in cross compilation for local needs, and if that’s the case just run sbt publishLocal command (SBT should be installed on the host machine)

pomadchin commented 10 months ago

I used to use these scripts and publish PDAL from the laptop, but since releases moved to CI it got simplified in some sense.

WhiteSte commented 10 months ago

is there something like a sbt publishLocalNative ? I only need the jar containing the mac os arm native

Running sbt publishLocal goes into

o]  delivering ivy file to /Users/steve/Downloads/java-main/core/target/scala-3.3.1/ivy-HEAD+20240119-1403.xml
[info]  published pdal_3 to /Users/steve/.ivy2/local/io.pdal/pdal_3/HEAD+20240119-1403/poms/pdal_3.pom
[info]  published pdal_3 to /Users/steve/.ivy2/local/io.pdal/pdal_3/HEAD+20240119-1403/jars/pdal_3.jar
[info]  published pdal_3 to /Users/steve/.ivy2/local/io.pdal/pdal_3/HEAD+20240119-1403/srcs/pdal_3-sources.jar
[info]  published pdal_3 to /Users/steve/.ivy2/local/io.pdal/pdal_3/HEAD+20240119-1403/docs/pdal_3-javadoc.jar
[info]  published ivy to /Users/steve/.ivy2/local/io.pdal/pdal_3/HEAD+20240119-1403/ivys/ivy.xml
[error] java.lang.RuntimeException: Conflicting cross-version suffixes in: com.github.sbt:sbt-jni-core
[error]     at scala.sys.package$.error(package.scala:30)
[error]     at sbt.librarymanagement.ConflictWarning$.processCrossVersioned(ConflictWarning.scala:39)
[error]     at sbt.librarymanagement.ConflictWarning$.apply(ConflictWarning.scala:19)
[error]     at sbt.Classpaths$.$anonfun$ivyBaseSettings$71(Defaults.scala:3294)
[error]     at scala.Function1.$anonfun$compose$1(Function1.scala:49)
[error]     at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:63)
[error]     at sbt.std.Transform$$anon$4.work(Transform.scala:69)
[error]     at sbt.Execute.$anonfun$submit$2(Execute.scala:283)
[error]     at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:24)
[error]     at sbt.Execute.work(Execute.scala:292)
[error]     at sbt.Execute.$anonfun$submit$1(Execute.scala:283)
[error]     at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
[error]     at sbt.CompletionService$$anon$2.call(CompletionService.scala:65)
[error]     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error]     at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
[error]     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error]     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[error]     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[error]     at java.base/java.lang.Thread.run(Thread.java:833)
[error] (core-scala / update) Conflicting cross-version suffixes in: com.github.sbt:sbt-jni-core
[error] Total time: 4 s, completed 19 gen 2024, 14:03:53
pomadchin commented 10 months ago

@WhiteSte what commands you're experiencing troubles with exactly?

The intended way to publishLocal it is via 

$ sbt +publishLocal

+ is not a typo, it takes care of the cross Scala versions build.

My output:

$ conda-activate pdal-java
(pdal-java) [pdal-java (upd/plugins)]$ sbt +publishLocal
[info] welcome to sbt 1.9.8 (Azul Systems, Inc. Java 11.0.17)
....
[info] loading settings for project root from build.sbt ...
[info] set current project to pdal-jni (in build file:/Users/.../pdal-java/)
[info] Setting Scala version to 2.12.18 on 3 projects.
[info] Excluded 1 projects, run ++ 2.12.18 -v for more details.
[info] Reapplying settings...
[info] set current project to pdal-jni (in build file:/Users/.../pdal-java/)
[info] Wrote /Users/.../pdal-java/core/target/scala-2.12/pdal_2.12-2.6.0+6-6f2649e7-SNAPSHOT.pom
[info] Wrote /Users/.../pdal-java/core-scala/target/scala-2.12/pdal-scala_2.12-2.6.0+6-6f2649e7-SNAPSHOT.pom
[info] Wrote /Users/.../pdal-java/native/target/pdal-native-arm64-darwin-2.6.0+6-6f2649e7-SNAPSHOT.pom
[info] Building library with native build tool CMake
[info] Using CMake version 3.28.1
...
[success] Total time: 3 s, completed Jan 19, 2024, 10:01:30 AM
pomadchin commented 10 months ago

If you want to publish locally native binaries only: sbt native/publishLocal

Also more help around SBT can be found in their docs, and questions about the sbt-jni plugin feel free to ask here / or under the https://github.com/sbt/sbt-jni repo (I'm maintainer there as well)

WhiteSte commented 10 months ago

I could make it work using sbt native/publishLocal , but now the java virtual machine is going into a fatal error whenever it tries to access pdal ..

What i did is to switch inside /.m2/io/pdal/pdal-native/2.6.0 the pdal-native.jar published by you with the one i have just built.. Actually i did the same the last year to make it work the version 2.4.3 (also there arm was not supported)

Going from v2.4.3 to v2.6.2 is going to be a pain for me

Thats the fatal error

`#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x0000000000000000, pid=141, tid=87299
#
# JRE version: OpenJDK Runtime Environment Temurin-17.0.7+7 (17.0.7+7) (build 17.0.7+7)
# Java VM: OpenJDK 64-Bit Server VM Temurin-17.0.7+7 (17.0.7+7, mixed mode, tiered, compressed oops, compressed class ptrs, g1 gc, bsd-aarch64)
# Problematic frame:
# C  0x0000000000000000
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /Users/steve/Desktop/Lavoro/ArchiMethodSA/spacepix/hs_err_pid141.log
#
# If you would like to submit a bug report, please visit:
#   https://github.com/adoptium/adoptium-support/issues
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
Abort trap: 6`
WhiteSte commented 10 months ago

ps: I'm doing everything in java 17 . Shouldn't be a problem i hope

Furthermore, my installed version is pdal v2.6.2, the only one available in nix after 2.6.0

btw, my log when building the native is quite different and contains few errors, even if it finishes


[nix-shell:~/Downloads/java-main]$ sbt native/publishLocal
[info] welcome to sbt 1.9.8 (Eclipse Adoptium Java 17.0.7)
[info] loading settings for project java-main-build from plugins.sbt ...
[info] loading project definition from /Users/steve/Downloads/java-main/project
[info] loading settings for project root from build.sbt ...
[info] set current project to pdal-jni (in build file:/Users/steve/Downloads/java-main/)
[info] Building library with native build tool CMake
[info] Using CMake version 3.27.0
[info] Wrote /Users/steve/Downloads/java-main/native/target/pdal-native-arm64-darwin-HEAD+20240119-1740.pom
[error] CMake Warning:
[error]   Ignoring extra path from command line:
[error]    "/Users/steve/Downloads/java-main/native/target/native/arm64-darwin/build/327"
[info] -- The C compiler identification is Clang 11.1.0
[info] -- The CXX compiler identification is Clang 11.1.0
[info] -- Detecting C compiler ABI info
[info] -- Detecting C compiler ABI info - done
[info] -- Check for working C compiler: /nix/store/lqiqj6s5jx196bpx0mr53d232b57y768-clang-wrapper-11.1.0/bin/clang - skipped
[info] -- Detecting C compile features
[info] -- Detecting C compile features - done
[info] -- Detecting CXX compiler ABI info
[info] -- Detecting CXX compiler ABI info - done
[info] -- Check for working CXX compiler: /nix/store/lqiqj6s5jx196bpx0mr53d232b57y768-clang-wrapper-11.1.0/bin/clang++ - skipped
[info] -- Detecting CXX compile features
[info] -- Detecting CXX compile features - done
[info] -- Found JNI: /Library/Java/JavaVirtualMachines/temurin-17.jdk/Contents/Home/include  found components: AWT JVM 
[info] -- JNI include directories: /Library/Java/JavaVirtualMachines/temurin-17.jdk/Contents/Home/include;/Library/Java/JavaVirtualMachines/temurin-17.jdk/Contents/Home/include/darwin;/Library/Java/JavaVirtualMachines/temurin-17.jdk/Contents/Home/include
[info] -- Configuring done (2.9s)
[info] -- Generating done (0.0s)
[info] -- Build files have been written to: /Users/steve/Downloads/java-main/native/target/native/arm64-darwin/build
[info] [ 20%] Building CXX object CMakeFiles/pdaljni.2.6.dir/Accessors.cpp.o
[info] [ 20%] Building CXX object CMakeFiles/pdaljni.2.6.dir/JavaPipeline.cpp.o
[info] [ 50%] Building CXX object CMakeFiles/pdaljni.2.6.dir/PointViewRawPtr.cpp.o
[info] [ 50%] Building CXX object CMakeFiles/pdaljni.2.6.dir/io_pdal_PointViewIterator.cpp.o
[info] [ 50%] Building CXX object CMakeFiles/pdaljni.2.6.dir/io_pdal_Pipeline.cpp.o
[info] [ 60%] Building CXX object CMakeFiles/pdaljni.2.6.dir/JavaExceptions.cpp.o
[info] [ 70%] Building CXX object CMakeFiles/pdaljni.2.6.dir/io_pdal_PointView.cpp.o
[info] [ 80%] Building CXX object CMakeFiles/pdaljni.2.6.dir/io_pdal_PointLayout.cpp.o
[info] [ 90%] Building CXX object CMakeFiles/pdaljni.2.6.dir/io_pdal_TriangularMesh.cpp.o
[info] [100%] Linking CXX shared library libpdaljni.2.6.dylib
[error] ld: warning: directory not found for option '-L/usr/local/opt/llvm/lib:'
[error] ld: warning: directory not found for option '-L/usr/local/opt/llvm/lib/c++'
[info] [100%] Built target pdaljni.2.6
[info] -- Install configuration: "Release"
[info] -- Installing: /Users/steve/Downloads/java-main/native/target/native/arm64-darwin/bin/./libpdaljni.2.6.dylib
[success] Library built in /Users/steve/Downloads/java-main/native/target/native/arm64-darwin/bin/libpdaljni.2.6.dylib
[info] :: delivering :: io.pdal#pdal-native;HEAD+20240119-1740 :: HEAD+20240119-1740 :: integration :: Fri Jan 19 17:40:18 CET 2024
[info]  delivering ivy file to /Users/steve/Downloads/java-main/native/target/ivy-HEAD+20240119-1740.xml
[info]  published pdal-native to /Users/steve/.ivy2/local/io.pdal/pdal-native/HEAD+20240119-1740/poms/pdal-native.pom
[info]  published pdal-native to /Users/steve/.ivy2/local/io.pdal/pdal-native/HEAD+20240119-1740/jars/pdal-native.jar
[info]  published pdal-native to /Users/steve/.ivy2/local/io.pdal/pdal-native/HEAD+20240119-1740/srcs/pdal-native-sources.jar
[info]  published pdal-native to /Users/steve/.ivy2/local/io.pdal/pdal-native/HEAD+20240119-1740/docs/pdal-native-javadoc.jar
[info]  published ivy to /Users/steve/.ivy2/local/io.pdal/pdal-native/HEAD+20240119-1740/ivys/ivy.xml
[success] Total time: 4 s, completed 19 gen 2024, 17:40:18
pomadchin commented 10 months ago

@WhiteSte what is the sbt core/test result?

WhiteSte commented 10 months ago

the sbt core.test result is the same fatal error

pomadchin commented 10 months ago

Gotcha, I had the same behavior with the "incorrectly" installed PDAL.

What works for me now is PDAL installed via conda, and build (+ test) with the active conda env.

What's your PDAL installation?

WhiteSte commented 10 months ago

my pdal installation is done through nix env, I've installed pdal 2.6.2. I can try with conda, will let you know

WhiteSte commented 10 months ago

I've created a separate env in coda and installed pdal in there with conda install -c conda-forge pdal

Now i've built the natives and executed the sbt core/test the result of the test is success

Then i took the created jar and substituted into the maven directories, i ran my java app and that's the result, where at least it is saying something about C

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x0000000189f7cf88, pid=10640, tid=73987
#
# JRE version: OpenJDK Runtime Environment Temurin-17.0.7+7 (17.0.7+7) (build 17.0.7+7)
# Java VM: OpenJDK 64-Bit Server VM Temurin-17.0.7+7 (17.0.7+7, mixed mode, tiered, compressed oops, compressed class ptrs, g1 gc, bsd-aarch64)
# Problematic frame:
# C  [libc++.1.dylib+0x37f88]  std::__1::istreambuf_iterator<char, std::__1::char_traits<char>> std::__1::num_get<char, std::__1::istreambuf_iterator<char, std::__1::char_traits<char>>>::__do_get_unsigned<unsigned int>(std::__1::istreambuf_iterator<char, std::__1::char_traits<char>>, std::__1::istreambuf_iterator<char, std::__1::char_traits<char>>, std::__1::ios_base&, unsigned int&, unsigned int&) const+0x48
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /Users/steve/Desktop/Lavoro/ArchiMethodSA/spacepix/hs_err_pid10640.log
(javapipeline filters.relaxationdartthrowing Debug) Retaining 
#
# If you would like to submit a bug report, please visit:
#   https://github.com/adoptium/adoptium-support/issues
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
WhiteSte commented 10 months ago

Big Update: if i remove filters.relaxationdartthrowing from the pipeline then it works..... Of course i'm using 5 filters and the bugged one is from my list

hobu commented 10 months ago

There was a bug with filters.relaxationdartthrowing https://github.com/PDAL/PDAL/pull/4184 but I don't know if it is relevant here or you've found a new bug with it.

WhiteSte commented 10 months ago

There was a bug with filters.relaxationdartthrowing PDAL/PDAL#4184 but I don't know if it is relevant here or you've found a new bug with it.

I've took a look on it. I could run pdal translate 2717-1103.las test.las relaxationdartthrowing --filters.relaxationdartthrowing.count=10000 on the dataset without any problem. But executing the same command inside a pipeline


  String json = new JSONObject()
        .put( "pipeline", new JSONArray()
        .put( new JSONObject()
        .put( "type", "readers.las" )
        .put( "filename","/2717-1103.las" )
        .put( "spatialreference", "EPSG:2056" ) ) 
        .put( new JSONObject()
                    .put( "type", "filters.crop" )
                    .put( "point", "POINT(2717706.2 1103466.2 677.54)" )
                    .put( "distance", 250 ) )
        .put( new JSONObject()
                    .put( "type", "filters.relaxationdartthrowing" )
                    .put( "count", 100000 )
                   // .put( "radius", 1) 
                   // .put( "decay", 0.9) 
        )

        ).toString();

It crashes

But if i comment out the filter from the pipeline it works. My deduction is that the bindings are not reflecting the new changes in the native pdal version?

But maybe we need to open another issue for that, for what regard the arm java build it has been solved using conda.

@pomadchin Should we update the documentation regard that? There could be a section in the readme dedicated to the macos -arm build

pomadchin commented 10 months ago

@WhiteSte does the same pipeline work from the command line (the entire json passed as a pipeline input)?

Yea we can add a section if its useful, but it's not really a part of the repo. There are lots of ways to install PDAL (:

WhiteSte commented 10 months ago

@WhiteSte does the same pipeline work from the command line (the entire json passed as a pipeline input)?

Yea we can add a section if its useful, but it's not really a part of the repo. There are lots of ways to install PDAL (:

I can confirm , if i execute the same pipeline from cli it works. I put here the json for reference

{"pipeline":[{"filename":"/Users/steve/Desktop/Lavoro/ArchiMethodSA/spacepix/projects/raw_data/SwissSurface3D/2717-1103.las","type":"readers.las","spatialreference":"EPSG:2056"},{"distance":250,"type":"filters.crop","point":"POINT(2717706.2 1103466.2 677.54)"},{"count":100000,"decay":0.9,"type":"filters.relaxationdartthrowing","radius":1}]}

At this point i'm quite sure that the filter is broken in the java bindings

pomadchin commented 10 months ago

@WhiteSte could you share the .las file to debug / confirm its failing for me as well?

Yes totally could be bindings / bindings build issue.

WhiteSte commented 10 months ago

I can't give you the original file as the size is 900MB

I can give you this sub file that is 300kb, on which is crashing too

I'm asking myself if it crashes on every file

test.las.zip

pomadchin commented 10 months ago

@WhiteSte what is exactly the code with the pipeline you're trying and is failing? I tried with pdal 2.6.2 installed via conda;

val pipeline = Pipeline(json)
pipeline.execute()
println(s"pipeline.getMetadata(): ${pipeline.getMetadata()}")
pipeline.close()

and it prints metadata successfully;

could be the local env / code specific issue

WhiteSte commented 10 months ago

you can try this file, just open it and execute. Remember to change te path to test.las with your path

testLASFile.java.zip

If you comment out 

  // .put( new JSONObject()
        //             .put( "type", "filters.relaxationdartthrowing" )
        //             .put( "count", 100000 )
        //            .put( "radius", 1) 
        //            .put( "decay", 0.9) 
        // )

It works, otherwise not

pomadchin commented 10 months ago

Ok, I was able to reproduce; crashes only with log level >= 3! I'll see what's going on there, could be a non existent log level anymore.

Try setting it to 0, 1, 2 https://github.com/PDAL/java/blob/main/core/src/main/scala/io/pdal/LogLevel.scala#L28-L30

WhiteSte commented 10 months ago

That works! How did you found out that?

pomadchin commented 10 months ago

Just played around with it for a bit;

It looks like it's a PDAL filters.relaxationdartthrowing issue - crashes on a log level >= 3; worth documenting and filing a PDAL issue!

The best is to have a reproducible example in C++ without any jni / py trickery around + a ticket in the PDAL repo.

WhiteSte commented 10 months ago

how did we assumed that is a c++ pdal issue? If you do pdal pipeline pipeline.json --verbose 3 it works

I suppose is a jni related problem

pomadchin commented 10 months ago

I did not assume it's a C++ PDAL issue, I assumed that if it's reproducible in bare PDAL than it's not a bindings issue.

Could be totally a bindings issue (:

pomadchin commented 10 months ago

But TBH I have hard times linking the log level setting affecting filter execution may be a bindings issue. Unless some pointer get lost somehow 🤔

pomadchin commented 10 months ago

Will make a PR shortly, that's indeed a bindings issue. Logger output stream is not properly initialized (?) and it's basically an NPE (??) 100% of times. It refuses to write into std::stringstream which is peculiar.

The only reason it never crashed before is that we were lucky enough to not have any logging statements for the specified levels (?)

It works with stdout / stderr / files.

WhiteSte commented 10 months ago

The only reason it never crashed before is that we were lucky enough to not have any logging statements for the specified levels (?)

That's strange because in v2.4.3 i was using log level 3, but i got the crash just after updating to v2.6.2 I'm trying to take a look but sincerly i'm not really sure where to put my hands on

pomadchin commented 10 months ago

So the way it should work is the following:

The idea is to have logs written into that stream and later on convert it to a string to access.

I tried replacing this constructor with the other one and it worked.

Dropped investigating yesterday in the mids of the code cleanup and figuring out the nature of a crash.

pomadchin commented 10 months ago

Also since 2.4.x there’s been a major refactor of the classes to be compatible with the modern PDAL versions, prior to the change logs collection was done via executor.

WhiteSte commented 10 months ago

good you just solved it so. I've added a little guide in #81 that resume this issue

pomadchin commented 10 months ago

Well it’s not solved yet, any help with debugging is really appreciated! Need still to figure out why it’s unhappy about the stringstream.

WhiteSte commented 10 months ago

never coded in c unfortunately. I'm trying to see but it's hard

Idk if it can help but running it inside a docker went though this

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f76b426c7f5, pid=7, tid=8
#
# JRE version: OpenJDK Runtime Environment (17.0.10+7) (build 17.0.10+7-alpine-r0)
# Java VM: OpenJDK 64-Bit Server VM (17.0.10+7-alpine-r0, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# C  [libpdaljni.2.6.so+0xe7f5]  libpdaljava::PipelineExecutor::setLogStream(std::ostream&)+0xa5
#

in particular C [libpdaljni.2.6.so+0xe7f5] libpdaljava::PipelineExecutor::setLogStream(std::ostream&)+0xa5