search

combust / mleap

MLeap: Deploy ML Pipelines to Production
https://combust.github.io/mleap-docs/
Apache License 2.0
1.5k stars 313 forks source link

Need Help on Contributing towards Mleap for 2.13 #869

Closed prashil1996 closed 1 year ago

prashil1996 commented 1 year ago

Hi Developers,

I've recently forked the main MLeap repository with the intention of contributing to support Scala 2.13. However, I'm unfamiliar with the build process and prerequisites. Could you please provide guidance on the necessary steps to make local changes and build MLeap with Scala 2.13 compatibility?

Thank you for your assistance. Prashil

jsleight commented 1 year ago

Use the make targets in the repo root to build and run tests. Once you have a build that you think works locally, then can commit and push to origin to create a PR. That will trigger a full build on our CI system.

Specifically regarding scala 2.13 compatibility, please aim for cross compiling to support both scala 2.12 and 2.13. sbt has native support for this and the repo should already be set up correctly to utilize it.

prashil1996 commented 1 year ago

@jsleight Thanks for replying, also what are the different requirements for compiling the build locally? Scala, Java, SBT?

I tried building it locally but I was getting some error. Can you help me with some steps? Any wiki?

The following were encountered after running make on the root repo: 

17:38:30.945 [pool-11-thread-20] ERROR ml.dmlc.xgboost4j.java.NativeLibLoader - Failed to load xgboost4j library from jar for platform macos/aarch64
17:38:30.948 [pool-11-thread-20] ERROR ml.dmlc.xgboost4j.java.DMatrix - Failed to load native library
java.io.FileNotFoundException: File /lib/macos/aarch64/libxgboost4j.dylib was not found inside JAR.
    at ml.dmlc.xgboost4j.java.NativeLibLoader.createTempFileFromResource(NativeLibLoader.java:285) ~[xgboost4j_2.12-1.7.6.jar:?]
    at ml.dmlc.xgboost4j.java.NativeLibLoader.loadLibraryFromJar(NativeLibLoader.java:228) ~[xgboost4j_2.12-1.7.6.jar:?]
    at ml.dmlc.xgboost4j.java.NativeLibLoader.initXGBoost(NativeLibLoader.java:168) ~[xgboost4j_2.12-1.7.6.jar:?]
    at ml.dmlc.xgboost4j.java.XGBoostJNI.<clinit>(XGBoostJNI.java:34) ~[xgboost4j_2.12-1.7.6.jar:?]
    at ml.dmlc.xgboost4j.java.DMatrix.<init>(DMatrix.java:69) ~[xgboost4j_2.12-1.7.6.jar:?]
    at ml.dmlc.xgboost4j.scala.DMatrix.<init>(DMatrix.scala:32) ~[xgboost4j_2.12-1.7.6.jar:?]
    at ml.combust.mleap.xgboost.runtime.testing.CachedDatasetUtils.$init$(CachedDatasetUtils.scala:21) ~[test-classes/:?]
    at ml.combust.mleap.xgboost.runtime.XGBoostPredictorClassificationModelParitySpec.<init>(XGBoostPredictorClassificationModelParitySpec.scala:14) ~[test-classes/:?]
    at jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
    at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
    at java.lang.reflect.ReflectAccess.newInstance(ReflectAccess.java:128) ~[?:?]
    at jdk.internal.reflect.ReflectionFactory.newInstance(ReflectionFactory.java:304) ~[?:?]
    at java.lang.Class.newInstance(Class.java:725) ~[?:?]
    at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:454) ~[scalatest-core_2.12-3.2.16.jar:3.2.16]
    at sbt.TestRunner.runTest$1(TestFramework.scala:153) ~[?:?]
    at sbt.TestRunner.run(TestFramework.scala:168) ~[?:?]
    at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.$anonfun$apply$1(TestFramework.scala:336) ~[?:?]
    at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:296) ~[?:?]
    at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336) ~[?:?]
    at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336) ~[?:?]
    at sbt.TestFunction.apply(TestFramework.scala:348) ~[?:?]
    at sbt.Tests$.processRunnable$1(Tests.scala:475) ~[?:?]
    at sbt.Tests$.$anonfun$makeSerial$1(Tests.scala:481) ~[?:?]
    at sbt.std.Transform$$anon$3.$anonfun$apply$2(Transform.scala:47) ~[?:?]
    at sbt.std.Transform$$anon$4.work(Transform.scala:69) ~[?:?]
    at sbt.Execute.$anonfun$submit$2(Execute.scala:283) ~[?:?]
    at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:24) ~[?:?]
    at sbt.Execute.work(Execute.scala:292) ~[?:?]
    at sbt.Execute.$anonfun$submit$1(Execute.scala:283) ~[?:?]
    at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265) ~[?:?]
    at sbt.CompletionService$$anon$2.call(CompletionService.scala:65) ~[?:?]
    at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
    at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
    at java.lang.Thread.run(Thread.java:1583) ~[?:?]
[info] XGBoostPredictorClassificationModelParitySpec:
[info] ml.combust.mleap.xgboost.runtime.XGBoostPredictorClassificationModelParitySpec *** ABORTED ***
[info]   java.lang.ExceptionInInitializerError:
[info]   at ml.dmlc.xgboost4j.java.DMatrix.<init>(DMatrix.java:69)
[info]   at ml.dmlc.xgboost4j.scala.DMatrix.<init>(DMatrix.scala:32)
[info]   at ml.combust.mleap.xgboost.runtime.testing.CachedDatasetUtils.$init$(CachedDatasetUtils.scala:21)
[info]   at ml.combust.mleap.xgboost.runtime.XGBoostPredictorClassificationModelParitySpec.<init>(XGBoostPredictorClassificationModelParitySpec.scala:14)
[info]   at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
[info]   at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
[info]   at java.base/java.lang.reflect.ReflectAccess.newInstance(ReflectAccess.java:128)
[info]   at java.base/jdk.internal.reflect.ReflectionFactory.newInstance(ReflectionFactory.java:304)
[info]   at java.base/java.lang.Class.newInstance(Class.java:725)
[info]   at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:454)
[info]   ...
[info]   Cause: java.lang.RuntimeException: java.io.FileNotFoundException: File /lib/macos/aarch64/libxgboost4j.dylib was not found inside JAR.
[info]   at ml.dmlc.xgboost4j.java.XGBoostJNI.<clinit>(XGBoostJNI.java:37)
[info]   at ml.dmlc.xgboost4j.java.DMatrix.<init>(DMatrix.java:69)
[info]   at ml.dmlc.xgboost4j.scala.DMatrix.<init>(DMatrix.scala:32)
[info]   at ml.combust.mleap.xgboost.runtime.testing.CachedDatasetUtils.$init$(CachedDatasetUtils.scala:21)
[info]   at ml.combust.mleap.xgboost.runtime.XGBoostPredictorClassificationModelParitySpec.<init>(XGBoostPredictorClassificationModelParitySpec.scala:14)
[info]   at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
[info]   at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
[info]   at java.base/java.lang.reflect.ReflectAccess.newInstance(ReflectAccess.java:128)
[info]   at java.base/jdk.internal.reflect.ReflectionFactory.newInstance(ReflectionFactory.java:304)
[info]   at java.base/java.lang.Class.newInstance(Class.java:725)
[info]   ...
[info]   Cause: java.io.FileNotFoundException: File /lib/macos/aarch64/libxgboost4j.dylib was not found inside JAR.
[info]   at ml.dmlc.xgboost4j.java.NativeLibLoader.createTempFileFromResource(NativeLibLoader.java:285)
[info]   at ml.dmlc.xgboost4j.java.NativeLibLoader.loadLibraryFromJar(NativeLibLoader.java:228)
[info]   at ml.dmlc.xgboost4j.java.NativeLibLoader.initXGBoost(NativeLibLoader.java:168)
[info]   at ml.dmlc.xgboost4j.java.XGBoostJNI.<clinit>(XGBoostJNI.java:34)
[info]   at ml.dmlc.xgboost4j.java.DMatrix.<init>(DMatrix.java:69)
[info]   at ml.dmlc.xgboost4j.scala.DMatrix.<init>(DMatrix.scala:32)
[info]   at ml.combust.mleap.xgboost.runtime.testing.CachedDatasetUtils.$init$(CachedDatasetUtils.scala:21)
[info]   at ml.combust.mleap.xgboost.runtime.XGBoostPredictorClassificationModelParitySpec.<init>(XGBoostPredictorClassificationModelParitySpec.scala:14)
[info]   at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
[info]   at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
[info]   ...
[error] java.lang.ExceptionInInitializerError
[error]     at ml.dmlc.xgboost4j.java.DMatrix.<init>(DMatrix.java:69)
[error]     at ml.dmlc.xgboost4j.scala.DMatrix.<init>(DMatrix.scala:32)
[error]     at ml.combust.mleap.xgboost.runtime.testing.CachedDatasetUtils.$init$(CachedDatasetUtils.scala:21)
[error]     at ml.combust.mleap.xgboost.runtime.XGBoostPredictorClassificationModelParitySpec.<init>(XGBoostPredictorClassificationModelParitySpec.scala:14)
[error]     at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
[error]     at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
[error]     at java.base/java.lang.reflect.ReflectAccess.newInstance(ReflectAccess.java:128)
[error]     at java.base/jdk.internal.reflect.ReflectionFactory.newInstance(ReflectionFactory.java:304)
[error]     at java.base/java.lang.Class.newInstance(Class.java:725)
[error]     at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:454)
[error]     at sbt.TestRunner.runTest$1(TestFramework.scala:153)
[error]     at sbt.TestRunner.run(TestFramework.scala:168)
[error]     at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.$anonfun$apply$1(TestFramework.scala:336)
[error]     at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:296)
[error]     at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336)
[error]     at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336)
[error]     at sbt.TestFunction.apply(TestFramework.scala:348)
[error]     at sbt.Tests$.processRunnable$1(Tests.scala:475)
[error]     at sbt.Tests$.$anonfun$makeSerial$1(Tests.scala:481)
[error]     at sbt.std.Transform$$anon$3.$anonfun$apply$2(Transform.scala:47)
[error]     at sbt.std.Transform$$anon$4.work(Transform.scala:69)
[error]     at sbt.Execute.$anonfun$submit$2(Execute.scala:283)
[error]     at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:24)
[error]     at sbt.Execute.work(Execute.scala:292)
[error]     at sbt.Execute.$anonfun$submit$1(Execute.scala:283)
[error]     at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
[error]     at sbt.CompletionService$$anon$2.call(CompletionService.scala:65)
[error]     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
[error]     at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
[error]     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
[error]     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
[error]     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
[error]     at java.base/java.lang.Thread.run(Thread.java:1583)
[error] Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: File /lib/macos/aarch64/libxgboost4j.dylib was not found inside JAR.
[error]     at ml.dmlc.xgboost4j.java.XGBoostJNI.<clinit>(XGBoostJNI.java:37)
[error]     at ml.dmlc.xgboost4j.java.DMatrix.<init>(DMatrix.java:69)
[error]     at ml.dmlc.xgboost4j.scala.DMatrix.<init>(DMatrix.scala:32)
[error]     at ml.combust.mleap.xgboost.runtime.testing.CachedDatasetUtils.$init$(CachedDatasetUtils.scala:21)
[error]     at ml.combust.mleap.xgboost.runtime.XGBoostPredictorClassificationModelParitySpec.<init>(XGBoostPredictorClassificationModelParitySpec.scala:14)
[error]     at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
[error]     at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
[error]     at java.base/java.lang.reflect.ReflectAccess.newInstance(ReflectAccess.java:128)
[error]     at java.base/jdk.internal.reflect.ReflectionFactory.newInstance(ReflectionFactory.java:304)
[error]     at java.base/java.lang.Class.newInstance(Class.java:725)
[error]     at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:454)
[error]     at sbt.TestRunner.runTest$1(TestFramework.scala:153)
[error]     at sbt.TestRunner.run(TestFramework.scala:168)
[error]     at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.$anonfun$apply$1(TestFramework.scala:336)
[error]     at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:296)
[error]     at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336)
[error]     at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336)
[error]     at sbt.TestFunction.apply(TestFramework.scala:348)
[error]     at sbt.Tests$.processRunnable$1(Tests.scala:475)
[error]     at sbt.Tests$.$anonfun$makeSerial$1(Tests.scala:481)
[error]     at sbt.std.Transform$$anon$3.$anonfun$apply$2(Transform.scala:47)
[error]     at sbt.std.Transform$$anon$4.work(Transform.scala:69)
[error]     at sbt.Execute.$anonfun$submit$2(Execute.scala:283)
[error]     at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:24)
[error]     at sbt.Execute.work(Execute.scala:292)
[error]     at sbt.Execute.$anonfun$submit$1(Execute.scala:283)
[error]     at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
[error]     at sbt.CompletionService$$anon$2.call(CompletionService.scala:65)
[error]     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
[error]     at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
[error]     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
[error]     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
[error]     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
[error]     at java.base/java.lang.Thread.run(Thread.java:1583)
[error] Caused by: java.io.FileNotFoundException: File /lib/macos/aarch64/libxgboost4j.dylib was not found inside JAR.
[error]     at ml.dmlc.xgboost4j.java.NativeLibLoader.createTempFileFromResource(NativeLibLoader.java:285)
[error]     at ml.dmlc.xgboost4j.java.NativeLibLoader.loadLibraryFromJar(NativeLibLoader.java:228)
[error]     at ml.dmlc.xgboost4j.java.NativeLibLoader.initXGBoost(NativeLibLoader.java:168)
[error]     at ml.dmlc.xgboost4j.java.XGBoostJNI.<clinit>(XGBoostJNI.java:34)
[error]     at ml.dmlc.xgboost4j.java.DMatrix.<init>(DMatrix.java:69)
[error]     at ml.dmlc.xgboost4j.scala.DMatrix.<init>(DMatrix.scala:32)
[error]     at ml.combust.mleap.xgboost.runtime.testing.CachedDatasetUtils.$init$(CachedDatasetUtils.scala:21)
[error]     at ml.combust.mleap.xgboost.runtime.XGBoostPredictorClassificationModelParitySpec.<init>(XGBoostPredictorClassificationModelParitySpec.scala:14)
[error]     at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
[error]     at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
[error]     at java.base/java.lang.reflect.ReflectAccess.newInstance(ReflectAccess.java:128)
[error]     at java.base/jdk.internal.reflect.ReflectionFactory.newInstance(ReflectionFactory.java:304)
[error]     at java.base/java.lang.Class.newInstance(Class.java:725)
[error]     at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:454)
[error]     at sbt.TestRunner.runTest$1(TestFramework.scala:153)
[error]     at sbt.TestRunner.run(TestFramework.scala:168)
[error]     at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.$anonfun$apply$1(TestFramework.scala:336)
[error]     at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:296)
[error]     at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336)
[error]     at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336)
[error]     at sbt.TestFunction.apply(TestFramework.scala:348)
[error]     at sbt.Tests$.processRunnable$1(Tests.scala:475)
[error]     at sbt.Tests$.$anonfun$makeSerial$1(Tests.scala:481)
[error]     at sbt.std.Transform$$anon$3.$anonfun$apply$2(Transform.scala:47)
[error]     at sbt.std.Transform$$anon$4.work(Transform.scala:69)
[error]     at sbt.Execute.$anonfun$submit$2(Execute.scala:283)
[error]     at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:24)
[error]     at sbt.Execute.work(Execute.scala:292)
[error]     at sbt.Execute.$anonfun$submit$1(Execute.scala:283)
[error]     at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
[error]     at sbt.CompletionService$$anon$2.call(CompletionService.scala:65)
[error]     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
[error]     at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
[error]     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
[error]     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
[error]     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
[error]     at java.base/java.lang.Thread.run(Thread.java:1583)
[error] (mleap-xgboost-runtime / Test / executeTests) java.lang.ExceptionInInitializerError
[error] Total time: 302 s (05:02), completed 14-Oct-2023, 5:38:31 pm
make: *** [test_xgboost_runtime] Error 1

Any inputs on how I can resolve this?

jsleight commented 1 year ago

Ah

The libxgboost4j being missing is a known thing when trying to build on mac. XGBoost's java libraries don't support mac. iirc tensorflow libraries might have this problem too.