search

JaneliaSciComp / BigStitcher-Spark

Running compute-intense parts of BigStitcher distributed
BSD 2-Clause "Simplified" License
18 stars 10 forks source link

BigStitcher spark Dependency problem #26

Open salammemphis opened 4 months ago

salammemphis commented 4 months ago

I was running BigStitcher Spark on a cluster single-node. At the end of first step "detect interesting point" its throwing an exceptions. Any suggestion ??

I have pasted logs below:

processing tpId=263 setupId=7, [724, 1536, 0] -> [1191, 1919, 46], dimensions (468, 384, 47) of full interval [724, 0, 0] -> [1191, 1919, 46], dimensions (468, 1920, 47) Total number of jobs: 44571648 Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties 24/04/05 01:44:11 INFO SparkContext: Running Spark version 3.3.2 24/04/05 01:44:12 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 24/04/05 01:44:12 INFO ResourceUtils: ============================================================== 24/04/05 01:44:12 INFO ResourceUtils: No custom resources configured for spark.driver. 24/04/05 01:44:12 INFO ResourceUtils: ============================================================== 24/04/05 01:44:12 INFO SparkContext: Submitted application: SparkInterestPointDetection 24/04/05 01:44:12 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memunt: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0) 24/04/05 01:44:12 INFO ResourceProfile: Limiting resource is cpu 24/04/05 01:44:12 INFO ResourceProfileManager: Added ResourceProfile id: 0 24/04/05 01:44:13 INFO SecurityManager: Changing view acls to: salam 24/04/05 01:44:13 INFO SecurityManager: Changing modify acls to: salam 24/04/05 01:44:13 INFO SecurityManager: Changing view acls groups to: 24/04/05 01:44:13 INFO SecurityManager: Changing modify acls groups to: 24/04/05 01:44:13 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(salam); groups with view permissiondify permissions: Set() 24/04/05 01:44:13 INFO Utils: Successfully started service 'sparkDriver' on port 37185. 24/04/05 01:44:13 INFO SparkEnv: Registering MapOutputTracker 24/04/05 01:44:13 INFO SparkEnv: Registering BlockManagerMaster 24/04/05 01:44:13 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information 24/04/05 01:44:13 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up Exception in thread "main" java.lang.IllegalAccessError: class org.apache.spark.storage.StorageUtils$ (in unnamed module @0x69ecddc6) cannot access class sun.nio.ch.Direort sun.nio.ch to unnamed module @0x69ecddc6 at org.apache.spark.storage.StorageUtils$.(StorageUtils.scala:213) at org.apache.spark.storage.StorageUtils$.(StorageUtils.scala) at org.apache.spark.storage.BlockManagerMasterEndpoint.(BlockManagerMasterEndpoint.scala:114) at org.apache.spark.SparkEnv$.$anonfun$create$9(SparkEnv.scala:353) at org.apache.spark.SparkEnv$.registerOrLookupEndpoint$1(SparkEnv.scala:290) at org.apache.spark.SparkEnv$.create(SparkEnv.scala:339) at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:194) at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:279) at org.apache.spark.SparkContext.(SparkContext.scala:464) at org.apache.spark.api.java.JavaSparkContext.(JavaSparkContext.scala:58) at net.preibisch.bigstitcher.spark.SparkInterestPointDetection.call(SparkInterestPointDetection.java:276) at net.preibisch.bigstitcher.spark.SparkInterestPointDetection.call(SparkInterestPointDetection.java:79) at picocli.CommandLine.executeUserObject(CommandLine.java:2041) at picocli.CommandLine.access$1500(CommandLine.java:148) at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461) at picocli.CommandLine$RunLast.handle(CommandLine.java:2453) at picocli.CommandLine$RunLast.handle(CommandLine.java:2415) at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273) at picocli.CommandLine$RunLast.execute(CommandLine.java:2417) at picocli.CommandLine.execute(CommandLine.java:2170) at net.preibisch.bigstitcher.spark.SparkInterestPointDetection.main(SparkInterestPointDetection.java:758)**

kgabor commented 3 months ago

For java 17+ you need the following interpreter options at the moment:

--add-exports java.base/sun.nio.ch=ALL-UNNAMED \
--add-opens java.base/java.util=ALL-UNNAMED \
--add-opens java.base/java.io=ALL-UNNAMED \
--add-opens java.base/java.nio=ALL-UNNAMED \
--add-opens java.base/java.lang=ALL-UNNAMED \
--add-opens java.base/java.lang.invoke=ALL-UNNAMED \
--add-opens java.base/sun.security.util=ALL-UNNAMED \
--add-opens java.base/sun.security.action=ALL-UNNAMED

https://stackoverflow.com/questions/72230174/java-17-solution-for-spark-java-lang-noclassdeffounderror-could-not-initializ

StephanPreibisch commented 3 months ago

I still use Java 8 ...