search

huawei-noah / streamDM

Stream Data Mining Library for Spark Streaming
http://streamdm.noahlab.com.hk/
Apache License 2.0
492 stars 147 forks source link

Getting started task #117

Open ELMERWANG opened 3 years ago

ELMERWANG commented 3 years ago

Hi,

I am quite new to stream mining and I am trying to get the streamDM running. However when I finished setting up all the dependencies I got some issues reported below.

Expected behavior

I want to obtain the confusion matrix by using the example commands.

Observed behavior

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 21/03/09 18:40:50 INFO SparkContext: Running Spark version 2.1.0 21/03/09 18:40:50 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 21/03/09 18:40:50 WARN Utils: Your hostname, localhost.localdomain resolves to a loopback address: 127.0.0.1; using 192.168.122.1 instead (on interface virbr0) 21/03/09 18:40:50 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address 21/03/09 18:40:50 INFO SecurityManager: Changing view acls to: shunyao 21/03/09 18:40:50 INFO SecurityManager: Changing modify acls to: shunyao 21/03/09 18:40:50 INFO SecurityManager: Changing view acls groups to: 21/03/09 18:40:50 INFO SecurityManager: Changing modify acls groups to: 21/03/09 18:40:50 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(shunyao); groups with view permissions: Set(); users with modify permissions: Set(shunyao); groups with modify permissions: Set() 21/03/09 18:40:50 INFO Utils: Successfully started service 'sparkDriver' on port 41123. 21/03/09 18:40:50 INFO SparkEnv: Registering MapOutputTracker 21/03/09 18:40:50 INFO SparkEnv: Registering BlockManagerMaster 21/03/09 18:40:50 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information 21/03/09 18:40:50 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up 21/03/09 18:40:50 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-8d18da96-7b2c-48c9-bbd2-b49a29b16274 21/03/09 18:40:50 INFO MemoryStore: MemoryStore started with capacity 413.9 MB 21/03/09 18:40:50 INFO SparkEnv: Registering OutputCommitCoordinator 21/03/09 18:40:51 INFO Utils: Successfully started service 'SparkUI' on port 4040. 21/03/09 18:40:51 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.122.1:4040 21/03/09 18:40:51 INFO SparkContext: Added JAR file:/home/shunyao/Downloads/streamDM-0.2/scripts/../target/scala-2.10/streamdm-spark-streaming-_2.10-0.2.jar at spark://192.168.122.1:41123/jars/streamdm-spark-streaming-_2.10-0.2.jar with timestamp 1615333251155 21/03/09 18:40:51 INFO Executor: Starting executor ID driver on host localhost 21/03/09 18:40:51 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 45261. 21/03/09 18:40:51 INFO NettyBlockTransferService: Server created on 192.168.122.1:45261 21/03/09 18:40:51 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy 21/03/09 18:40:51 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.122.1, 45261, None) 21/03/09 18:40:51 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.122.1:45261 with 413.9 MB RAM, BlockManagerId(driver, 192.168.122.1, 45261, None) 21/03/09 18:40:51 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.122.1, 45261, None) 21/03/09 18:40:51 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.122.1, 45261, None) Exception in thread "main" java.lang.Exception: Problem creating instance of class: EvaluatePrequential at com.github.javacliparser.ClassOption.cliStringToObject(ClassOption.java:139) at org.apache.spark.streamdm.streamDMJob$.main(streamDMJob.scala:43) at org.apache.spark.streamdm.streamDMJob.main(streamDMJob.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.IllegalArgumentException: Problems with option: streamReader at com.github.javacliparser.ClassOption.setValueViaCLIString(ClassOption.java:60) at com.github.javacliparser.AbstractOption.resetToDefault(AbstractOption.java:90) at com.github.javacliparser.AbstractClassOption.(AbstractClassOption.java:84) at com.github.javacliparser.AbstractClassOption.(AbstractClassOption.java:63) at com.github.javacliparser.ClassOption.(ClassOption.java:34) at org.apache.spark.streamdm.tasks.EvaluatePrequential.(EvaluatePrequential.scala:47) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at java.lang.Class.newInstance(Class.java:442) at com.github.javacliparser.ClassOption.cliStringToObject(ClassOption.java:137) ... 11 more Caused by: java.lang.Exception: Class not found: FileReader at com.github.javacliparser.ClassOption.cliStringToObject(ClassOption.java:132) at com.github.javacliparser.ClassOption.setValueViaCLIString(ClassOption.java:57) ... 22 more 21/03/09 18:40:51 INFO SparkContext: Invoking stop() from shutdown hook 21/03/09 18:40:51 INFO SparkUI: Stopped Spark web UI at http://192.168.122.1:4040 21/03/09 18:40:51 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 21/03/09 18:40:51 INFO MemoryStore: MemoryStore cleared 21/03/09 18:40:51 INFO BlockManager: BlockManager stopped 21/03/09 18:40:51 INFO BlockManagerMaster: BlockManagerMaster stopped 21/03/09 18:40:51 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 21/03/09 18:40:51 INFO SparkContext: Successfully stopped SparkContext 21/03/09 18:40:51 INFO ShutdownHookManager: Shutdown hook called 21/03/09 18:40:51 INFO ShutdownHookManager: Deleting directory /tmp/spark-6f0a2908-b3f5-477c-bccc-6302be0e18f8

Steps to reproduce the issue

I tried different OS: macOS, Kali, Ubuntu and CentOS 7.

These steps are what I did and followed: Step 1: Download & install the Java openjdk8. Step 2: Download & install the Spark (I tried 2.3.2 and 2.1.0 with Hadoop or without Hadoop) Step 3: Download & install Scala (from https://www.scala-lang.org/files/archive/scala-2.11.8.deb) Step 4: Download & install SBT (from https://sbt.bintray.com/native-packages/sbt/0.13.15/sbt-0.13.15.tgz) Step 5: Download StreamDM from Github. Step 6: Change the SPARK_HOME to the Spark path. Step 7: In the streamDM directory, run sbt package Step 8: In the /scripts, run ./spark.sh or ./spark.sh "EvaluatePrequential -l (SGDLearner -l 0.01 -o LogisticLoss -r ZeroRegularizer) –s (FileReader –k 100 –d 60 –f ../data/mydata)" Result: Errors occurred.

Command line sbt package ./spark.sh ./spark.sh "EvaluatePrequential -l (SGDLearner -l 0.01 -o LogisticLoss -r ZeroRegularizer) –s (FileReader –k 100 –d 60 –f ../data/mydata)"

Infrastructure details (Contains the versions that I tried)

For the Spark, do I need to download with Hadoop or without Hadoop or the file name only contains spark-x.x.x.tgz? Such as:

Is anything wrong with my setup? I produced the same error on different OS. Please help, thanks in advance :)

zhangjiajin commented 3 years ago

@abifet @hmgomes

hmgomes commented 3 years ago

Hi @ELMERWANG It looks like a problem with SGDLearner. Can you run it using a Hoeffding tree?

./spark.sh "EvaluatePrequential -l (trees.HoeffdingTree) -s (FileReader -f ../data/mydata.arff -k 450 -i 45500)" 1> results_ht_mydata.csv 2> log_ht_mydata.log

Btw, if you are new to stream mining, I suggest this open book to get started

ELMERWANG commented 3 years ago

Hi @hmgomes,

Thank you for your reply, I have tried the command that you provide, however the error now changed to:

Error message: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 21/03/21 19:46:27 INFO SparkContext: Running Spark version 2.1.0 21/03/21 19:46:27 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 21/03/21 19:46:27 WARN Utils: Your hostname, localhost.localdomain resolves to a loopback address: 127.0.0.1; using 192.168.122.1 instead (on interface virbr0) 21/03/21 19:46:27 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address 21/03/21 19:46:27 INFO SecurityManager: Changing view acls to: shunyao 21/03/21 19:46:27 INFO SecurityManager: Changing modify acls to: shunyao 21/03/21 19:46:27 INFO SecurityManager: Changing view acls groups to: 21/03/21 19:46:27 INFO SecurityManager: Changing modify acls groups to: 21/03/21 19:46:27 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(shunyao); groups with view permissions: Set(); users with modify permissions: Set(shunyao); groups with modify permissions: Set() 21/03/21 19:46:28 INFO Utils: Successfully started service 'sparkDriver' on port 45035. 21/03/21 19:46:28 INFO SparkEnv: Registering MapOutputTracker 21/03/21 19:46:28 INFO SparkEnv: Registering BlockManagerMaster 21/03/21 19:46:28 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information 21/03/21 19:46:28 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up 21/03/21 19:46:28 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-deb8bb4e-c0e0-450e-b68d-3f96548bece1 21/03/21 19:46:28 INFO MemoryStore: MemoryStore started with capacity 413.9 MB 21/03/21 19:46:28 INFO SparkEnv: Registering OutputCommitCoordinator 21/03/21 19:46:28 INFO Utils: Successfully started service 'SparkUI' on port 4040. 21/03/21 19:46:28 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.122.1:4040 21/03/21 19:46:28 INFO SparkContext: Added JAR file:/home/shunyao/Downloads/streamDM-0.2/scripts/../target/scala-2.10/streamdm-spark-streaming-_2.10-0.2.jar at spark://192.168.122.1:45035/jars/streamdm-spark-streaming-_2.10-0.2.jar with timestamp 1616370388511 21/03/21 19:46:28 INFO Executor: Starting executor ID driver on host localhost 21/03/21 19:46:28 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 38860. 21/03/21 19:46:28 INFO NettyBlockTransferService: Server created on 192.168.122.1:38860 21/03/21 19:46:28 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy 21/03/21 19:46:28 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.122.1, 38860, None) 21/03/21 19:46:28 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.122.1:38860 with 413.9 MB RAM, BlockManagerId(driver, 192.168.122.1, 38860, None) 21/03/21 19:46:28 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.122.1, 38860, None) 21/03/21 19:46:28 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.122.1, 38860, None) Exception in thread "main" java.lang.Exception: Problem creating instance of class: EvaluatePrequential at com.github.javacliparser.ClassOption.cliStringToObject(ClassOption.java:139) at org.apache.spark.streamdm.streamDMJob$.main(streamDMJob.scala:43) at org.apache.spark.streamdm.streamDMJob.main(streamDMJob.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.IllegalArgumentException: Problems with option: streamReader at com.github.javacliparser.ClassOption.setValueViaCLIString(ClassOption.java:60) at com.github.javacliparser.AbstractOption.resetToDefault(AbstractOption.java:90) at com.github.javacliparser.AbstractClassOption.(AbstractClassOption.java:84) at com.github.javacliparser.AbstractClassOption.(AbstractClassOption.java:63) at com.github.javacliparser.ClassOption.(ClassOption.java:34) at org.apache.spark.streamdm.tasks.EvaluatePrequential.(EvaluatePrequential.scala:47) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at java.lang.Class.newInstance(Class.java:442) at com.github.javacliparser.ClassOption.cliStringToObject(ClassOption.java:137) ... 11 more Caused by: java.lang.Exception: Class not found: FileReader at com.github.javacliparser.ClassOption.cliStringToObject(ClassOption.java:132) at com.github.javacliparser.ClassOption.setValueViaCLIString(ClassOption.java:57) ... 22 more 21/03/21 19:46:28 INFO SparkContext: Invoking stop() from shutdown hook 21/03/21 19:46:28 INFO SparkUI: Stopped Spark web UI at http://192.168.122.1:4040 21/03/21 19:46:28 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 21/03/21 19:46:28 INFO MemoryStore: MemoryStore cleared 21/03/21 19:46:28 INFO BlockManager: BlockManager stopped 21/03/21 19:46:28 INFO BlockManagerMaster: BlockManagerMaster stopped 21/03/21 19:46:28 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 21/03/21 19:46:28 INFO SparkContext: Successfully stopped SparkContext 21/03/21 19:46:28 INFO ShutdownHookManager: Shutdown hook called 21/03/21 19:46:28 INFO ShutdownHookManager: Deleting directory /tmp/spark-d340aa5a-9e9b-4d68-a6fd-b24a0276338d

And the result file is empty...

hmgomes commented 3 years ago

Hi @ELMERWANG

That is interesting. Can you try using Spark 2.4.5?

Cheers, Heitor

ELMERWANG commented 3 years ago

Hi @hmgomes,

Thanks, may I ask which version should I download? With or without hadoop?

[ ] spark-2.4.5-bin-hadoop2.6.tgz 2020-02-02 20:27 220M

[ ] spark-2.4.5-bin-hadoop2.7.tgz 2020-02-02 20:27 222M

[ ] spark-2.4.5-bin-without-hadoop-scala-2.12.tgz 2020-02-02 20:27 139M

[ ] spark-2.4.5-bin-without-hadoop.tgz 2020-02-02 20:27 160M

[ ] spark-2.4.5.tgz 2020-02-02 20:27 15M

LYZ1125 commented 3 years ago

@ELMERWANG Hi,I met the same problem as yours. Do you solve your problem?I want to get some suggestions to solve these problems. Thank you!

Infrastructure details Java Version: 8 Scala Version: 2.11.8 Spark version: 2.3.2 OS version: Ubuntu sbt version:0.13.18

When I run these Command lines sbt package ./spark.sh

I get some errors:

java.lang.ArrayIndexOutOfBoundsException: 4
    at org.apache.spark.streamdm.core.DenseInstance.add(DenseInstance.scala:84)
    at org.apache.spark.streamdm.core.DenseInstance.add(DenseInstance.scala:26)
    at org.apache.spark.streamdm.classifiers.SGDLearner$$anonfun$train$1$$anonfun$3.apply(SGDLearner.scala:91)
    at org.apache.spark.streamdm.classifiers.SGDLearner$$anonfun$train$1$$anonfun$3.apply(SGDLearner.scala:85)
    at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
    at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
    at scala.collection.Iterator$class.foreach(Iterator.scala:893)
    at org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
    at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:157)
    at org.apache.spark.InterruptibleIterator.foldLeft(InterruptibleIterator.scala:28)
    at scala.collection.TraversableOnce$class.aggregate(TraversableOnce.scala:214)
    at org.apache.spark.InterruptibleIterator.aggregate(InterruptibleIterator.scala:28)
    at org.apache.spark.rdd.RDD$$anonfun$aggregate$1$$anonfun$21.apply(RDD.scala:1122)
    at org.apache.spark.rdd.RDD$$anonfun$aggregate$1$$anonfun$21.apply(RDD.scala:1122)
    at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:2130)
    at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:2130)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
    at org.apache.spark.scheduler.Task.run(Task.scala:109)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
2021-03-30 06:25:41 ERROR Executor:91 - Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.ArrayIndexOutOfBoundsException: 4
    at org.apache.spark.streamdm.core.DenseInstance.add(DenseInstance.scala:84)
    at org.apache.spark.streamdm.core.DenseInstance.add(DenseInstance.scala:26)
    at org.apache.spark.streamdm.classifiers.SGDLearner$$anonfun$train$1$$anonfun$3.apply(SGDLearner.scala:91)
    at org.apache.spark.streamdm.classifiers.SGDLearner$$anonfun$train$1$$anonfun$3.apply(SGDLearner.scala:85)
    at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
    at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
    at scala.collection.Iterator$class.foreach(Iterator.scala:893)
    at org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
    at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:157)
    at org.apache.spark.InterruptibleIterator.foldLeft(InterruptibleIterator.scala:28)
    at scala.collection.TraversableOnce$class.aggregate(TraversableOnce.scala:214)
    at org.apache.spark.InterruptibleIterator.aggregate(InterruptibleIterator.scala:28)
    at org.apache.spark.rdd.RDD$$anonfun$aggregate$1$$anonfun$21.apply(RDD.scala:1122)
    at org.apache.spark.rdd.RDD$$anonfun$aggregate$1$$anonfun$21.apply(RDD.scala:1122)
    at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:2130)
    at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:2130)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
    at org.apache.spark.scheduler.Task.run(Task.scala:109)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

after I ran ./spark.sh "EvaluatePrequential -l (SGDLearner -l 0.01 -o LogisticLoss -r ZeroRegularizer) –s (FileReader –k 100 –d 60 –f ../data/mydata)",I got the follewing result:

/home/yuanzhanglin/local/spark-2.3.2/conf/spark-env.sh: line 71: syntax error near unexpected token `newline'
/home/yuanzhanglin/local/spark-2.3.2/conf/spark-env.sh: line 71: `<!--主节点IP-->'
2021-03-30 06:28:28 WARN  Utils:66 - Your hostname, ubuntu resolves to a loopback address: 127.0.1.1; using 192.168.65.128 instead (on interface ens33)
2021-03-30 06:28:28 WARN  Utils:66 - Set SPARK_LOCAL_IP if you need to bind to another address
2021-03-30 06:28:32 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2021-03-30 06:28:32 INFO  SparkContext:54 - Running Spark version 2.3.2
2021-03-30 06:28:32 INFO  SparkContext:54 - Submitted application: streamDM
2021-03-30 06:28:32 INFO  SecurityManager:54 - Changing view acls to: yuanzhanglin
2021-03-30 06:28:32 INFO  SecurityManager:54 - Changing modify acls to: yuanzhanglin
2021-03-30 06:28:32 INFO  SecurityManager:54 - Changing view acls groups to: 
2021-03-30 06:28:32 INFO  SecurityManager:54 - Changing modify acls groups to: 
2021-03-30 06:28:32 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(yuanzhanglin); groups with view permissions: Set(); users  with modify permissions: Set(yuanzhanglin); groups with modify permissions: Set()
2021-03-30 06:28:33 INFO  Utils:54 - Successfully started service 'sparkDriver' on port 44865.
2021-03-30 06:28:33 INFO  SparkEnv:54 - Registering MapOutputTracker
2021-03-30 06:28:33 INFO  SparkEnv:54 - Registering BlockManagerMaster
2021-03-30 06:28:33 INFO  BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
2021-03-30 06:28:33 INFO  BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up
2021-03-30 06:28:33 INFO  DiskBlockManager:54 - Created local directory at /tmp/blockmgr-beb224af-932d-4d9f-9695-42a4a64c2287
2021-03-30 06:28:33 INFO  MemoryStore:54 - MemoryStore started with capacity 366.3 MB
2021-03-30 06:28:33 INFO  SparkEnv:54 - Registering OutputCommitCoordinator
2021-03-30 06:28:33 INFO  log:192 - Logging initialized @6306ms
2021-03-30 06:28:33 INFO  Server:351 - jetty-9.3.z-SNAPSHOT, build timestamp: unknown, git hash: unknown
2021-03-30 06:28:33 INFO  Server:419 - Started @6448ms
2021-03-30 06:28:33 INFO  AbstractConnector:278 - Started ServerConnector@21a5fd96{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2021-03-30 06:28:33 INFO  Utils:54 - Successfully started service 'SparkUI' on port 4040.
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@42a9a63e{/jobs,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5bd1ceca{/jobs/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@30c31dd7{/jobs/job,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@596df867{/jobs/job/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@c1fca1e{/stages,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@241a53ef{/stages/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@344344fa{/stages/stage,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@615f972{/stages/stage/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@285f09de{/stages/pool,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@73393584{/stages/pool/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@31500940{/storage,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1827a871{/storage/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@48e64352{/storage/rdd,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7249dadf{/storage/rdd/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4362d7df{/environment,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@66238be2{/environment/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1c25b8a7{/executors,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@200606de{/executors/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@750fe12e{/executors/threadDump,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@f8908f6{/executors/threadDump/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3e587920{/static,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@77b7ffa4{/,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5ed190be{/api,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@41382722{/jobs/job/kill,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7dac3fd8{/stages/stage/kill,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://192.168.65.128:4040
2021-03-30 06:28:34 INFO  SparkContext:54 - Added JAR file:/home/yuanzhanglin/streamDM/scripts/../target/scala-2.11/streamdm-spark-streaming-_2.11-0.2.jar at spark://192.168.65.128:44865/jars/streamdm-spark-streaming-_2.11-0.2.jar with timestamp 1617110914021
2021-03-30 06:28:34 INFO  Executor:54 - Starting executor ID driver on host localhost
2021-03-30 06:28:34 INFO  Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 37021.
2021-03-30 06:28:34 INFO  NettyBlockTransferService:54 - Server created on 192.168.65.128:37021
2021-03-30 06:28:34 INFO  BlockManager:54 - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
2021-03-30 06:28:34 INFO  BlockManagerMaster:54 - Registering BlockManager BlockManagerId(driver, 192.168.65.128, 37021, None)
2021-03-30 06:28:34 INFO  BlockManagerMasterEndpoint:54 - Registering block manager 192.168.65.128:37021 with 366.3 MB RAM, BlockManagerId(driver, 192.168.65.128, 37021, None)
2021-03-30 06:28:34 INFO  BlockManagerMaster:54 - Registered BlockManager BlockManagerId(driver, 192.168.65.128, 37021, None)
2021-03-30 06:28:34 INFO  BlockManager:54 - Initialized BlockManager: BlockManagerId(driver, 192.168.65.128, 37021, None)
2021-03-30 06:28:34 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3703bf3c{/metrics/json,null,AVAILABLE,@Spark}
Exception in thread "main" java.lang.Exception: Class not found: or
    at com.github.javacliparser.ClassOption.cliStringToObject(ClassOption.java:132)
    at org.apache.spark.streamdm.streamDMJob$.main(streamDMJob.scala:55)
    at org.apache.spark.streamdm.streamDMJob.main(streamDMJob.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
2021-03-30 06:28:34 INFO  SparkContext:54 - Invoking stop() from shutdown hook
2021-03-30 06:28:34 INFO  AbstractConnector:318 - Stopped Spark@21a5fd96{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2021-03-30 06:28:34 INFO  SparkUI:54 - Stopped Spark web UI at http://192.168.65.128:4040
2021-03-30 06:28:34 INFO  MapOutputTrackerMasterEndpoint:54 - MapOutputTrackerMasterEndpoint stopped!
2021-03-30 06:28:34 INFO  MemoryStore:54 - MemoryStore cleared
2021-03-30 06:28:34 INFO  BlockManager:54 - BlockManager stopped
2021-03-30 06:28:34 INFO  BlockManagerMaster:54 - BlockManagerMaster stopped
2021-03-30 06:28:34 INFO  OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 - OutputCommitCoordinator stopped!
2021-03-30 06:28:34 INFO  SparkContext:54 - Successfully stopped SparkContext
2021-03-30 06:28:34 INFO  ShutdownHookManager:54 - Shutdown hook called
2021-03-30 06:28:34 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-27030bd4-3077-4318-8fec-a42efb3c4dbb
2021-03-30 06:28:34 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-d29803f8-d90b-491c-82e5-cc2599ac2e0f
ELMERWANG commented 3 years ago

@ELMERWANG Hi,I met the same problem as yours. Do you solve your problem?I want to get some suggestions to solve these problems. Thank you!

Infrastructure details Java Version: 8 Scala Version: 2.11.8 Spark version: 2.3.2 OS version: Ubuntu sbt version:0.13.18

When I run these Command lines sbt package ./spark.sh

I get some errors:

java.lang.ArrayIndexOutOfBoundsException: 4
  at org.apache.spark.streamdm.core.DenseInstance.add(DenseInstance.scala:84)
  at org.apache.spark.streamdm.core.DenseInstance.add(DenseInstance.scala:26)
  at org.apache.spark.streamdm.classifiers.SGDLearner$$anonfun$train$1$$anonfun$3.apply(SGDLearner.scala:91)
  at org.apache.spark.streamdm.classifiers.SGDLearner$$anonfun$train$1$$anonfun$3.apply(SGDLearner.scala:85)
  at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
  at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
  at scala.collection.Iterator$class.foreach(Iterator.scala:893)
  at org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
  at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:157)
  at org.apache.spark.InterruptibleIterator.foldLeft(InterruptibleIterator.scala:28)
  at scala.collection.TraversableOnce$class.aggregate(TraversableOnce.scala:214)
  at org.apache.spark.InterruptibleIterator.aggregate(InterruptibleIterator.scala:28)
  at org.apache.spark.rdd.RDD$$anonfun$aggregate$1$$anonfun$21.apply(RDD.scala:1122)
  at org.apache.spark.rdd.RDD$$anonfun$aggregate$1$$anonfun$21.apply(RDD.scala:1122)
  at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:2130)
  at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:2130)
  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
  at org.apache.spark.scheduler.Task.run(Task.scala:109)
  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)
2021-03-30 06:25:41 ERROR Executor:91 - Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.ArrayIndexOutOfBoundsException: 4
  at org.apache.spark.streamdm.core.DenseInstance.add(DenseInstance.scala:84)
  at org.apache.spark.streamdm.core.DenseInstance.add(DenseInstance.scala:26)
  at org.apache.spark.streamdm.classifiers.SGDLearner$$anonfun$train$1$$anonfun$3.apply(SGDLearner.scala:91)
  at org.apache.spark.streamdm.classifiers.SGDLearner$$anonfun$train$1$$anonfun$3.apply(SGDLearner.scala:85)
  at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
  at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
  at scala.collection.Iterator$class.foreach(Iterator.scala:893)
  at org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
  at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:157)
  at org.apache.spark.InterruptibleIterator.foldLeft(InterruptibleIterator.scala:28)
  at scala.collection.TraversableOnce$class.aggregate(TraversableOnce.scala:214)
  at org.apache.spark.InterruptibleIterator.aggregate(InterruptibleIterator.scala:28)
  at org.apache.spark.rdd.RDD$$anonfun$aggregate$1$$anonfun$21.apply(RDD.scala:1122)
  at org.apache.spark.rdd.RDD$$anonfun$aggregate$1$$anonfun$21.apply(RDD.scala:1122)
  at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:2130)
  at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:2130)
  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
  at org.apache.spark.scheduler.Task.run(Task.scala:109)
  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)

after I ran ./spark.sh "EvaluatePrequential -l (SGDLearner -l 0.01 -o LogisticLoss -r ZeroRegularizer) –s (FileReader –k 100 –d 60 –f ../data/mydata)",I got the follewing result:

/home/yuanzhanglin/local/spark-2.3.2/conf/spark-env.sh: line 71: syntax error near unexpected token `newline'
/home/yuanzhanglin/local/spark-2.3.2/conf/spark-env.sh: line 71: `<!--主节点IP-->'
2021-03-30 06:28:28 WARN  Utils:66 - Your hostname, ubuntu resolves to a loopback address: 127.0.1.1; using 192.168.65.128 instead (on interface ens33)
2021-03-30 06:28:28 WARN  Utils:66 - Set SPARK_LOCAL_IP if you need to bind to another address
2021-03-30 06:28:32 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2021-03-30 06:28:32 INFO  SparkContext:54 - Running Spark version 2.3.2
2021-03-30 06:28:32 INFO  SparkContext:54 - Submitted application: streamDM
2021-03-30 06:28:32 INFO  SecurityManager:54 - Changing view acls to: yuanzhanglin
2021-03-30 06:28:32 INFO  SecurityManager:54 - Changing modify acls to: yuanzhanglin
2021-03-30 06:28:32 INFO  SecurityManager:54 - Changing view acls groups to: 
2021-03-30 06:28:32 INFO  SecurityManager:54 - Changing modify acls groups to: 
2021-03-30 06:28:32 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(yuanzhanglin); groups with view permissions: Set(); users  with modify permissions: Set(yuanzhanglin); groups with modify permissions: Set()
2021-03-30 06:28:33 INFO  Utils:54 - Successfully started service 'sparkDriver' on port 44865.
2021-03-30 06:28:33 INFO  SparkEnv:54 - Registering MapOutputTracker
2021-03-30 06:28:33 INFO  SparkEnv:54 - Registering BlockManagerMaster
2021-03-30 06:28:33 INFO  BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
2021-03-30 06:28:33 INFO  BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up
2021-03-30 06:28:33 INFO  DiskBlockManager:54 - Created local directory at /tmp/blockmgr-beb224af-932d-4d9f-9695-42a4a64c2287
2021-03-30 06:28:33 INFO  MemoryStore:54 - MemoryStore started with capacity 366.3 MB
2021-03-30 06:28:33 INFO  SparkEnv:54 - Registering OutputCommitCoordinator
2021-03-30 06:28:33 INFO  log:192 - Logging initialized @6306ms
2021-03-30 06:28:33 INFO  Server:351 - jetty-9.3.z-SNAPSHOT, build timestamp: unknown, git hash: unknown
2021-03-30 06:28:33 INFO  Server:419 - Started @6448ms
2021-03-30 06:28:33 INFO  AbstractConnector:278 - Started ServerConnector@21a5fd96{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2021-03-30 06:28:33 INFO  Utils:54 - Successfully started service 'SparkUI' on port 4040.
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@42a9a63e{/jobs,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5bd1ceca{/jobs/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@30c31dd7{/jobs/job,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@596df867{/jobs/job/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@c1fca1e{/stages,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@241a53ef{/stages/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@344344fa{/stages/stage,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@615f972{/stages/stage/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@285f09de{/stages/pool,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@73393584{/stages/pool/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@31500940{/storage,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1827a871{/storage/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@48e64352{/storage/rdd,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7249dadf{/storage/rdd/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4362d7df{/environment,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@66238be2{/environment/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1c25b8a7{/executors,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@200606de{/executors/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@750fe12e{/executors/threadDump,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@f8908f6{/executors/threadDump/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3e587920{/static,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@77b7ffa4{/,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5ed190be{/api,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@41382722{/jobs/job/kill,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7dac3fd8{/stages/stage/kill,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://192.168.65.128:4040
2021-03-30 06:28:34 INFO  SparkContext:54 - Added JAR file:/home/yuanzhanglin/streamDM/scripts/../target/scala-2.11/streamdm-spark-streaming-_2.11-0.2.jar at spark://192.168.65.128:44865/jars/streamdm-spark-streaming-_2.11-0.2.jar with timestamp 1617110914021
2021-03-30 06:28:34 INFO  Executor:54 - Starting executor ID driver on host localhost
2021-03-30 06:28:34 INFO  Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 37021.
2021-03-30 06:28:34 INFO  NettyBlockTransferService:54 - Server created on 192.168.65.128:37021
2021-03-30 06:28:34 INFO  BlockManager:54 - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
2021-03-30 06:28:34 INFO  BlockManagerMaster:54 - Registering BlockManager BlockManagerId(driver, 192.168.65.128, 37021, None)
2021-03-30 06:28:34 INFO  BlockManagerMasterEndpoint:54 - Registering block manager 192.168.65.128:37021 with 366.3 MB RAM, BlockManagerId(driver, 192.168.65.128, 37021, None)
2021-03-30 06:28:34 INFO  BlockManagerMaster:54 - Registered BlockManager BlockManagerId(driver, 192.168.65.128, 37021, None)
2021-03-30 06:28:34 INFO  BlockManager:54 - Initialized BlockManager: BlockManagerId(driver, 192.168.65.128, 37021, None)
2021-03-30 06:28:34 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3703bf3c{/metrics/json,null,AVAILABLE,@Spark}
Exception in thread "main" java.lang.Exception: Class not found: or
  at com.github.javacliparser.ClassOption.cliStringToObject(ClassOption.java:132)
  at org.apache.spark.streamdm.streamDMJob$.main(streamDMJob.scala:55)
  at org.apache.spark.streamdm.streamDMJob.main(streamDMJob.scala)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
  at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
  at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
  at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
  at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
  at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
2021-03-30 06:28:34 INFO  SparkContext:54 - Invoking stop() from shutdown hook
2021-03-30 06:28:34 INFO  AbstractConnector:318 - Stopped Spark@21a5fd96{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2021-03-30 06:28:34 INFO  SparkUI:54 - Stopped Spark web UI at http://192.168.65.128:4040
2021-03-30 06:28:34 INFO  MapOutputTrackerMasterEndpoint:54 - MapOutputTrackerMasterEndpoint stopped!
2021-03-30 06:28:34 INFO  MemoryStore:54 - MemoryStore cleared
2021-03-30 06:28:34 INFO  BlockManager:54 - BlockManager stopped
2021-03-30 06:28:34 INFO  BlockManagerMaster:54 - BlockManagerMaster stopped
2021-03-30 06:28:34 INFO  OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 - OutputCommitCoordinator stopped!
2021-03-30 06:28:34 INFO  SparkContext:54 - Successfully stopped SparkContext
2021-03-30 06:28:34 INFO  ShutdownHookManager:54 - Shutdown hook called
2021-03-30 06:28:34 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-27030bd4-3077-4318-8fec-a42efb3c4dbb
2021-03-30 06:28:34 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-d29803f8-d90b-491c-82e5-cc2599ac2e0f

Unfortunately, no luck on fixing it, sorry..

hmgomes commented 3 years ago

Hi @hmgomes,

Thanks, may I ask which version should I download? With or without hadoop?

[ ] spark-2.4.5-bin-hadoop2.6.tgz 2020-02-02 20:27 220M

[ ] spark-2.4.5-bin-hadoop2.7.tgz 2020-02-02 20:27 222M

[ ] spark-2.4.5-bin-without-hadoop-scala-2.12.tgz 2020-02-02 20:27 139M

[ ] spark-2.4.5-bin-without-hadoop.tgz 2020-02-02 20:27 160M

[ ] spark-2.4.5.tgz 2020-02-02 20:27 15M

Hi @ELMERWANG

I have the version with Hadoop "spark-2.4.5-bin-hadoop2.7"

Regards, Heitor

hmgomes commented 3 years ago

Hi @LYZ1125

Spark version: 2.3.2 Try spark 2.4.5

Regards, Heitor