jornfranke
commented
6 years ago Increase the memory or reduce the data in the graph. I am not sure how much data you input, but if it is the full blockchain and you want to do a graph then you can see if there is something to optimize the code and/or increase memory. Another alternative is to use the library to import the blockchain data into ORC or parquet format and then do the graph. Yet another solution is to enable spill to disk, in case you run out of memory. Newer Spark versions might have better memory usage.
On 8. Jun 2018, at 08:45, foonsun notifications@github.com wrote:
@jornfranke I have run the example according to the wiki: https://github.com/ZuInnoTe/hadoopcryptoledger/wiki/Using-Spark-Scala-Graphx-to-analyze-the-Bitcoin-transaction-graph
it has run over 4 hours,and the issue occurs below. and i run it again for about 4hours.the issue still happens. it seems that the memory is not enough.but the memory of my docker is enough big.
[root@quickstart scala-spark-graphx-bitcointransaction]# df -h Filesystem Size Used Avail Use% Mounted on overlay 2.0T 1.2T 797G 60% / tmpfs 32G 0 32G 0% /dev tmpfs 32G 0 32G 0% /sys/fs/cgroup /dev/xvda1 2.0T 1.2T 797G 60% /etc/resolv.conf /dev/xvda1 2.0T 1.2T 797G 60% /etc/hostname /dev/xvda1 2.0T 1.2T 797G 60% /etc/hosts shm 64M 0 64M 0% /dev/shm [root@quickstart scala-spark-graphx-bitcointransaction]# free total used free shared buffers cached Mem: 65960180 25773772 40186408 686156 102472 14819768 -/+ buffers/cache: 10851532 55108648 Swap: 0 0 0 [root@quickstart scala-spark-graphx-bitcointransaction]# free -h total used free shared buffers cached Mem: 62G 24G 38G 670M 100M 14G -/+ buffers/cache: 10G 52G Swap: 0B 0B 0B how i can deal with this problem?thanks very much.
18/06/08 06:17:35 INFO collection.ExternalSorter: Thread 913 spilling in-memory map of 24.6 MB to disk (9 times so far) 18/06/08 06:17:35 INFO collection.ExternalSorter: Thread 947 spilling in-memory map of 25.0 MB to disk (28 times so far) 18/06/08 06:17:55 INFO collection.ExternalAppendOnlyMap: Thread 922 spilling in-memory map of 27.4 MB to disk (3 times so far) 18/06/08 06:18:02 INFO collection.ExternalSorter: Thread 947 spilling in-memory map of 24.7 MB to disk (29 times so far) 18/06/08 06:18:04 INFO collection.ExternalSorter: Thread 950 spilling in-memory map of 24.7 MB to disk (15 times so far) 18/06/08 06:18:09 INFO collection.ExternalSorter: Thread 909 spilling in-memory map of 24.8 MB to disk (18 times so far) 18/06/08 06:18:25 INFO collection.ExternalSorter: Thread 913 spilling in-memory map of 24.7 MB to disk (10 times so far) 18/06/08 06:18:26 WARN memory.TaskMemoryManager: leak 18.3 MB memory from org.apache.spark.util.collection.ExternalSorter@12719270 18/06/08 06:18:26 ERROR executor.Executor: Managed memory leak detected; size = 19172698 bytes, TID = 7855 18/06/08 06:18:26 ERROR executor.Executor: Exception in task 168.0 in stage 8.0 (TID 7855) java.lang.OutOfMemoryError: Java heap space at java.io.ObjectInputStream$HandleTable.grow(ObjectInputStream.java:3464) at java.io.ObjectInputStream$HandleTable.assign(ObjectInputStream.java:3271) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1789) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1706) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1344) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1706) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1344) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76) at org.apache.spark.serializer.DeserializationStream.readValue(Serializer.scala:171) at org.apache.spark.util.collection.ExternalAppendOnlyMap$DiskMapIterator.readNextItem(ExternalAppendOnlyMap.scala:515) at org.apache.spark.util.collection.ExternalAppendOnlyMap$DiskMapIterator.hasNext(ExternalAppendOnlyMap.scala:535) at scala.collection.Iterator$$anon$1.hasNext(Iterator.scala:847) at org.apache.spark.util.collection.ExternalAppendOnlyMap$ExternalIterator.org$apache$spark$util$collection$ExternalAppendOnlyMap$ExternalIterator$$readNextHashCode(ExternalAppendOnlyMap.scala:335) at org.apache.spark.util.collection.ExternalAppendOnlyMap$ExternalIterator$$anonfun$next$1.apply(ExternalAppendOnlyMap.scala:408) at org.apache.spark.util.collection.ExternalAppendOnlyMap$ExternalIterator$$anonfun$next$1.apply(ExternalAppendOnlyMap.scala:406) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at org.apache.spark.util.collection.ExternalAppendOnlyMap$ExternalIterator.next(ExternalAppendOnlyMap.scala:406) at org.apache.spark.util.collection.ExternalAppendOnlyMap$ExternalIterator.next(ExternalAppendOnlyMap.scala:301) at org.apache.spark.InterruptibleIterator.next(InterruptibleIterator.scala:43) at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:200) at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73) 18/06/08 06:18:26 ERROR util.SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[Executor task launch worker-67,5,main] java.lang.OutOfMemoryError: Java heap space at java.io.ObjectInputStream$HandleTable.grow(ObjectInputStream.java:3464) at java.io.ObjectInputStream$HandleTable.assign(ObjectInputStream.java:3271) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1789) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1706) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1344) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1706) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1344) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76) at org.apache.spark.serializer.DeserializationStream.readValue(Serializer.scala:171) at org.apache.spark.util.collection.ExternalAppendOnlyMap$DiskMapIterator.readNextItem(ExternalAppendOnlyMap.scala:515) at org.apache.spark.util.collection.ExternalAppendOnlyMap$DiskMapIterator.hasNext(ExternalAppendOnlyMap.scala:535) at scala.collection.Iterator$$anon$1.hasNext(Iterator.scala:847) at org.apache.spark.util.collection.ExternalAppendOnlyMap$ExternalIterator.org$apache$spark$util$collection$ExternalAppendOnlyMap$ExternalIterator$$readNextHashCode(ExternalAppendOnlyMap.scala:335) at org.apache.spark.util.collection.ExternalAppendOnlyMap$ExternalIterator$$anonfun$next$1.apply(ExternalAppendOnlyMap.scala:408) at org.apache.spark.util.collection.ExternalAppendOnlyMap$ExternalIterator$$anonfun$next$1.apply(ExternalAppendOnlyMap.scala:406) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at org.apache.spark.util.collection.ExternalAppendOnlyMap$ExternalIterator.next(ExternalAppendOnlyMap.scala:406) at org.apache.spark.util.collection.ExternalAppendOnlyMap$ExternalIterator.next(ExternalAppendOnlyMap.scala:301) at org.apache.spark.InterruptibleIterator.next(InterruptibleIterator.scala:43) at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:200) at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73) 18/06/08 06:18:26 INFO spark.SparkContext: Invoking stop() from shutdown hook 18/06/08 06:18:26 INFO scheduler.TaskSetManager: Starting task 175.0 in stage 8.0 (TID 7862, localhost, executor driver, partition 175, PROCESS_LOCAL, 2049 bytes) 18/06/08 06:18:26 INFO executor.Executor: Running task 175.0 in stage 8.0 (TID 7862) 18/06/08 06:18:26 WARN scheduler.TaskSetManager: Lost task 168.0 in stage 8.0 (TID 7855, localhost, executor driver): java.lang.OutOfMemoryError: Java heap space at java.io.ObjectInputStream$HandleTable.grow(ObjectInputStream.java:3464) at java.io.ObjectInputStream$HandleTable.assign(ObjectInputStream.java:3271) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1789) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1706) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1344) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1706) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1344) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76) at org.apache.spark.serializer.DeserializationStream.readValue(Serializer.scala:171) at org.apache.spark.util.collection.ExternalAppendOnlyMap$DiskMapIterator.readNextItem(ExternalAppendOnlyMap.scala:515) at org.apache.spark.util.collection.ExternalAppendOnlyMap$DiskMapIterator.hasNext(ExternalAppendOnlyMap.scala:535) at scala.collection.Iterator$$anon$1.hasNext(Iterator.scala:847) at org.apache.spark.util.collection.ExternalAppendOnlyMap$ExternalIterator.org$apache$spark$util$collection$ExternalAppendOnlyMap$ExternalIterator$$readNextHashCode(ExternalAppendOnlyMap.scala:335) at org.apache.spark.util.collection.ExternalAppendOnlyMap$ExternalIterator$$anonfun$next$1.apply(ExternalAppendOnlyMap.scala:408) at org.apache.spark.util.collection.ExternalAppendOnlyMap$ExternalIterator$$anonfun$next$1.apply(ExternalAppendOnlyMap.scala:406) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at org.apache.spark.util.collection.ExternalAppendOnlyMap$ExternalIterator.next(ExternalAppendOnlyMap.scala:406) at org.apache.spark.util.collection.ExternalAppendOnlyMap$ExternalIterator.next(ExternalAppendOnlyMap.scala:301) at org.apache.spark.InterruptibleIterator.next(InterruptibleIterator.scala:43) at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:200) at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
18/06/08 06:18:26 ERROR scheduler.TaskSetManager: Task 168 in stage 8.0 failed 1 times; aborting job 18/06/08 06:18:26 INFO storage.ShuffleBlockFetcherIterator: Getting 1280 non-empty blocks out of 1280 blocks 18/06/08 06:18:26 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 18/06/08 06:18:26 INFO scheduler.TaskSchedulerImpl: Cancelling stage 12 18/06/08 06:18:26 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 12.0, whose tasks have all completed, from pool 18/06/08 06:18:26 INFO scheduler.TaskSchedulerImpl: Stage 12 was cancelled 18/06/08 06:18:26 INFO scheduler.DAGScheduler: ShuffleMapStage 12 (map at SparkScalaBitcoinTransactionGraph.scala:89) failed in 18010.838 s due to Job aborted due to stage failure: Task 168 in stage 8.0 failed 1 times, most recent failure: Lost task 168.0 in stage 8.0 (TID 7855, localhost, executor driver): java.lang.OutOfMemoryError: Java heap space at java.io.ObjectInputStream$HandleTable.grow(ObjectInputStream.java:3464) at java.io.ObjectInputStream$HandleTable.assign(ObjectInputStream.java:3271) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1789) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1706) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1344) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1706) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1344) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76) at org.apache.spark.serializer.DeserializationStream.readValue(Serializer.scala:171) at org.apache.spark.util.collection.ExternalAppendOnlyMap$DiskMapIterator.readNextItem(ExternalAppendOnlyMap.scala:515) at org.apache.spark.util.collection.ExternalAppendOnlyMap$DiskMapIterator.hasNext(ExternalAppendOnlyMap.scala:535) at scala.collection.Iterator$$anon$1.hasNext(Iterator.scala:847) at org.apache.spark.util.collection.ExternalAppendOnlyMap$ExternalIterator.org$apache$spark$util$collection$ExternalAppendOnlyMap$ExternalIterator$$readNextHashCode(ExternalAppendOnlyMap.scala:335) at org.apache.spark.util.collection.ExternalAppendOnlyMap$ExternalIterator$$anonfun$next$1.apply(ExternalAppendOnlyMap.scala:408) at org.apache.spark.util.collection.ExternalAppendOnlyMap$ExternalIterator$$anonfun$next$1.apply(ExternalAppendOnlyMap.scala:406) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at org.apache.spark.util.collection.ExternalAppendOnlyMap$ExternalIterator.next(ExternalAppendOnlyMap.scala:406) at org.apache.spark.util.collection.ExternalAppendOnlyMap$ExternalIterator.next(ExternalAppendOnlyMap.scala:301) at org.apache.spark.InterruptibleIterator.next(InterruptibleIterator.scala:43) at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:200) at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
Driver stacktrace: 18/06/08 06:18:26 INFO scheduler.TaskSchedulerImpl: Cancelling stage 9 18/06/08 06:18:26 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 9.0, whose tasks have all completed, from pool 18/06/08 06:18:26 INFO scheduler.TaskSchedulerImpl: Stage 9 was cancelled 18/06/08 06:18:26 INFO scheduler.DAGScheduler: ShuffleMapStage 9 (map at SparkScalaBitcoinTransactionGraph.scala:81) failed in 9262.181 s due to Job aborted due to stage failure: Task 168 in stage 8.0 failed 1 times, most recent failure: Lost task 168.0 in stage 8.0 (TID 7855, localhost, executor driver): java.lang.OutOfMemoryError: Java heap space at java.io.ObjectInputStream$HandleTable.grow(ObjectInputStream.java:3464) at java.io.ObjectInputStream$HandleTable.assign(ObjectInputStream.java:3271) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1789) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1706) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1344) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1706) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1344) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76) at org.apache.spark.serializer.DeserializationStream.readValue(Serializer.scala:171) at org.apache.spark.util.collection.ExternalAppendOnlyMap$DiskMapIterator.readNextItem(ExternalAppendOnlyMap.scala:515) at org.apache.spark.util.collection.ExternalAppendOnlyMap$DiskMapIterator.hasNext(ExternalAppendOnlyMap.scala:535) at scala.collection.Iterator$$anon$1.hasNext(Iterator.scala:847) at org.apache.spark.util.collection.ExternalAppendOnlyMap$ExternalIterator.org$apache$spark$util$collection$ExternalAppendOnlyMap$ExternalIterator$$readNextHashCode(ExternalAppendOnlyMap.scala:335) at org.apache.spark.util.collection.ExternalAppendOnlyMap$ExternalIterator$$anonfun$next$1.apply(ExternalAppendOnlyMap.scala:408) at org.apache.spark.util.collection.ExternalAppendOnlyMap$ExternalIterator$$anonfun$next$1.apply(ExternalAppendOnlyMap.scala:406) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at org.apache.spark.util.collection.ExternalAppendOnlyMap$ExternalIterator.next(ExternalAppendOnlyMap.scala:406) at org.apache.spark.util.collection.ExternalAppendOnlyMap$ExternalIterator.next(ExternalAppendOnlyMap.scala:301) at org.apache.spark.InterruptibleIterator.next(InterruptibleIterator.scala:43) at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:200) at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
Driver stacktrace: 18/06/08 06:18:26 INFO scheduler.TaskSchedulerImpl: Cancelling stage 8 18/06/08 06:18:26 INFO scheduler.TaskSchedulerImpl: Stage 8 was cancelled 18/06/08 06:18:26 INFO scheduler.DAGScheduler: ShuffleMapStage 8 (map at SparkScalaBitcoinTransactionGraph.scala:85) failed in 1948.098 s due to Job aborted due to stage failure: Task 168 in stage 8.0 failed 1 times, most recent failure: Lost task 168.0 in stage 8.0 (TID 7855, localhost, executor driver): java.lang.OutOfMemoryError: Java heap space at java.io.ObjectInputStream$HandleTable.grow(ObjectInputStream.java:3464) at java.io.ObjectInputStream$HandleTable.assign(ObjectInputStream.java:3271) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1789) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1706) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1344) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1706) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1344) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76) at org.apache.spark.serializer.DeserializationStream.readValue(Serializer.scala:171) at org.apache.spark.util.collection.ExternalAppendOnlyMap$DiskMapIterator.readNextItem(ExternalAppendOnlyMap.scala:515) at org.apache.spark.util.collection.ExternalAppendOnlyMap$DiskMapIterator.hasNext(ExternalAppendOnlyMap.scala:535) at scala.collection.Iterator$$anon$1.hasNext(Iterator.scala:847) at org.apache.spark.util.collection.ExternalAppendOnlyMap$ExternalIterator.org$apache$spark$util$collection$ExternalAppendOnlyMap$ExternalIterator$$readNextHashCode(ExternalAppendOnlyMap.scala:335) at org.apache.spark.util.collection.ExternalAppendOnlyMap$ExternalIterator$$anonfun$next$1.apply(ExternalAppendOnlyMap.scala:408) at org.apache.spark.util.collection.ExternalAppendOnlyMap$ExternalIterator$$anonfun$next$1.apply(ExternalAppendOnlyMap.scala:406) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at org.apache.spark.util.collection.ExternalAppendOnlyMap$ExternalIterator.next(ExternalAppendOnlyMap.scala:406) at org.apache.spark.util.collection.ExternalAppendOnlyMap$ExternalIterator.next(ExternalAppendOnlyMap.scala:301) at org.apache.spark.InterruptibleIterator.next(InterruptibleIterator.scala:43) at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:200) at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
Driver stacktrace: 18/06/08 06:18:26 INFO executor.Executor: Executor is trying to kill task 172.0 in stage 8.0 (TID 7859) 18/06/08 06:18:26 INFO executor.Executor: Executor is trying to kill task 169.0 in stage 8.0 (TID 7856) 18/06/08 06:18:26 INFO executor.Executor: Executor is trying to kill task 173.0 in stage 8.0 (TID 7860) 18/06/08 06:18:26 INFO executor.Executor: Executor is trying to kill task 170.0 in stage 8.0 (TID 7857) 18/06/08 06:18:26 INFO executor.Executor: Executor is trying to kill task 167.0 in stage 8.0 (TID 7854) 18/06/08 06:18:26 INFO executor.Executor: Executor is trying to kill task 174.0 in stage 8.0 (TID 7861) 18/06/08 06:18:26 INFO executor.Executor: Executor is trying to kill task 171.0 in stage 8.0 (TID 7858) 18/06/08 06:18:26 INFO executor.Executor: Executor is trying to kill task 175.0 in stage 8.0 (TID 7862) 18/06/08 06:18:26 INFO scheduler.DAGScheduler: Job 1 failed: top at SparkScalaBitcoinTransactionGraph.scala:100, took 18011.687877 s 18/06/08 06:18:26 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1280 blocks 18/06/08 06:18:26 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 168 in stage 8.0 failed 1 times, most recent failure: Lost task 168.0 in stage 8.0 (TID 7855, localhost, executor driver): java.lang.OutOfMemoryError: Java heap space at java.io.ObjectInputStream$HandleTable.grow(ObjectInputStream.java:3464) at java.io.ObjectInputStream$HandleTable.assign(ObjectInputStream.java:3271) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1789) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1706) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1344) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1706) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1344) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76) at org.apache.spark.serializer.DeserializationStream.readValue(Serializer.scala:171) at org.apache.spark.util.collection.ExternalAppendOnlyMap$DiskMapIterator.readNextItem(ExternalAppendOnlyMap.scala:515) at org.apache.spark.util.collection.ExternalAppendOnlyMap$DiskMapIterator.hasNext(ExternalAppendOnlyMap.scala:535) at scala.collection.Iterator$$anon$1.hasNext(Iterator.scala:847) at org.apache.spark.util.collection.ExternalAppendOnlyMap$ExternalIterator.org$apache$spark$util$collection$ExternalAppendOnlyMap$ExternalIterator$$readNextHashCode(ExternalAppendOnlyMap.scala:335) at org.apache.spark.util.collection.ExternalAppendOnlyMap$ExternalIterator$$anonfun$next$1.apply(ExternalAppendOnlyMap.scala:408) at org.apache.spark.util.collection.ExternalAppendOnlyMap$ExternalIterator$$anonfun$next$1.apply(ExternalAppendOnlyMap.scala:406) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at org.apache.spark.util.collection.ExternalAppendOnlyMap$ExternalIterator.next(ExternalAppendOnlyMap.scala:406) at org.apache.spark.util.collection.ExternalAppendOnlyMap$ExternalIterator.next(ExternalAppendOnlyMap.scala:301) at org.apache.spark.InterruptibleIterator.next(InterruptibleIterator.scala:43) at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:200) at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1457) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1445) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1444) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1444) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799) at scala.Option.foreach(Option.scala:236) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1668) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1627) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1616) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1862) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1982) at org.apache.spark.rdd.RDD$$anonfun$reduce$1.apply(RDD.scala:1025) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111) at org.apache.spark.rdd.RDD.withScope(RDD.scala:316) at org.apache.spark.rdd.RDD.reduce(RDD.scala:1007) at org.apache.spark.rdd.RDD$$anonfun$takeOrdered$1.apply(RDD.scala:1397) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111) at org.apache.spark.rdd.RDD.withScope(RDD.scala:316) at org.apache.spark.rdd.RDD.takeOrdered(RDD.scala:1384) at org.apache.spark.rdd.RDD$$anonfun$top$1.apply(RDD.scala:1365) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111) at org.apache.spark.rdd.RDD.withScope(RDD.scala:316) at org.apache.spark.rdd.RDD.top(RDD.scala:1364) at org.zuinnote.spark.bitcoin.example.SparkScalaBitcoinTransactionGraph$.jobTop5AddressInput(SparkScalaBitcoinTransactionGraph.scala:100) at org.zuinnote.spark.bitcoin.example.SparkScalaBitcoinTransactionGraph$.main(SparkScalaBitcoinTransactionGraph.scala:53) at org.zuinnote.spark.bitcoin.example.SparkScalaBitcoinTransactionGraph.main(SparkScalaBitcoinTransactionGraph.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:730) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.OutOfMemoryError: Java heap space at java.io.ObjectInputStream$HandleTable.grow(ObjectInputStream.java:3464) at java.io.ObjectInputStream$HandleTable.assign(ObjectInputStream.java:3271) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1789) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1706) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1344) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1706) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1344) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76) at org.apache.spark.serializer.DeserializationStream.readValue(Serializer.scala:171) at org.apache.spark.util.collection.ExternalAppendOnlyMap$DiskMapIterator.readNextItem(ExternalAppendOnlyMap.scala:515) at org.apache.spark.util.collection.ExternalAppendOnlyMap$DiskMapIterator.hasNext(ExternalAppendOnlyMap.scala:535) at scala.collection.Iterator$$anon$1.hasNext(Iterator.scala:847) at org.apache.spark.util.collection.ExternalAppendOnlyMap$ExternalIterator.org$apache$spark$util$collection$ExternalAppendOnlyMap$ExternalIterator$$readNextHashCode(ExternalAppendOnlyMap.scala:335) at org.apache.spark.util.collection.ExternalAppendOnlyMap$ExternalIterator$$anonfun$next$1.apply(ExternalAppendOnlyMap.scala:408) at org.apache.spark.util.collection.ExternalAppendOnlyMap$ExternalIterator$$anonfun$next$1.apply(ExternalAppendOnlyMap.scala:406) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at org.apache.spark.util.collection.ExternalAppendOnlyMap$ExternalIterator.next(ExternalAppendOnlyMap.scala:406) at org.apache.spark.util.collection.ExternalAppendOnlyMap$ExternalIterator.next(ExternalAppendOnlyMap.scala:301) at org.apache.spark.InterruptibleIterator.next(InterruptibleIterator.scala:43) at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:200) at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73) 18/06/08 06:18:26 INFO executor.Executor: Executor killed task 175.0 in stage 8.0 (TID 7862) 18/06/08 06:18:26 WARN scheduler.TaskSetManager: Lost task 175.0 in stage 8.0 (TID 7862, localhost, executor driver): TaskKilled (killed intentionally) 18/06/08 06:18:26 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/metrics/json,null} 18/06/08 06:18:26 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/kill,null} 18/06/08 06:18:26 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/api,null} 18/06/08 06:18:26 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/,null} 18/06/08 06:18:26 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/static,null} 18/06/08 06:18:26 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump/json,null} 18/06/08 06:18:26 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump,null} 18/06/08 06:18:26 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/json,null} 18/06/08 06:18:26 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors,null} 18/06/08 06:18:26 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment/json,null} 18/06/08 06:18:26 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment,null} 18/06/08 06:18:26 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd/json,null} 18/06/08 06:18:26 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd,null} 18/06/08 06:18:26 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/json,null} 18/06/08 06:18:26 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage,null} 18/06/08 06:18:26 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool/json,null} 18/06/08 06:18:26 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool,null} 18/06/08 06:18:26 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/json,null} 18/06/08 06:18:26 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage,null} 18/06/08 06:18:26 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/json,null} 18/06/08 06:18:26 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages,null} 18/06/08 06:18:26 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job/json,null} 18/06/08 06:18:26 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job,null} 18/06/08 06:18:26 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/json,null} 18/06/08 06:18:26 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs,null} 18/06/08 06:18:26 INFO ui.SparkUI: Stopped Spark web UI at http://172.17.0.2:4040 18/06/08 06:18:26 INFO executor.Executor: Executor killed task 170.0 in stage 8.0 (TID 7857) 18/06/08 06:18:26 WARN scheduler.TaskSetManager: Lost task 170.0 in stage 8.0 (TID 7857, localhost, executor driver): TaskKilled (killed intentionally) 18/06/08 06:18:26 INFO executor.Executor: Executor killed task 172.0 in stage 8.0 (TID 7859) 18/06/08 06:18:26 ERROR scheduler.TaskSchedulerImpl: Exception in statusUpdate java.util.concurrent.RejectedExecutionException: Task org.apache.spark.scheduler.TaskResultGetter$$anon$3@3eec2833 rejected from java.util.concurrent.ThreadPoolExecutor@67fd2e27[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7857] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372) at org.apache.spark.scheduler.TaskResultGetter.enqueueFailedTask(TaskResultGetter.scala:104) at org.apache.spark.scheduler.TaskSchedulerImpl.liftedTree2$1(TaskSchedulerImpl.scala:387) at org.apache.spark.scheduler.TaskSchedulerImpl.statusUpdate(TaskSchedulerImpl.scala:361) at org.apache.spark.scheduler.local.LocalEndpoint$$anonfun$receive$1.applyOrElse(LocalBackend.scala:66) at org.apache.spark.rpc.netty.Inbox$$anonfun$process$1.apply$mcV$sp(Inbox.scala:116) at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:204) at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:100) at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:217) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 18/06/08 06:18:26 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 18/06/08 06:18:26 INFO executor.Executor: Executor killed task 171.0 in stage 8.0 (TID 7858) 18/06/08 06:18:26 INFO executor.Executor: Executor killed task 173.0 in stage 8.0 (TID 7860) 18/06/08 06:18:26 INFO executor.Executor: Executor killed task 174.0 in stage 8.0 (TID 7861) 18/06/08 06:18:27 ERROR executor.Executor: Exception in task 169.0 in stage 8.0 (TID 7856) java.lang.IllegalArgumentException: requirement failed: File segment length cannot be negative (got -16024861) at scala.Predef$.require(Predef.scala:233) at org.apache.spark.storage.FileSegment.
(FileSegment.scala:28) at org.apache.spark.storage.DiskBlockObjectWriter.fileSegment(DiskBlockObjectWriter.scala:236) at org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$2.apply(ExternalSorter.scala:729) at org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$2.apply(ExternalSorter.scala:721) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) at org.apache.spark.util.collection.ExternalSorter.writePartitionedFile(ExternalSorter.scala:721) at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:73) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) at org.apache.spark.scheduler.Task.run(Task.scala:89) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:242) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 18/06/08 06:18:27 INFO executor.Executor: Not reporting error to driver during JVM shutdown. 18/06/08 06:18:27 ERROR executor.Executor: Exception in task 167.0 in stage 8.0 (TID 7854) java.io.FileNotFoundException: /tmp/blockmgr-3d6f28c7-9d39-4c84-ad36-cd4864d0474f/1a/temp_shuffle_5b3ac818-3769-4488-8a4a-1bd053c342d6 (No such file or directory) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream. (FileInputStream.java:146) at org.apache.spark.util.collection.ExternalSorter$SpillReader.nextBatchStream(ExternalSorter.scala:526) at org.apache.spark.util.collection.ExternalSorter$SpillReader.org$apache$spark$util$collection$ExternalSorter$SpillReader$$readNextItem(ExternalSorter.scala:578) at org.apache.spark.util.collection.ExternalSorter$SpillReader$$anon$5.hasNext(ExternalSorter.scala:601) at scala.collection.TraversableOnce$FlattenOps$$anon$1.hasNext(TraversableOnce.scala:432) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) at org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$2.apply(ExternalSorter.scala:725) at org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$2.apply(ExternalSorter.scala:721) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) at org.apache.spark.util.collection.ExternalSorter.writePartitionedFile(ExternalSorter.scala:721) at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:73) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) at org.apache.spark.scheduler.Task.run(Task.scala:89) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:242) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 18/06/08 06:18:27 INFO executor.Executor: Not reporting error to driver during JVM shutdown. 18/06/08 06:18:33 INFO storage.MemoryStore: MemoryStore cleared 18/06/08 06:18:33 INFO storage.BlockManager: BlockManager stopped 18/06/08 06:18:33 INFO storage.BlockManagerMaster: BlockManagerMaster stopped 18/06/08 06:18:33 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 18/06/08 06:18:33 INFO spark.SparkContext: Successfully stopped SparkContext 18/06/08 06:18:33 INFO util.ShutdownHookManager: Shutdown hook called 18/06/08 06:18:33 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-7aa73177-a289-47ce-ab1a-f2c3432aa428 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.
foonsun
@jornfranke I have run the example according to the wiki: https://github.com/ZuInnoTe/hadoopcryptoledger/wiki/Using-Spark-Scala-Graphx-to-analyze-the-Bitcoin-transaction-graph
it has run over 4 hours,and the issue occurs below. and i run it again for about 4hours.the issue still happens. it seems that the memory is not enough.but the memory of my docker is enough big.
how i can deal with this problem?thanks very much.