willtsoft
commented
3 years ago comment out: //config.set("fs.s3a.multipart.size", "100") //config.set("fs.s3a.threads.core", "10") //config.set("fs.s3a.block.size", "32")
seems to work locally
Closed willtsoft closed 3 years ago
willtsoft
commented
3 years ago comment out: //config.set("fs.s3a.multipart.size", "100") //config.set("fs.s3a.threads.core", "10") //config.set("fs.s3a.block.size", "32")
seems to work locally
When using entire Warc File. Almost finishes and then I get Out of memory java.lang.OutOfMemoryError: Java heap space
maybe we can use spark.executor.memory=6g, maybe be able to use this
found on Amazon
java.lang.OutOfMemoryError: Java heap space
-XX:OnOutOfMemoryError="kill -9 %p"
Executing /bin/sh -c "kill -9 2138"...