akiradeveloper / akashic-storage

Filey system backed S3-compatible storage in Scala/Akka-Http
Apache License 2.0
22 stars 2 forks source link

memory fault kills the daemon #64

Open akiradeveloper opened 8 years ago

akiradeveloper commented 8 years ago

I tried to put an object of size 160MB. the server's resource is tiny: memory 1GB and 1 cpu. Set smaller value for the put size makes sense but what size it should be?

To begin with, the memory allocation error shouldn't kill the daemon in any way. This showstopper. should be fixed before 1.0

Uncaught error from thread [akashic-storage-akka.actor.default-dispatcher-31] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[akashic-storage]
java.lang.OutOfMemoryError: Java heap space
        at scala.reflect.ManifestFactory$$anon$6.newArray(Manifest.scala:93)
        at scala.reflect.ManifestFactory$$anon$6.newArray(Manifest.scala:91)
        at scala.Array$.ofDim(Array.scala:218)
        at akka.util.ByteIterator.toArray(ByteIterator.scala:459)
        at akka.util.ByteString.toArray(ByteString.scala:430)
        at akka.http.scaladsl.unmarshalling.PredefinedFromEntityUnmarshallers$$anonfun$byteArrayUnmarshaller$1.apply(PredefinedFromEntityUnmarshallers.scala:20)
        at akka.http.scaladsl.unmarshalling.PredefinedFromEntityUnmarshallers$$anonfun$byteArrayUnmarshaller$1.apply(PredefinedFromEntityUnmarshallers.scala:20)
        at akka.http.scaladsl.util.FastFuture$$anonfun$map$extension$1.apply(FastFuture.scala:23)
        at akka.http.scaladsl.util.FastFuture$$anonfun$map$extension$1.apply(FastFuture.scala:23)
        at akka.http.scaladsl.util.FastFuture$.akka$http$scaladsl$util$FastFuture$$strictTransform$1(FastFuture.scala:41)
        at akka.http.scaladsl.util.FastFuture$$anonfun$transformWith$extension1$1.apply(FastFuture.scala:51)
        at akka.http.scaladsl.util.FastFuture$$anonfun$transformWith$extension1$1.apply(FastFuture.scala:50)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
        at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
        at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
        at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
        at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
        at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
        at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:405)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Service exit with a return value of 255
akiradeveloper commented 8 years ago

I think this was because the daemon wasn't set up the max-content-length. it is by default MB so sending 160MB object will fault. But still strange it was killed by OOM

Anyway, this is not frequent in production. So will drop this off from v1