search

stephenh / mirror

A tool for real-time, two-way sync for remote (e.g. desktop/laptop) development
Apache License 2.0
391 stars 37 forks source link

"OutOfMemoryError: Java heap space" error while running mirror #47

Closed rleon closed 4 years ago

rleon commented 4 years ago

Hi,

My server and client have such an error while syncing. server: _ src free -h total used free shared buff/cache available Mem: 62Gi 3.7Gi 36Gi 1.0Mi 22Gi 58Gi Swap: 2.0Gi 0B 2.0Gi

java.lang.OutOfMemoryError: Java heap space Dumping heap to java_pid21.hprof ... Unable to create java_pid21.hprof: Permission denied Exception in thread "4-SaveToRemote-0" java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOfRange(Arrays.java:3520) at com.google.protobuf.ByteString$ArraysByteArrayCopier.copyFrom(ByteString.java:117) at com.google.protobuf.ByteString.copyFrom(ByteString.java:353) at com.google.protobuf.ByteString.readChunk(ByteString.java:543) at com.google.protobuf.ByteString.readFrom(ByteString.java:508) at com.google.protobuf.ByteString.readFrom(ByteString.java:476) at mirror.NativeFileAccess.read(NativeFileAccess.java:61) at mirror.SaveToRemote.sendToRemote(SaveToRemote.java:54) at mirror.SaveToRemote.runOneLoop(SaveToRemote.java:36) at mirror.tasks.ThreadBasedTask.run(ThreadBasedTask.java:62) at mirror.tasks.ThreadBasedTask.lambda$new$0(ThreadBasedTask.java:39) at mirror.tasks.ThreadBasedTask$$Lambda$17/93472147.run(Unknown Source) at java.lang.Thread.run(Thread.java:748)

client: ➜ src free -h total used free shared buff/cache available Mem: 15Gi 5.1Gi 3.1Gi 1.0Gi 7.4Gi 9.1Gi Swap: 7.8Gi 540Mi 7.3Gi

java.lang.OutOfMemoryError: Java heap space Dumping heap to java_pid21.hprof ... Unable to create java_pid21.hprof: Permission denied Exception in thread "5-SaveToRemote-0" java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOfRange(Arrays.java:3520) at com.google.protobuf.ByteString$ArraysByteArrayCopier.copyFrom(ByteString.java:117) at com.google.protobuf.ByteString.copyFrom(ByteString.java:353) at com.google.protobuf.ByteString.readChunk(ByteString.java:543) at com.google.protobuf.ByteString.readFrom(ByteString.java:508) at com.google.protobuf.ByteString.readFrom(ByteString.java:476) at mirror.NativeFileAccess.read(NativeFileAccess.java:61) at mirror.SaveToRemote.sendToRemote(SaveToRemote.java:54) at mirror.SaveToRemote.runOneLoop(SaveToRemote.java:36) at mirror.tasks.ThreadBasedTask.run(ThreadBasedTask.java:62) at mirror.tasks.ThreadBasedTask.lambda$new$0(ThreadBasedTask.java:39) at mirror.tasks.ThreadBasedTask$$Lambda$13/1990160809.run(Unknown Source) at java.lang.Thread.run(Thread.java:748) According to http://javaeesupportpatterns.blogspot.com/2011/08/gc-overhead-limit-exceeded-problem-and.html, there is a leek in application.

stephenh commented 4 years ago

What's the biggest file / files you're trying to sync?

It is technically possible, but it's fairly unlikely mirror has a true memory leak.

What's more likely is that either you're sending a large file / series of large files that can't fit in the JVM heap space that the mirror client/server process have allocated.

You can try increasing the heap size (see the -Xmx2G flag in the mirror file), or potentially not syncing the really large files (via an include/exclude pattern).

In theory mirror could/should do streaming / or even zero-copying of large files, but at the time I wrote mirror, the underlying grpc-java framework didn't support this. Not sure if they do now.

Another thing that could be going wrong is that, instead of a single individual file being too large, mirror is trying to send too many files at once, i.e. loading ~100/~1000 some files in memory at a time, however there is some flow control (based on the underlying grpc flow control) that is supposed to keep this from happening, but its never been enough of a problem for me that I had to add instrumentation/metrics to have visibility into that aspect of things.

stephenh commented 4 years ago

Unable to create java_pid21.hprof: Permission denied

You could also try and fix this permission error so that the hprof file can be created, and then look inside to see what's taking up the heap.

rleon commented 4 years ago

The largest file is 1.5G and the next is 228M.

My goal is to sync my working directory (linux kernel and qemu gits). Right now, my workaround is to put ".git" to be out of reach for "mirror". So I keep git on my laptop, while pushing only sources to the server for the build and CI.

Thanks

stephenh commented 4 years ago

Cool; glad you have a work around; going with the assumption it was that large file causing issues, I'm going to close this in favor of a feature ticket, #48.