JVM crashed hard while feeding

[karussell]

Although the data on disc is under 200 MB I got the following error indicating too few RAM or SWAP space when the JVM crashed hard:

A fatal error has been detected by the Java Runtime Environment:

java.lang.OutOfMemoryError: requested 536870920 bytes for Chunk::new. Out of swap space?

Internal Error (allocation.cpp:215), pid=3080, tid=1023273840 Error: Chunk::new

JRE version: 6.0_22-b04 Java VM: Java HotSpot(TM) Server VM (17.1-b03 mixed mode linux-x86 ) If you would like to submit a bug report, please visit: http://java.sun.com/webapps/bugreport/crash.jsp

--------------- T H R E A D ---------------

Current thread (0x096d5400): JavaThread "CompilerThread1" daemon [_thread_in_native, id=3091, stack(0x3cf5e000,0x3cfdf000)]

Stack: [0x3cf5e000,0x3cfdf000], sp=0x3cfdc150, free space=1f83cfdf000k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [libjvm.so+0x6a92d2] V [libjvm.so+0x2b27cf] V [libjvm.so+0x12e08c] V [libjvm.so+0x12e586] V [libjvm.so+0x5aa97b] V [libjvm.so+0x5aa6ff] V [libjvm.so+0x5acd3f] V [libjvm.so+0x5ac780] V [libjvm.so+0x27b71b] V [libjvm.so+0x278093] V [libjvm.so+0x2097b7] V [libjvm.so+0x280fdc] V [libjvm.so+0x280889] V [libjvm.so+0x66ff16] V [libjvm.so+0x6695fe] V [libjvm.so+0x57a8fe] C [libpthread.so.0+0x580e]

Current CompileTask: C2:2085 org.apache.cassandra.db.ColumnIndexer.serializeInternal(Lorg/apache/cassandra/io/util/IIterableColumns;Ljava/io/DataOutput;)V (395 bytes)

--------------- P R O C E S S ---------------

Java Threads: ( => current thread ) 0x09c68800 JavaThread "pool-6-thread-107" [_thread_blocked, id=3725, stack(0x3a9be000,0x3a9df000)] 0x09c68000 JavaThread "pool-6-thread-106" [_thread_blocked, id=3724, stack(0x3ca3a000,0x3ca5b000)] 0x09915800 JavaThread "btpool0-10" [_thread_blocked, id=3489, stack(0x3a93a000,0x3a95b000)] 0x09c79800 JavaThread "MultiThreadedHttpConnectionManager cleanup" daemon [_thread_blocked, id=3208, stack(0x3a9df000,0x3aa00000)] 0x0998e000 JavaThread "pool-5-thread-1" [_thread_blocked, id=3205, stack(0x3ad4d000,0x3ad6e000)] 0x098ba800 JavaThread "pool-4-thread-1" [_thread_blocked, id=3204, stack(0x3ad6e000,0x3ad8f000)] 0x3b8b3800 JavaThread "DestroyJavaVM" [_thread_blocked, id=3081, stack(0xb6b5b000,0xb6b7c000)] 0x3b8a2400 JavaThread "WRITE-/127.0.0.1" [_thread_blocked, id=3203, stack(0x3ad8f000,0x3adb0000)] 0x3b8a1400 JavaThread "WRITE-/127.0.0.1" [_thread_blocked, id=3202, stack(0x3adb0000,0x3add1000)] 0x3b8a0800 JavaThread "ACCEPT-peters-laptop/127.0.1.1" [_thread_in_native, id=3201, stack(0x3add1000,0x3adf2000)] 0x0986dc00 JavaThread "CompactionExecutor:1" [_thread_blocked, id=3200, stack(0x3adf2000,0x3ae13000)] 0x3b89c800 JavaThread "MiscStage:1" [_thread_blocked, id=3199, stack(0x3ae13000,0x3ae34000)] 0x3b89b000 JavaThread "MigrationStage:1" [_thread_blocked, id=3198, stack(0x3ae34000,0x3ae55000)] 0x3b899800 JavaThread "AntiEntropyStage:1" [_thread_blocked, id=3197, stack(0x3ae55000,0x3ae76000)] 0x3b898000 JavaThread "GossipStage:1" [_thread_blocked, id=3196, stack(0x3ae76000,0x3ae97000)] 0x3b896800 JavaThread "StreamStage:1" [_thread_blocked, id=3195, stack(0x3ae97000,0x3aeb8000)] 0x3b895000 JavaThread "InternalResponseStage:2" [_thread_blocked, id=3194, stack(0x3aeb8000,0x3aed9000)] 0x3b893c00 JavaThread "InternalResponseStage:1" [_thread_blocked, id=3193, stack(0x3aed9000,0x3aefa000)] 0x3b892400 JavaThread "RequestResponseStage:2" [_thread_blocked, id=3192, stack(0x3aefa000,0x3af1b000)] 0x3b890c00 JavaThread "RequestResponseStage:1" [_thread_blocked, id=3191, stack(0x3af1b000,0x3af3c000)] 0x3b88f400 JavaThread "ReadStage:32" [_thread_blocked, id=3190, stack(0x3af3c000,0x3af5d000)] 0x3b88dc00 JavaThread "ReadStage:31" [_thread_blocked, id=3189, stack(0x3af5d000,0x3af7e000)] ...

0x097bac00 JavaThread "ReadRepair:1" [_thread_blocked, id=3114, stack(0x3bb51000,0x3bb72000)] 0x3b837000 JavaThread "Thread-16" [_thread_in_native, id=3113, stack(0x3bb86000,0x3bba7000)] 0x3b839000 JavaThread "pool-2-thread-1" [_thread_blocked, id=3112, stack(0x3bba7000,0x3bbc8000)] 0x3b843800 JavaThread "pool-1-thread-1" [_thread_blocked, id=3111, stack(0x3b9cc000,0x3b9ed000)] 0x0995fc00 JavaThread "Timer-2" daemon [_thread_blocked, id=3110, stack(0x3c64e000,0x3c66f000)] 0x3cb48400 JavaThread "btpool0-9" [_thread_blocked, id=3108, stack(0x3c8df000,0x3c900000)]
0x3cd0d400 JavaThread "btpool0-8" [_thread_in_native, id=3107, stack(0x3ca19000,0x3ca3a000)] 0x3cd88400 JavaThread "btpool0-6" [_thread_in_native, id=3105, stack(0x3ca5b000,0x3ca7c000)] 0x3cc59800 JavaThread "btpool0-5" [_thread_in_native, id=3104, stack(0x3ca7c000,0x3ca9d000)] 0x3cdfa400 JavaThread "btpool0-4" [_thread_blocked, id=3103, stack(0x3ca9d000,0x3cabe000)] 0x3cc30000 JavaThread "btpool0-3" [_thread_blocked, id=3102, stack(0x3cabe000,0x3cadf000)] 0x3ccadc00 JavaThread "btpool0-2 - Acceptor0 SocketConnector@0.0.0.0:8983" [_thread_in_native, id=3101, stack(0x3cadf000,0x3cb00000)] 0x0979f000 JavaThread "btpool0-1" [_thread_blocked, id=3100, stack(0x3ce00000,0x3ce21000)] 0x09877000 JavaThread "btpool0-0" [_thread_blocked, id=3099, stack(0x3ce21000,0x3ce42000)] 0x0978a000 JavaThread "Timer-1" daemon [_thread_blocked, id=3098, stack(0x3ce42000,0x3ce63000)] 0x09780c00 JavaThread "Timer-0" daemon [_thread_blocked, id=3097, stack(0x3ce63000,0x3ce84000)] 0x3d0fb000 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=3095, stack(0x3cf1c000,0x3cf3d000)]
0x3d0f7000 JavaThread "RMI TCP Accept-8084" daemon [_thread_in_native, id=3094, stack(0x3cf3d000,0x3cf5e000)] 0x3d08fc00 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=3093, stack(0x3d11e000,0x3d13f000)]
0x3d001000 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=3092, stack(0x3d153000,0x3d174000)]
=>0x096d5400 JavaThread "CompilerThread1" daemon [_thread_in_native, id=3091, stack(0x3cf5e000,0x3cfdf000)] 0x096d3c00 JavaThread "CompilerThread0" daemon [_thread_blocked, id=3090, stack(0x3d174000,0x3d1f5000)]
0x096d2000 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=3089, stack(0x3cfdf000,0x3d000000)] 0x096d0800 JavaThread "Surrogate Locker Thread (CMS)" daemon [_thread_blocked, id=3088, stack(0x3d1f5000,0x3d216000)] 0x096bd000 JavaThread "Finalizer" daemon [_thread_blocked, id=3087, stack(0x3d25c000,0x3d27d000)] 0x096bb800 JavaThread "Reference Handler" daemon [_thread_blocked, id=3086, stack(0x3d27d000,0x3d29e000)]

Other Threads: 0x096b9000 VMThread [stack: 0x3d29e000,0x3d31f000] [id=3085] 0x3d0fd400 WatcherThread [stack: 0x3ce9b000,0x3cf1c000] [id=3096]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap par new generation total 29504K, used 23569K [0x41910000, 0x43910000, 0x43910000) eden space 26240K, 84% used [0x41910000, 0x42e9a248, 0x432b0000) from space 3264K, 46% used [0x432b0000, 0x4342a1c8, 0x435e0000) to space 3264K, 0% used [0x435e0000, 0x435e0000, 0x43910000) concurrent mark-sweep generation total 1769472K, used 1041301K [0x43910000, 0xaf910000, 0xaf910000) concurrent-mark-sweep perm gen total 34468K, used 20701K [0xaf910000, 0xb1ab9000, 0xb3910000)

Dynamic libraries: 08048000-08052000 r-xp 00000000 08:07 394700 /usr/lib/jvm/java-6-sun-1.6.0.22/jre/bin/java 08052000-08053000 rwxp 00009000 08:07 394700 /usr/lib/jvm/java-6-sun-1.6.0.22/jre/bin/java 095ee000-0a40c000 rwxp 00000000 00:00 0 [heap] 18ff9000-38c00000 rwxp 00000000 00:00 0

...

VM Arguments: jvm_args: -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1759M -Xmx1759M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC -XX:+UseConcMarkSweep ... DEFAULT ... -Dshards.at.once=2 java_command: start.jar etc/jetty-logging.xml etc/jetty.xml Launcher Type: SUN_STANDARD --------------- S Y S T E M ---------------

OS:squeeze/sid

uname:Linux 2.6.31-22-generic #69-Ubuntu SMP Wed Nov 24 08:51:08 UTC 2010 i686 libc:glibc 2.10.1 NPTL 2.10.1 rlimit: STACK 8192k, CORE 0k, NPROC infinity, NOFILE 1024, AS infinity
load average:3.57 2.73 2.29

CPU:total 2 (2 cores per cpu, 1 threads per core) family 6 model 23 stepping 10, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1

Memory: 4k page, physical 3602792k(104512k free), swap 7815580k(7647756k free)

vm_info: Java HotSpot(TM) Server VM (17.1-b03) for linux-x86 JRE (1.6.0_22-b04), built on Sep 15 2010 01:02:09 by "java_re" with gcc 3.2.1-7a (J2SE release)

time: Thu Jan 6 13:52:18 2011 elapsed time: 1896 seconds

[tjake]

Are you only indexing at this point?

[tjake]

For a lower memory machine (1.7 gigs is low) you can:

change the following in cassandra.yaml

The threshold size in megabytes the binary memtable must grow to,

before it's submitted for flushing to disk.

binary_memtable_throughput_in_mb: 256

You can also change the caching parameters in solandra.cml

[karussell]

Yes, only indexing. I'll try your suggestion

[karussell]

"1.7 gigs is low", ok. I hoped that cassandra wouldn't consume that much RAM with less data in the index

[tjake]

Well both Solr and Cassandra running on the same box will require more memory than just Cassandra or Solr :)

[karussell]

Well both Solr and Cassandra running on the same box

ok

Yes, only indexing. I'll try your suggestion

forget about this: I'm querying before indexing e.g. to count retweets of existing tweets etc

so: I'm heavily querying and then indexing (in an "endless loop"). I even have the unnecessary facets added (they are only necessary for normal queries from UI). This could lead to heavy RAM use!?

[tjake]

Can you give me an idea of how many documents you have? how much ram do you see solr using for the same?

[karussell]

Not that many: that were about 1k-10k

Tried you suggestions: only one shard + the memory setting to only 160 mb

The jvm didn't crash yet. Still having some issues ... I'll investigate :)

how much ram do you see solr using for the same?

you mean pure solr? I have no problems to run jetwick with 1 mio docs/tweets for the same RAM settings on my laptop@home

[tjake]

closed