search

zioc / contrail-devstack-plugin

6 stars 21 forks source link

Cassandra #61

Closed DEEM78 closed 5 years ago

DEEM78 commented 5 years ago

I am having a recurrent issue with Cassandra which crashes a few minutes after each restart attempt. I am running a Devstack Queens+ Contrail Master and, ubuntu Xenial.

Tried to google on the problem but didn't find a suitable solution

Below the error I am getting

ERROR [SASI-General:3] 2018-10-30 14:34:57,484 JVMStabilityInspector.java:139 - JVM state determined to be unstable. Exiting forcefully due to: java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3181) ~[na:1.8.0_181] at java.util.ArrayList.grow(ArrayList.java:265) ~[na:1.8.0_181] at java.util.ArrayList.ensureExplicitCapacity(ArrayList.java:239) ~[na:1.8.0_181] at java.util.ArrayList.ensureCapacityInternal(ArrayList.java:231) ~[na:1.8.0_181] at java.util.ArrayList.add(ArrayList.java:462) ~[na:1.8.0_181] at org.apache.cassandra.index.sasi.sa.SA.add(SA.java:51) ~[apache-cassandra-3.4.jar:3.4] at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.finish(OnDiskIndexBuilder.java:242) ~[apache-cassandra-3.4.jar:3.4] at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.finish(OnDiskIndexBuilder.java:227) ~[apache-cassandra-3.4.jar:3.4] at org.apache.cassandra.index.sasi.disk.PerSSTableIndexWriter$Index.lambda$scheduleSegmentFlush$260(PerSSTableIndexWriter.java:253) ~[apache-cassandra-3.4.jar:3.4] at org.apache.cassandra.index.sasi.disk.PerSSTableIndexWriter$Index$$Lambda$221/1995845129.call(Unknown Source) ~[na:na] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_181] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_181] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_181] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181] INFO [Service Thread] 2018-10-30 14:34:57,484 StatusLogger.java:106 - svc_monitor_keyspace.healthmonitor_table 0,0 INFO [ScheduledTasks:1] 2018-10-30 14:34:57,484 StatusLogger.java:106 - system.peers 0,0 INFO [Service Thread] 2018-10-30 14:34:57,484 StatusLogger.java:106 - svc_monitor_keyspace.pool_table 0,0 INFO [ScheduledTasks:1] 2018-10-30 14:34:57,484 StatusLogger.java:106 - system.batches 0,0 INFO [Service Thread] 2018-10-30 14:34:57,485 StatusLogger.java:106 - svc_monitor_keyspace.service_instance_table 0,0 INFO [ScheduledTasks:1] 2018-10-30 14:34:57,485 StatusLogger.java:106 - system.schema_keyspaces 0,0 INFO [Service Thread] 2018-10-30 14:34:57,485 StatusLogger.java:106 - svc_monitor_keyspace.loadbalancer_table 0,0 INFO [ScheduledTasks:1] 2018-10-30 14:34:57,485 StatusLogger.java:106 - system.schema_usertypes 0,0 INFO [Service Thread] 2018-10-30 14:34:57,485 StatusLogger.java:106 - system_auth.roles 0,0 INFO [ScheduledTasks:1] 2018-10-30 14:34:57,485 StatusLogger.java:106 - system.local 0,0 INFO [Service Thread] 2018-10-30 14:34:57,486 StatusLogger.java:106 - system_auth.role_members 0,0 INFO [ScheduledTasks:1] 2018-10-30 14:34:57,486 StatusLogger.java:106 - system.sstable_activity 193,2818 INFO [Service Thread] 2018-10-30 14:34:57,486 StatusLogger.java:106 - system_auth.resource_role_permissons_index 0,0 INFO [ScheduledTasks:1] 2018-10-30 14:34:57,486 StatusLogger.java:106 - system.available_ranges 0,0 INFO [Service Thread] 2018-10-30 14:34:57,486 StatusLogger.java:106 - system_auth.role_permissions 0,0

Your help is appreciated

ethuleau commented 5 years ago

From the log we can see Java virtual machine exits forcefully because it gets out of memory java.lang.OutOfMemoryError: Java heap space. You have probably not enough memory allocated to Cassandra. By default, that script configures Cassandra to run Java virtual machine with 500Mo for the heap but you can increase it with the devstack config flags https://github.com/zioc/contrail-devstack-plugin/blob/master/devstack/settings#L67-L68. Be careful to have enough memory on your host or linux will kill jvm itself.