search

scylladb / scylla-jmx

Scylla JMX proxy
GNU Affero General Public License v3.0
29 stars 54 forks source link

scylla-jmx process crashes on out of memory #50

Closed slivne closed 6 years ago

slivne commented 6 years ago

scylla 2.1.1

Apr 23 23:55:01 ... scylla-jmx[10764]: Exception in thread "main" javax.management.RuntimeErrorException: Error thrown in preRegister method Apr 23 23:55:01 ... scylla-jmx[10764]: at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.throwMBeanRegistrationException(DefaultMBeanServerInterceptor.java:9 88) Apr 23 23:55:01 ... scylla-jmx[10764]: at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.preRegister(DefaultMBeanServerInterceptor.java:1009) Apr 23 23:55:01 ... scylla-jmx[10764]: at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:919) Apr 23 23:55:01 ... scylla-jmx[10764]: at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900) Apr 23 23:55:01 ... scylla-jmx[10764]: at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324) Apr 23 23:55:01 ... scylla-jmx[10764]: at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522) Apr 23 23:55:01 ... scylla-jmx[10764]: at com.scylladb.jmx.metrics.APIMBean.checkRegistration(APIMBean.java:78) Apr 23 23:55:01 ... scylla-jmx[10764]: at org.apache.cassandra.db.ColumnFamilyStore.checkRegistration(ColumnFamilyStore.java:112) Apr 23 23:55:01 ... scylla-jmx[10764]: at com.scylladb.jmx.utils.APIMBeanServer.checkRegistrations(APIMBeanServer.java:280) Apr 23 23:55:01 ... scylla-jmx[10764]: at com.scylladb.jmx.utils.APIMBeanServer.queryNames(APIMBeanServer.java:95) Apr 23 23:55:01 ... scylla-jmx[10764]: at com.scylladb.jmx.main.Main.main(Main.java:49) Apr 23 23:55:01 ... scylla-jmx[10764]: Caused by: java.lang.OutOfMemoryError: Java heap space Apr 23 23:55:01 ... scylla-jmx[10764]: at java.util.Arrays.copyOfRange(Arrays.java:3664) Apr 23 23:55:01 ... scylla-jmx[10764]: at java.lang.String.(String.java:207) Apr 23 23:55:01 ... scylla-jmx[10764]: at java.lang.String.substring(String.java:1969) Apr 23 23:55:01 ... scylla-jmx[10764]: at javax.management.ObjectName.getDomain(ObjectName.java:1566) Apr 23 23:55:01 ... scylla-jmx[10764]: at com.scylladb.jmx.metrics.MetricsMBean.lambda$getTypePredicate$0(MetricsMBean.java:46) Apr 23 23:55:01 ... scylla-jmx[10764]: at com.scylladb.jmx.metrics.MetricsMBean$$Lambda$8/1784662007.test(Unknown Source) Apr 23 23:55:01 ... scylla-jmx[10764]: at com.scylladb.jmx.metrics.APIMBean$1.apply(APIMBean.java:105) Apr 23 23:55:01 ... scylla-jmx[10764]: at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.objectNamesFromFilteredNamedObjects(DefaultMBeanServerInterceptor.ja va:1521) Apr 23 23:55:01 ... scylla-jmx[10764]: at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.queryNamesImpl(DefaultMBeanServerInterceptor.java:564) Apr 23 23:55:01 ... scylla-jmx[10764]: at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.queryNames(DefaultMBeanServerInterceptor.java:554) Apr 23 23:55:01 ... scylla-jmx[10764]: at com.sun.jmx.mbeanserver.JmxMBeanServer.queryNames(JmxMBeanServer.java:619) Apr 23 23:55:01 ... scylla-jmx[10764]: at com.scylladb.jmx.metrics.APIMBean.queryNames(APIMBean.java:97) Apr 23 23:55:01 ... scylla-jmx[10764]: at com.scylladb.jmx.metrics.MetricsMBean.register(MetricsMBean.java:52) Apr 23 23:55:01 ... scylla-jmx[10764]: at com.scylladb.jmx.metrics.MetricsMBean.preRegister(MetricsMBean.java:66) Apr 23 23:55:01 ... scylla-jmx[10764]: at com.sun.jmx.mbeanserver.MBeanSupport.preRegister(MBeanSupport.java:167) Apr 23 23:55:01 ... scylla-jmx[10764]: at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.preRegister(DefaultMBeanServerInterceptor.java:1007) Apr 23 23:55:01 ... scylla-jmx[10764]: ... 9 more Apr 23 23:55:01 ... systemd[1]: scylla-jmx.service: main process exited, code=exited, status=1/FAILURE Apr 23 23:55:01 ... systemd[1]: Unit scylla-jmx.service entered failed state. Apr 23 23:55:01 ... systemd[1]: scylla-jmx.service failed.

slivne commented 6 years ago

1 keyspace

400 tables + 2*400 indexes + 1*400 materialized views

penberg commented 6 years ago

Fixed by memory footprint optimizations in e27312df10d7dce727d1f7b148e90aaaced66359 and/or 71f857b1de2d4cdaacc517402ba39a8275b5e52e?

/cc @haaawk

haaawk commented 6 years ago

400 tables 800 indexes and 400 materialized views should easily fit into 256MB of RAM after the changes you have mentioned but of course you can still cause out of memory error by just making way more tables :).

After those changes jmx server takes 82M of RAM when there are 2000 tables present so I guess we probably can close this issue.

avikivity commented 6 years ago

Fixed for 2.3.

avikivity commented 6 years ago

(by 71f857b1de2d4cdaacc517402ba39a8275b5e52e and a follow-up)