search

scylladb / scylla-jmx

Scylla JMX proxy
GNU Affero General Public License v3.0
29 stars 54 forks source link

scylla-jmx OOM when running 5k tables backup tasks #88

Open fgelcer opened 4 years ago

fgelcer commented 4 years ago

it also failed to dump when crashing due to "Permission denied" error:

Dec 25 09:48:05 ip-10-0-244-119.eu-west-1.compute.internal scylla-jmx[3305]: Starting the JMX server
Dec 25 09:49:32 ip-10-0-244-119.eu-west-1.compute.internal scylla-jmx[3305]: java.lang.OutOfMemoryError: Java heap space
Dec 25 09:49:32 ip-10-0-244-119.eu-west-1.compute.internal scylla-jmx[3305]: Dumping heap to java_pid3305.hprof ...
Dec 25 09:49:32 ip-10-0-244-119.eu-west-1.compute.internal scylla-jmx[3305]: Unable to create java_pid3305.hprof: Permission denied

logs will be uploaded and posted in next comment.

fgelcer commented 4 years ago

logs: https://drive.google.com/file/d/1iw_do7gcc08uLY4ukYSwojiJtjCwp_af/view?usp=sharing https://drive.google.com/file/d/11GbJaTydOK8KDCdZ2s0v3y-b4xQQ4Jo1/view?usp=sharing

slivne commented 4 years ago

you need to change scylla-jmx memory .... /usr/lib/scylla/jmx/scylla-jmx change memory to be -Xmx1024m

fgelcer commented 4 years ago

you need to change scylla-jmx memory .... /usr/lib/scylla/jmx/scylla-jmx change memory to be -Xmx1024m

yes, this did the trick... jmx is now up and running.

tzach commented 4 years ago

you need to change scylla-jmx memory .... /usr/lib/scylla/jmx/scylla-jmx change memory to be -Xmx1024m

@slivne why not making this the default? (at least for anything bigger than t2)

avikivity commented 4 years ago

Many machines don't have a gigabyte to spare.

Looks like we're consuming 100-200kB per table, that's crazy.

tzach commented 4 years ago

Many machines don't have a gigabyte to spare.

Sure, this is why it can be a per machine (or percent) parameter. It does not make sense to let users fail at this time after time.