search

spectra-cluster / spectra-cluster-cli

Prototype of a CLI to run the clustering process
1 stars 3 forks source link

OutOfMemoryError: GC overhead limit exceeded #10

Closed hahahaL closed 2 years ago

hahahaL commented 2 years ago

the env like these:

the command like these: export JAVA_OPTS="-Xmx10g -XX:-UseGCOverheadLimit" java -jar spectra-cluster-cli-1.1.2.jar -major_peak_jobs 10 -x_min_comparisons 8000 -fast_mode -filter mz_200 -precursor_tolerance 2 -fragment_tolerance 0.02 -output_path cluster.cdf a1.mfg a2.mfg a3.mfg ...a70.mfg

the errror message: Converting 70 input files... Clustering 367 binary files... java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: GC overhead limit exceeded at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at uk.ac.ebi.pride.spectracluster.clustering.BinaryFileClusterer.waitForCompletedJobs(BinaryFileClusterer.java:80) at uk.ac.ebi.pride.spectracluster.clustering.BinaryFileClusterer.clusterFiles(BinaryFileClusterer.java:51) at uk.ac.ebi.pride.spectracluster.implementation.SpectraClusterStandalone.clusterFiles(SpectraClusterStandalone.java:516) at uk.ac.ebi.pride.spectracluster.implementation.SpectraClusterStandalone.clusterBinaryFiles(SpectraClusterStandalone.java:119) at uk.ac.ebi.pride.spectracluster.implementation.SpectraClusterStandalone.clusterPeaklistFiles(SpectraClusterStandalone.java:103) at uk.ac.ebi.pride.spectracluster.cli.SpectraClusterCliMain.run(SpectraClusterCliMain.java:344) at uk.ac.ebi.pride.spectracluster.cli.SpectraClusterCliMain.main(SpectraClusterCliMain.java:43) Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded at java.util.stream.StreamSupport.stream(StreamSupport.java:69) at java.util.Collection.stream(Collection.java:581) at uk.ac.ebi.pride.spectracluster.spectrum.Spectrum.asMajorPeakMZs(Spectrum.java:211) at uk.ac.ebi.pride.spectracluster.util.predicate.spectrum_comparison.ShareMajorPeaksPredicate.apply(ShareMajorPeaksPredicate.java:23) at uk.ac.ebi.pride.spectracluster.util.predicate.spectrum_comparison.ShareMajorPeaksPredicate.apply(ShareMajorPeaksPredicate.java:9) at uk.ac.ebi.pride.spectracluster.util.predicate.cluster_comparison.ClusterShareMajorPeakPredicate.apply(ClusterShareMajorPeakPredicate.java:27) at uk.ac.ebi.pride.spectracluster.util.predicate.cluster_comparison.ClusterShareMajorPeakPredicate.apply(ClusterShareMajorPeakPredicate.java:14) at uk.ac.ebi.pride.spectracluster.engine.GreedyIncrementalClusteringEngine.addToClusters(GreedyIncrementalClusteringEngine.java:213) at uk.ac.ebi.pride.spectracluster.engine.GreedyIncrementalClusteringEngine.addClusterIncremental(GreedyIncrementalClusteringEngine.java:148) at uk.ac.ebi.pride.spectracluster.engine.GreedyIncrementalClusteringEngine.addClusterIncremental(GreedyIncrementalClusteringEngine.java:31) at uk.ac.ebi.pride.spectracluster.clustering.BinaryFileClusteringCallable.call(BinaryFileClusteringCallable.java:159) at uk.ac.ebi.pride.spectracluster.clustering.BinaryFileClusteringCallable.call(BinaryFileClusteringCallable.java:33) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Error: java.lang.OutOfMemoryError: GC overhead limit exceeded

jgriss commented 2 years ago

Hi @hahahaL ,

Thanks a lot for your feedback!

When running Java applications, the maximum memory is set through the -Xmx10g parameter.

To allow a maximum of 100 GB of memory, you'd need to adapt the parameter to -Xmx100g.

Kind regards, Johannes

hahahaL commented 2 years ago

Thanks for your replay ,if not the code cause the mem leak，I will try to increase the jvm to run the workload