tools4j / stacked-off

Offline StackExchange indexer and search engine
MIT License
77 stars 8 forks source link

GC out of memory error #6

Open sprajagopal opened 4 years ago

sprajagopal commented 4 years ago

Hi, Thanks for this wonderful project! I successfully used it for smaller sites such as meta.chemistry before trying to add Stackoverflow. I get this error at the following error in the linking phase (in localhost/sites I see linking staging_indices against stackoverflow). I was hoping if I added the site again, it would pick up from the error but turns out it restarts the whole process again. Is there any way to continue after this error?

11:41:52.037 [Thread-9] DEBUG org.tools4j.stacked.index.SeZipFileParser - Parsing /media/p/Seagate Backup Plus Drive/stackexchange/dumps/stackoverflow.com-Votes.7z
Exception in thread "Thread-9" java.lang.OutOfMemoryError: GC overhead limit exceeded
        at org.tools4j.stacked.index.UnscoredCollector.search(DocCollector.kt:51)
        at org.tools4j.stacked.index.DocIdIndex.searchByQueryForDocs(DocIdIndex.kt:121)
        at org.tools4j.stacked.index.DocIdIndex.searchByQuery(DocIdIndex.kt:117)
        at org.tools4j.stacked.index.DocIdIndex.searchByTerm(DocIdIndex.kt:80)
        at org.tools4j.stacked.index.DocIdIndex.searchByTerm(DocIdIndex.kt:84)
        at org.tools4j.stacked.index.QuestionIndexer$index$1.invokeSuspend(QuestionIndexer.kt:21)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:238)
        at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.kt:116)
        at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:80)
        at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:54)
        at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
        at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:36)
        at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
        at org.tools4j.stacked.index.QuestionIndexer.index(QuestionIndexer.kt:18)
        at org.tools4j.stacked.index.SeDirParserListener.onFinishParseSite(SeDirParserListener.kt:31)
        at org.tools4j.stacked.index.SeDirParser.parseSeSite(SeDirParser.kt:60)
        at org.tools4j.stacked.index.SeDirParser.parse(SeDirParser.kt:35)
        at org.tools4j.stacked.web.Server$Companion$mainServer$5$9$1.run(Server.kt:182)
        at java.lang.Thread.run(Thread.java:748)
11:45:43.180 [nettyCallPool-4-2] ERROR org.tools4j.stacked.index.Utils - GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.lang.Double.valueOf(Double.java:519)
        at io.ktor.features.Compression$comparator$2.invoke(Compression.kt:54)
        at kotlin.comparisons.ComparisonsKt__ComparisonsKt.compareValuesByImpl$ComparisonsKt__ComparisonsKt(Comparisons.kt:25)
        at kotlin.comparisons.ComparisonsKt__ComparisonsKt.access$compareValuesByImpl(Comparisons.kt:1)
        at kotlin.comparisons.ComparisonsKt__ComparisonsKt$compareBy$1.compare(Comparisons.kt:91)
        at java.util.Collections$ReverseComparator2.compare(Collections.java:5232)
        at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355)
        at java.util.TimSort.sort(TimSort.java:220)
        at java.util.Arrays.sort(Arrays.java:1438)
        at kotlin.collections.ArraysKt___ArraysJvmKt.sortWith(_ArraysJvm.kt:1862)
        at kotlin.collections.CollectionsKt___CollectionsKt.sortedWith(_Collections.kt:947)
        at io.ktor.features.Compression.interceptor(Compression.kt:75)
        at io.ktor.features.Compression$Feature$install$1.invokeSuspend(Compression.kt:174)
        at io.ktor.features.Compression$Feature$install$1.invoke(Compression.kt)
        at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:278)
        at io.ktor.util.pipeline.SuspendFunctionGun.access$loop(PipelineContext.kt:63)
        at io.ktor.util.pipeline.SuspendFunctionGun.proceed(PipelineContext.kt:137)
        at io.ktor.util.pipeline.SuspendFunctionGun.proceedWith(PipelineContext.kt:147)
        at io.ktor.features.ContentNegotiation$Feature$install$2.invokeSuspend(ContentNegotiation.kt:90)
        at io.ktor.features.ContentNegotiation$Feature$install$2.invoke(ContentNegotiation.kt)
        at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:278)
        at io.ktor.util.pipeline.SuspendFunctionGun.access$loop(PipelineContext.kt:63)
        at io.ktor.util.pipeline.SuspendFunctionGun.proceed(PipelineContext.kt:137)
        at io.ktor.util.pipeline.SuspendFunctionGun.execute(PipelineContext.kt:157)
        at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:23)
        at org.tools4j.stacked.web.Server$Companion$mainServer$5$10.invokeSuspend(Server.kt:238)
        at org.tools4j.stacked.web.Server$Companion$mainServer$5$10.invoke(Server.kt)
        at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:278)
        at io.ktor.util.pipeline.SuspendFunctionGun.access$loop(PipelineContext.kt:63)
        at io.ktor.util.pipeline.SuspendFunctionGun.proceed(PipelineContext.kt:137)
        at io.ktor.util.pipeline.SuspendFunctionGun.execute(PipelineContext.kt:157)
        at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:23)