search

oracle / opengrok

OpenGrok is a fast and usable source code search and cross reference engine, written in Java
http://oracle.github.io/opengrok/
Other
4.29k stars 739 forks source link

limit addFile() processing time #4550

Open vladak opened 4 months ago

vladak commented 4 months ago

When indexing bunch of projects, the indexer (based on 1.13.4) had a trailing worker thread that was spinning on the CPU for inordinate amount of time (minutes) while all the other files were finished. The stack trace looked like this (comes from my branch when experimenting with the fix for #4549 so thread name and line numbers do not match):

"OpenGrok-index-worker-48" #48 prio=5 os_prio=0 cpu=773733,68ms elapsed=1140,14s tid=0x00007f3648023000 nid=0x2134a7 runnable  [0x00007f364e5fd000]
   java.lang.Thread.State: RUNNABLE
    at org.opengrok.indexer.analysis.javascript.JavaScriptSymbolTokenizer.yylex(JavaScriptSymbolTokenizer.java:1645)
    at org.opengrok.indexer.analysis.JFlexTokenizer.incrementToken(JFlexTokenizer.java:106)
    at org.apache.lucene.index.IndexingChain$PerField.invertTokenStream(IndexingChain.java:1167)
    at org.apache.lucene.index.IndexingChain$PerField.invert(IndexingChain.java:1145)
    at org.apache.lucene.index.IndexingChain.processField(IndexingChain.java:697)
    at org.apache.lucene.index.IndexingChain.processDocument(IndexingChain.java:576)
    at org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:243)
    at org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:421)
    at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1551)
    at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1836)
    at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1476)
    at org.opengrok.indexer.index.IndexDatabase.addFile(IndexDatabase.java:1278)
    at org.opengrok.indexer.index.IndexDatabase.lambda$indexParallel$9(IndexDatabase.java:1920)
    at org.opengrok.indexer.index.IndexDatabase$$Lambda$370/0x00000008401e7c40.call(Unknown Source)

Specifically, the file in question came from the https://hg.mozilla.org/mozilla-central , it was testing/modules/sinon-7.2.7.js which contains long lists of numbers. Such files are known to cause problems. Ideally the analyzers should skip/truncate the files with huge number of terms.

The indexer eventually finished, however there should be some time limit for addFile() or writer.addDocument() processing. There is already timeout enforced for xref production via the XrefWork class.

vladak commented 4 months ago

Rather than trying to terminate and cleanup the writer.addDocument() call it would be easier to check the term count in certain fields and skip the document altogether if the count is above given threshold.

Another idea is to trim the number if terms.

saurav-max commented 1 month ago

Review the code of the JavaScriptSymbolTokenizer class to understand what it's doing and where the error might be occurring. Look for any potential bugs, errors, or exceptions that could be causing the issue.