Open asfimport opened 14 years ago
Sanne Grinovero (migrated from JIRA)
I'm going to see if I can contribute a patch myself, but I don't think I'll be able to provide a unit test.
Yonik Seeley (@yonik) (migrated from JIRA)
Background: via irc we brainstormed that the most likely cause of the exception was that listing the files in a directory is probably not atomic (at the JVM level) - hence it's possible to miss the segments file in a rapidly changing index. The simplest fix would seem to be to retry the directory listing a few times if the segments file isn't found.
Michael McCandless (@mikemccand) (migrated from JIRA)
Man it's hard to add a comment here. Had to scroll way to the right... silly Jira.
The best guess here is that this is due to non-atomicity of listing a directory right? Ie, Lucene, in order to find the most recent segments_N file, lists the directory. But if, as the listing is happening, a commit is done from IndexWriter, writing a new segments_N+1 and removing the old one, it's possible that the directory listing would show no segments file.
Lucene then falls back to reading segments.gen, but somehow this is also stale/unusable (probably because another commit kicked off after the dir listing and before we could read segments.gen).
I'm not sure offhand how we can fix this...
Can you describe the stress test you're running?
Sanne Grinovero (migrated from JIRA)
sure, the test is totally open source; the directory implementation based on Infinispan is hosted as submodule of Infinispan: http://anonsvn.jboss.org/repos/infinispan/branches/4.1.x/lucene-directory/
The test is org.infinispan.lucene.profiling.PerformanceCompareStressTest
it is included in the default test suite but disabled in Maven's configuration, so you should run it manually mvn clean test -Dtest=PerformanceCompareStressTest (running it requires the jboss.org repositories to be enabled in maven settings)
To describe it at higher level: there are 5 IndexRead-ing threads using reopen() before each search, 2 threads writing to the index, 1 additional thread as a coordinator and asserting that readers find what they expect to see in the index. Exactly the same test scenario is then applied in sequence to RAMDirectory (not having issues), NIOFSDirectory, and 4 differently configured Infinispan directories. Only the FSDirectory is affected by the issue, and it can never complete the full hour of stresstest succesfully, while all other implementations behave fine.
IndexWriter is set to MaxMergeDocs(5000) and setUseCompoundFile(false); the issue is reveled both using SerialMergeScheduler and while using the default merger.
During the last execution the test managed to perform 22,192,006 searches and 26,875 writes before hitting the exceptional case.
If you deem it useful I'd be happy in contributing a similar testcase to Lucene, but I assume you won't be excited in having such a long running test. Open to ideas to build a simpler one.
Sanne Grinovero (migrated from JIRA)
reformatted the description: all filenames where on the same line making this page hard to use.
Michael McCandless (@mikemccand) (migrated from JIRA)
Thanks for the details Sanne! Your Infinispan directories sounds interesting.
http://fixunix.com/linux/356378-opendir-readdir-atomicity.html is relevant, assuming the JVM is using opendir/readdir on Linux (which I assume it is?).
Basically Posix makes no guarantee that opendir/readir will see a "point in time" directory listing. Ie, file add/deletes can be seen out-of-order, much like write operations in different threads in Java if you don't sync.
So maybe we should add an additional retry cycle in the case that we don't find a segments file? (We already have various retries if we do see a segments file in the listing, but, we hit an IOExc when trying to load it).
Sanne do you want to work out a patch?
Sanne Grinovero (migrated from JIRA)
Hello, sorry for the late answer, for some reason I didn't see the notification.
Sure I'm very interested in providing a patch for this. I've to say I was only able to reproduce this issue in synthetic benchmarks, so I thought it might be very unlikely in real world scenarios, but now I actually received reports of people having issues with this during real use cases, so I'll definitely give another look.
Thanks for the pointers!
Robert Muir (@rmuir) (migrated from JIRA)
moving out... there is no patch
Shai Erera (@shaie) (migrated from JIRA)
There is no patch, moving to 3.2
Robert Muir (@rmuir) (migrated from JIRA)
bulk move 3.2 -> 3.3
Chris M. Hostetter (@hossman) (migrated from JIRA)
Bulk changing fixVersion 3.6 to 4.0 for any open issues that are unassigned and have not been updated since March 19.
Email spam suppressed for this bulk edit; search for hoss20120323nofix36 to identify all issues edited
Steven Rowe (@sarowe) (migrated from JIRA)
Bulk move 4.4 issues to 4.5 and 5.0
Uwe Schindler (@uschindler) (migrated from JIRA)
Move issue to Lucene 4.9.
Ashutosh Deshpande (migrated from JIRA)
I seem to be hitting this issue in my application. The simplified use case is: we have about 1000 new documents being processed simultaneously by multiple threads and multiple threads potentially try to store/read the same document, but with different data. One thread searches, finds nothing and updates the document; and at the same time other thread tries to search for it before update and ends up not finding it, thus creating a duplicate document with different data instead of having a single merged document. This happens very frequently and we end up having over 50% duplicates at times. This causes an issue in search.
Is there any fix available for this issue?
Thanks and Regards, Ashutosh.
Han-Wen NIenhuys (migrated from JIRA)
I've seen race condition diagnostics for Lucene code that could be related to this.
Line numbers are for Lucene 5.5.2, but the problem seems present in master too.
AFAICT: StandardDirectoryReader has an IndexWriter, and it checks things are up-to-date by checking if there are pending in-memory updates by looking at IndexWriter.SegmentInfos.version. That read is not synchronized relative to its write from IndexWriter.updateDocument
WARNING: ThreadSanitizer: data race (pid=21636)
Write of size 8 at 0x7f24ad9d3210 by thread T36 (mutexes: write M431922409782456832):
#0
org.apache.lucene.index.SegmentInfos.changed()V (SegmentInfos.java:944)
#1
org.apache.lucene.index.IndexWriter.newSegmentName()Ljava/lang/String; (IndexWriter.java:1652)
#2
org.apache.lucene.index.DocumentsWriter.ensureInitialized(Lorg/apache/lucene/index/DocumentsWriterPerThreadPool$ThreadState;)V (DocumentsWriter.java:391)
#3
org.apache.lucene.index.DocumentsWriter.updateDocument(Ljava/lang/Iterable;Lorg/apache/lucene/analysis/Analyzer;Lorg/apache/lucene/index/Term;)Z (DocumentsWriter.java:445)
#4
org.apache.lucene.index.IndexWriter.updateDocument(Lorg/apache/lucene/index/Term;Ljava/lang/Iterable;)V (IndexWriter.java:1477)
#5
com.google.gerrit.lucene.AutoCommitWriter.updateDocument(Lorg/apache/lucene/index/Term;Ljava/lang/Iterable;)V (AutoCommitWriter.java:100)
#6
org.apache.lucene.index.TrackingIndexWriter.updateDocument(Lorg/apache/lucene/index/Term;Ljava/lang/Iterable;)J (TrackingIndexWriter.java:55)
#7
com.google.gerrit.lucene.AbstractLuceneIndex$4.call()Ljava/lang/Long; (AbstractLuceneIndex.java:250)
#8
com.google.gerrit.lucene.AbstractLuceneIndex$4.call()Ljava/lang/Object; (AbstractLuceneIndex.java:247)
#9
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly()V (TrustedListenableFutureTask.java:108)
#10
com.google.common.util.concurrent.InterruptibleTask.run()V (InterruptibleTask.java:41)
#11
com.google.common.util.concurrent.TrustedListenableFutureTask.run()V (TrustedListenableFutureTask.java:77)
#12
java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (ThreadPoolExecutor.java:1142)
#13
java.util.concurrent.ThreadPoolExecutor$Worker.run()V (ThreadPoolExecutor.java:617)
#14
java.lang.Thread.run()V (Thread.java:745)
#15
(Generated Stub)
Previous read of size 8 at 0x7f24ad9d3210 by thread T29 (mutexes: write M1060737507754061632):
#0
org.apache.lucene.index.IndexWriter.nrtIsCurrent(Lorg/apache/lucene/index/SegmentInfos;)Z (IndexWriter.java:4592)
#1
org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(Lorg/apache/lucene/index/IndexCommit;)Lorg/apache/lucene/index/DirectoryReader; (StandardDirectoryReader.java:282)
#2
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(Lorg/apache/lucene/index/IndexCommit;)Lorg/apache/lucene/index/DirectoryReader; (StandardDirectoryReader.java:261)
#3
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged()Lorg/apache/lucene/index/DirectoryReader; (StandardDirectoryReader.java:251)
#4
org.apache.lucene.index.DirectoryReader.openIfChanged(Lorg/apache/lucene/index/DirectoryReader;)Lorg/apache/lucene/index/DirectoryReader; (DirectoryReader.java:137)
#5
com.google.gerrit.lucene.WrappableSearcherManager.refreshIfNeeded(Lorg/apache/lucene/search/IndexSearcher;)Lorg/apache/lucene/search/IndexSearcher; (WrappableSearcherManager.java:148)
#6
com.google.gerrit.lucene.WrappableSearcherManager.refreshIfNeeded(Ljava/lang/Object;)Ljava/lang/Object; (WrappableSearcherManager.java:68)
#7
org.apache.lucene.search.ReferenceManager.doMaybeRefresh()V (ReferenceManager.java:176)
#8
org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking()V (ReferenceManager.java:253)
#9
org.apache.lucene.search.ControlledRealTimeReopenThread.run()V (ControlledRealTimeReopenThread.java:245)
#10
(Generated Stub)
Michael McCandless (@mikemccand) (migrated from JIRA)
Thanks Han-Wen NIenhuys, I think you are correct. How would you suggest we fix it? Could you open a new issue to track this? Thanks.
Han-Wen NIenhuys (migrated from JIRA)
opened #8550 - I know too little of lucene to give a good answer for fixing this, but I gave a suggestion anyway.
I could reproduce the issue several times but only by running long and stressfull benchmarks, the high number of files is likely part of the scenario. All tests run on local disk, using ext3.
Sample stacktrace:
Migrated from LUCENE-2585 by Sanne Grinovero, updated Oct 17 2016 Linked issues:
6987