Use NIO positional read to avoid synchronization in FSIndexInput [LUCENE-753]

[asfimport]

As suggested by Doug, we could use NIO pread to avoid synchronization on the underlying file. This could mitigate any MT performance drop caused by reducing the number of files in the index format.

---

Migrated from LUCENE-753 by Yonik Seeley (@yonik), 5 votes, resolved Jan 25 2011 Attachments: FileReadTest.java (versions: 8), FSDirectoryPool.patch, FSIndexInput.patch (versions: 2), lucene-753.patch (versions: 2), LUCENE-753.patch (versions: 5)

[asfimport]

Michael McCandless (@mikemccand) (migrated from JIRA)

And for term expanding queries, you can get a lot of them all at once.

Right but that'd all be under one thread right? The pool would always give the same RandomAccessFile (private or shared) for the same filename X thread.

one would have to reserve the last fd for synchronized access... can't really hand it out for unsynchronized exclusive access and then go and share it later.

Well, I think you'd hand it out first, as a shared file (so you reserve the right to hand it out again, later). If other threads come along you would open a new one (if you are under the budget) and loan it to them privately (so no syncing during read). I think sync'ing with no contention (the first shared file we hand out) should be OK performance in modern JVMs.

the shared access should use pread... not seek+read

But not on Windows...

At first blush, sounds a bit too complex for the benefits.

Yeah I'm on the fence too ... but this lack of concurrency hurts our search performance. It's ashame users have to resort to multiple IndexReaders. Though it still remains to be seen how much the pool or pread approaches really improve end to end search performance (vs other bottlenecks like IndexReader.isDeleted).

Windows is an important platform and doing the pool approach, vs leaving Windows with classic if we do pread approach, lets us have good concurrency on Windows too.

[asfimport]

Brian Gardner (migrated from JIRA)

This probably doesn't help much, but I implemented a pool and submitted a patch very similar to the SeparateFile approach. Before being directed to this thread:

2414

In our implementation the synchronization/lack of concurrency has been a big issue for us. On several occasions we've had to remove new features that perform searches from frequently hit pages, because threads build up waiting for synchronized access to the underlying files. It is possible that I would still have issue even with my patch, considering from my tests that I'm only increasing throughput by 300%, but it would be easier for me to tune and scale my application since resource utilization and contention would be visible from the OS level.

> At first blush, sounds a bit too complex for the benefits.

My vote is that the benefits outway the complexity, especially considering it's an out-of-the box solutions that works well for all platforms and single threaded as well as multi-threaded envirnments. If it's helpful, I can spend the time to implement some of the missing feature(s) of the pool that will be needed for it to be an acceptable solution (i.e, shared access once a file has been deleted, and perhaps a time-based closing mechanism).

[asfimport]

Michael McCandless (@mikemccand) (migrated from JIRA)

In our implementation the synchronization/lack of concurrency has been a big issue for us. On several occasions we've had to remove new features that perform searches from frequently hit pages, because threads build up waiting for synchronized access to the underlying files. It is possible that I would still have issue even with my patch, considering from my tests that I'm only increasing throughput by 300%, but it would be easier for me to tune and scale my application since resource utilization and contention would be visible from the OS level.

Can you describe your test – OS, JRE version, size/type of your index, number of cores, amount of RAM, type of IO system, etc? It's awesome that you see 300% gain in search throughput. Is your index largely cached in the OS's IO cache, or not?

My vote is that the benefits outway the complexity, especially considering it's an out-of-the box solutions that works well for all platforms and single threaded as well as multi-threaded envirnments. If it's helpful, I can spend the time to implement some of the missing feature(s) of the pool that will be needed for it to be an acceptable solution (i.e, shared access once a file has been deleted, and perhaps a time-based closing mechanism).

If we can see sizable concurreny gains, reliably & across platforms, I agree we should pursue this approach. One particular frustration is: if you optimize your index, thinking this gains you better search performance, you're actually making things far worse as far as concurrency is concerned because now you are down to a single immense file. I think we do need to fix this situation.

On your patch, I think in addition to shared-access on a now-deleted file, we should add a global control on the "budget" of number of open files (right now I think your patch has a fixed cap per-filename). Probably the budget should be expressed as a multiplier off the minimum number of open files, rather than a fixed cap, so that an index with many segments is allowed to use more. Ideally over time the pool works out such that for small files in the index (small segments) since there is very little contention they only hold 1 descriptor open, but for large files many descriptors are opened.

I created a separate test (will post a patch & details to this issue) to explore using SeparateFile inside FSDirectory, but unfortunately I see mixed results on both the cached & uncached cases. I'll post details separately.

One issue with your patch is it's using Java 5 only classes (Lucene is still on 1.4); once you downgrade to 1.4 I wonder if the added synchronization will become costly.

I like how your approach is to pull a RandomAccessFile from the pool only when a read is taking place – this automatically takes care of creating new descriptors when there truly is contention. But one concern I have is that this defeats the OS's IO system's read-ahead optimization since from the OS's perspective the file descriptors are getting shuffled. I'm not sure if this really matters much in Lucene, because many things (reading stored fields & term vectors) are likely not helped much by read-ahead, but for example a simple TermQuery on a large term should in theory benefit from read-ahead. You could gain this back with a simple thread affinity, such that the same thread gets the same file descriptor it got last time, if it's available. But that added complexity may offset any gains.

[asfimport]

Michael McCandless (@mikemccand) (migrated from JIRA)

I attached FSDirectoryPool.patch, which adds oal.store.FSDirectoryPool, a Directory that will open a new file for every unique thread.

This is intended only as a test (to see if shows consistent improvement in concurrency) – eg it does not close all these files, nor make any effort to budget itself if there are too many threads, it's not really a pool, etc. But it should give us an upper bound on the gains we could hope for.

I also added a "pool=true|false" config option to contrib/benchmark so you can run tests with and without separate files.

I ran some quick initial tests but didn't see obvious gains. I'll go back & re-run more carefully to confirm, and post back.

[asfimport]

Michael McCandless (@mikemccand) (migrated from JIRA)

I created a large index (indexed Wikipedia 4X times over, with stored fields & tv offsets/positions = 72 GB). I then randomly sampled 50 terms > 1 million freq, plus 200 terms > 100,000 freq plus 100 terms > 10,000 freq plus 100 terms > 1000 freq. Then I warmed the OS so these queries are fully cached in the IO cache.

It's a highly synthetic test. I'd really love to test on real queries, instead of single term queries.

Then I ran this alg:

analyzer=org.apache.lucene.analysis.standard.StandardAnalyzer

query.maker = org.apache.lucene.benchmark.byTask.feeds.FileBasedQueryMaker
file.query.maker.file = /lucene/wikiQueries.txt

directory=FSDirectory
pool=true

work.dir=/lucene/bigwork

OpenReader

{ "Warmup" SearchTrav(20) > : 5

{ "Rounds"
  [{ "Search" Search > : 500]: 16
  NewRound
}: 2

CloseReader 

RepSumByPrefRound Search

I ran with 2, 4, 8 and 16 threads, on a Intel quad Mac Pro (2 cpus, each dual core) OS X 10.5.4, with 6 GB RAM, Sun JRE 1.6.0_05 and a single WD Velociraptor hard drive. To keep the number of searches constant I changed the 500 count above to match (ie with 8 threads I changed 500 -> 1000, 4 threads I changed it to 2000, etc.).

Here're the results – each run is best of 2, and all searches are fully cached in OS's IO cache:

Number of Threads	Patch rec/s	Trunk rec/s	Pctg gain
2	78.7	74.9	5.1%
4	74.1	68.2	8.7%
8	37.7	32.7	15.3%
16	19.2	16.3	17.8%

I also ran the same alg, replacing Search task with SearchTravRet(10) (retrieves the first 10 docs (hits) of each search), first warming so it's all fully cached:

Number of Threads	Patch rec/s	Trunk rec/s	Pctg gain
2	1589.6	1519.8	4.6%
4	1460.9	1395.3	4.7%
8	748.9	676.0	10.8%
16	382.7	338.4	13.1%

So there are smallish gains, but rememember these are upper bounds on the gains because no pooling is happening. I'll test uncached next.

[asfimport]

Michael McCandless (@mikemccand) (migrated from JIRA)

OK I ran the uncached test, using the Search task. JRE & hardware are the same as above.

I generated a larger (6150) set of queries to make sure the threads never wrap around and do the same queries again. I also run only 1 round for the same reason. Between tests I evict the OS's IO cache.

Number of Threads	Patch rec/s	Trunk rec/s	Pctg gain
2	32.2	23.8	35.3%
4	16.4	12.7	29.1%
8	8.5	3.5	142.9%
16	3.8	2.7	40.7%

The gains are better. The 8 thread case I don't get; I re-ran it and it still came out much better (135.7%). It could be 8 threads is the sweet spot for concurrency on this hardware.

[asfimport]

Matthew Mastracci (migrated from JIRA)

I just tried out the latest NIOFSDirectory patch and I'm seeing a bug. If I go back to the regular FSDirectory, everything works fine.

I can't reproduce it on a smaller testcase. It only happens with the live index.

Any ideas on where to debug?

Caused by: java.lang.IndexOutOfBoundsException: Index: 24444, Size: 4
    at java.util.ArrayList.RangeCheck(ArrayList.java:547)
    at java.util.ArrayList.get(ArrayList.java:322)
    at org.apache.lucene.index.FieldInfos.fieldInfo(FieldInfos.java:260)
    at org.apache.lucene.index.FieldInfos.fieldName(FieldInfos.java:249)
    at org.apache.lucene.index.TermBuffer.read(TermBuffer.java:68)
    at org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java:123)
    at org.apache.lucene.index.SegmentTermEnum.scanTo(SegmentTermEnum.java:154)
    at org.apache.lucene.index.TermInfosReader.scanEnum(TermInfosReader.java:223)
    at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:217)
    at org.apache.lucene.index.SegmentReader.docFreq(SegmentReader.java:678)
    at org.apache.lucene.index.MultiSegmentReader.docFreq(MultiSegmentReader.java:373)
    at org.apache.lucene.search.IndexSearcher.docFreq(IndexSearcher.java:87)
    at org.apache.lucene.search.Similarity.idf(Similarity.java:457)
    at org.apache.lucene.search.TermQuery$TermWeight.<init>(TermQuery.java:44)
    at org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:146)
    at org.apache.lucene.search.BooleanQuery$BooleanWeight.<init>(BooleanQuery.java:187)
    at org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:362)
    at org.apache.lucene.search.Query.weight(Query.java:95)
    at org.apache.lucene.search.Searcher.createWeight(Searcher.java:171)
    at org.apache.lucene.search.Searcher.search(Searcher.java:132)

The index is not using the compound file format:

7731499698 Jul 28 03:46 _6zk.fdt
 232014520 Jul 28 03:50 _6zk.fdx
        32 Jul 28 03:50 _6zk.fnm
3775713450 Jul 28 04:06 _6zk.frq
  58003634 Jul 28 04:07 _6zk.nrm
2944298834 Jul 28 04:18 _6zk.prx
    432418 Jul 28 04:18 _6zk.tii
  30784106 Jul 28 04:19 _6zk.tis
 217354711 Jul 28 08:18 _76i.fdt
   6509864 Jul 28 08:18 _76i.fdx
        32 Jul 28 08:18 _76i.fnm
 144348761 Jul 28 08:18 _76i.frq
   1627470 Jul 28 08:18 _76i.nrm
 295528445 Jul 28 08:19 _76i.prx
     52622 Jul 28 08:19 _76i.tii
   3858378 Jul 28 08:19 _76i.tis
 199621206 Jul 29 13:29 _7cm.fdt
   5994720 Jul 29 13:29 _7cm.fdx
        32 Jul 29 13:29 _7cm.fnm
 136445620 Jul 29 13:29 _7cm.frq
   1498684 Jul 29 13:29 _7cm.nrm
 284805312 Jul 29 13:30 _7cm.prx
     48346 Jul 29 13:30 _7cm.tii
   3522117 Jul 29 13:30 _7cm.tis
   3914068 Jul 29 13:30 _7cn.fdt
    119184 Jul 29 13:30 _7cn.fdx
        32 Jul 29 13:30 _7cn.fnm
   2993343 Jul 29 13:30 _7cn.frq
     29800 Jul 29 13:30 _7cn.nrm
   7380878 Jul 29 13:30 _7cn.prx
      5277 Jul 29 13:30 _7cn.tii
    378816 Jul 29 13:30 _7cn.tis
    383147 Jul 29 13:30 _7cq.fdt
     11240 Jul 29 13:30 _7cq.fdx
        32 Jul 29 13:30 _7cq.fnm
    290398 Jul 29 13:30 _7cq.frq
      2814 Jul 29 13:30 _7cq.nrm
    763135 Jul 29 13:30 _7cq.prx
      1581 Jul 29 13:30 _7cq.tii
    115971 Jul 29 13:30 _7cq.tis
        19 Jul 29 13:30 date
        20 Jul 21 01:53 segments.gen
       155 Jul 29 13:30 segments_d61

[asfimport]

Michael McCandless (@mikemccand) (migrated from JIRA)

I just tried out the latest NIOFSDirectory patch and I'm seeing a bug. If I go back to the regular FSDirectory, everything works fine.

Is the index itself corrupt, ie, NIOFSDirectory did something bad when writing the index? Or, is it only in reading the index with NIOFSDirectory that you see this? IE, can you swap in FSDirectory on your existing index and the problem goes away?

[asfimport]

Matthew Mastracci (migrated from JIRA)

Is the index itself corrupt, ie, NIOFSDirectory did something bad when writing the index? Or, is it only in reading the index with NIOFSDirectory that you see this? IE, can you swap in FSDirectory on your existing index and the problem goes away?

I haven't seen any issues with writing the index under NIOFSDirectory. The failures seem to happen only when reading. When I switch to FSDirectory (or MMapDirectory), the same index that fails under NIOFSDirectory works flawlessly (indicating that the index is not corrupt).

The error with NIOFSDirectory is determinate and repeatable (same error every time, same location, same query during warmup).

I couldn't reproduce this on a smaller index, unfortunately.

[asfimport]

Michael McCandless (@mikemccand) (migrated from JIRA)

The error with NIOFSDirectory is determinate and repeatable (same error every time, same location, same query during warmup).

Did you see a prior exception, before hitting the AIOOBE? If so, I think this is just #2339 all over again. That issue was fixed in BufferedIndexInput, but the NIOFSIndexInput has copied a bunch of code from BufferedIndexInput (something I think we must fix before committing it – I think it should inherit from BufferedIndexInput instead) and so it still has that bug. I'll post a patch with the bug re-fixed so you can at least test it to see if it resolves your exception.

[asfimport]

Jason Rutherglen (migrated from JIRA)

I can possibly work on this, just go through and reedit the BufferedIndexInput portions of the code. Inheriting is difficult because of the ByteBuffer code. Needs to be done line by line.

[asfimport]

Michael McCandless (@mikemccand) (migrated from JIRA)

Attached new rev of NIOFSDirectory.

Besides re-fixing #2339, I also found & fixed a bug in the NIOFSIndexInput.clone() method.

Matthew, could you give this one a shot to see if it fixes your case? Thanks.

[asfimport]

Michael McCandless (@mikemccand) (migrated from JIRA)

I can possibly work on this, just go through and reedit the BufferedIndexInput portions of the code. Inheriting is difficult because of the ByteBuffer code. Needs to be done line by line.

That would be awesome, Jason. I think we should then commit NIOFSDirectory to core as at least a way around this bottleneck on all platforms but Windows. Maybe we can do this in time for 2.4?

[asfimport]

Matthew Mastracci (migrated from JIRA)

Matthew, could you give this one a shot to see if it fixes your case? Thanks.

Michael,

I ran this new patch against our big index and it works very well. If I have time, I'll run some benchmarks to see what our real-life performance improvements are like.

Note that I'm only running it for our read-only snapshot of the index, however, so this hasn't been tested for writing to a large index.

[asfimport]

Michael McCandless (@mikemccand) (migrated from JIRA)

I ran this new patch against our big index and it works very well. If I have time, I'll run some benchmarks to see what our real-life performance improvements are like.

Super, thanks! This was the same index that would reliably hit the exception above?

[asfimport]

Matthew Mastracci (migrated from JIRA)

Super, thanks! This was the same index that would reliably hit the exception above?

Correct - it would hit the exception every time at startup.

I've been running NIOFSDirectory for the last couple of hours with zero exceptions (except for running out of file descriptors after starting it the first time :)). The previous incarnation was running MMapDirectory.

Thanks for all the work on this patch.

[asfimport]

Matthew Mastracci (migrated from JIRA)

This exception popped up out of the blue a few hours in. No exceptions before it. I'll see if I can figure out whether it was caused by our index snapshotting or if it's a bug elsewhere in NIOFSDirectory.

I haven't seen any exceptions like this with MMapDirectory, but it's possible there's something that we're doing that isn't correct.

Caused by: java.nio.channels.ClosedChannelException
    at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:91)
    at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:616)
    at com.dotspots.analyzer.index.NIOFSDirectory$NIOFSIndexInput.read(NIOFSDirectory.java:186)
    at com.dotspots.analyzer.index.NIOFSDirectory$NIOFSIndexInput.refill(NIOFSDirectory.java:218)
    at com.dotspots.analyzer.index.NIOFSDirectory$NIOFSIndexInput.readByte(NIOFSDirectory.java:232)
    at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:76)
    at org.apache.lucene.index.TermBuffer.read(TermBuffer.java:63)
    at org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java:123)
    at org.apache.lucene.index.SegmentTermEnum.scanTo(SegmentTermEnum.java:154)
    at org.apache.lucene.index.TermInfosReader.scanEnum(TermInfosReader.java:223)
    at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:217)
    at org.apache.lucene.index.SegmentReader.docFreq(SegmentReader.java:678)
    at org.apache.lucene.index.MultiSegmentReader.docFreq(MultiSegmentReader.java:373)
    at org.apache.lucene.index.MultiReader.docFreq(MultiReader.java:310)
    at org.apache.lucene.search.IndexSearcher.docFreq(IndexSearcher.java:87)
    at org.apache.lucene.search.Searcher.docFreqs(Searcher.java:178)

[asfimport]

Michael McCandless (@mikemccand) (migrated from JIRA)

Interesting...

Are you really sure you're not accidentally closing the searcher before calling Searcher.docFreqs? Are you calling docFreqs directly from your app?

It looks like MMapIndexInput.close() is a noop so it would not have detected calling Searcher.docFreqs after close, whereas NIOFSdirectory (and the normal FSDirectory) will.

If you try the normal FSDirectory do you also an exception like this?

Incidentally, what sort of performance differences are you noticing between these three different ways of accessing an index in the file system?

[asfimport]

Yonik Seeley (@yonik) (migrated from JIRA)

Maybe we can do this in time for 2.4?

+1

Latest patch is looking good to me! Is there a reason we don't do lazy allocation in clone() like FSIndexInput?

Also, our finalizers aren't technically thread safe which could lead to a double close in the finalizer (although I doubt if this particular case would ever happen). If we need to keep them, we could change Descriptor.isOpen to volatile and there should be pretty much no cost since it's only checked in close().

[asfimport]

Michael McCandless (@mikemccand) (migrated from JIRA)

Is there a reason we don't do lazy allocation in clone() like FSIndexInput?

Yonik, do you mean BufferedIndexInput.clone (not FSIndexInput)?

I think once we fix NIOFSIndexInput to subclass from BufferedIndexInput, then cloning should be lazy again. Jason are you working on this (subclassing from BufferedIndexInput)? If not I can take it.

Also, our finalizers aren't technically thread safe which could lead to a double close in the finalizer

Hmmm... I'll update both FSDirectory and NIOFSDiretory's isOpen's to be volatile.

[asfimport]

Jason Rutherglen (migrated from JIRA)

Mike I have have not started on the subclassing from BufferedIndexInput yet. I can work on it monday though.

[asfimport]

Michael McCandless (@mikemccand) (migrated from JIRA)

Mike I have have not started on the subclassing from BufferedIndexInput yet. I can work on it monday though.

OK, thanks!

[asfimport]

Michael McCandless (@mikemccand) (migrated from JIRA)

Updated patch with Yonik's volatile suggestion – thanks Yonik!

Also, I removed NIOFSDirectory.createOutput since it was doing the same thing as super().

[asfimport]

Matthew Mastracci (migrated from JIRA)

Michael,

Are you really sure you're not accidentally closing the searcher before calling Searcher.docFreqs? Are you calling docFreqs directly from your app?

Our IndexReaders are actually managed in a shared pool (currently 8 IndexReaders, shared round-robin style as requests come in). We have some custom reference counting logic that's supposed to keep the readers alive as long as somebody has them open. As new index snapshots come in, the IndexReaders are re-opened and reference counts ensure that any old index readers in use are kept alive until the searchers are done with them. I'm guessing we have an error in our reference counting logic that just doesn't show up under MMapDirectory (as you mentioned, close() is a no-op).

We're calling docFreqs directly from our app. I'm guessing that it just happens to be the most likely item to be called after we roll to a new index snapshot.

I don't have hard performance numbers right now, but we were having a hard time saturating I/O or CPU with FSDirectory. The locking was basically killing us. When we switched to MMapDirectory and turned on compound files, our performance jumped at least 2x. The preliminary results I'm seeing with NIOFSDirectory seem to indicate that it's slightly faster than MMapDirectory.

I'll try setting our app back to using the old FSDirectory and see if the exceptions still occur. I'll also try to fiddle with our unit tests to make sure we're correctly ref-counting all of our index readers.

BTW, I ran a quick FSDirectory/MMapDirectory/NIOFSDirectory shootout. It uses a parallel benchmark that roughly models what our real-life benchmark is like. I ran the benchmark once through to warm the disk cache, then got the following. The numbers are fairly stable across various runs once the disk caches are warm:

FS: 33644ms MMap: 28616ms NIOFS: 33189ms

I'm a bit surprised at the results myself, but I've spent a bit of time tuning the indexes to maximize concurrency. I'll double-check that the benchmark is correctly running all of the tests.

The benchmark effectively runs 10-20 queries in parallel at a time, then waits for all queries to complete. It does this end-to-end for a number of different query batches, then totals up the time to complete each batch.

[asfimport]

Jason Rutherglen (migrated from JIRA)

LUCENE-753.patch

NIOFSIndexInput now extends BufferedIndexInput. I was unable to test however and wanted to just get this up.

[asfimport]

Michael McCandless (@mikemccand) (migrated from JIRA)

FS: 33644ms MMap: 28616ms NIOFS: 33189ms

I'm a bit surprised at the results myself, but I've spent a bit of time tuning the indexes to maximize concurrency. I'll double-check that the benchmark is correctly running all of the tests.

This is surprising – your benchmark is very concurrent, yet FSDir and NIOFSDir are close to the same net throughput, while MMapDir is quite a bit faster. Is this on a non-Windows OS?

[asfimport]

Michael McCandless (@mikemccand) (migrated from JIRA)

New patch attached. Matthew if you could try this version out on your index, that'd be awesome.

I didn't like how we were still copying the hairy readBytes & refill methods from BufferedIndexInput, so I made some small additional mods to BufferedIndexInput to notify subclass when a byte[] buffer gets allocated, which then allowed us to fully inherit these methods.

But, then I realized we were duplicating alot of code from FSIndexInput, so I switched to subclassing that instead and that made things even simpler.

Some other things also fixed:

• We were ignoring bufferSize (eg setBufferSize).

• We weren't closing the FileChannel

• clone() now lazily clones the buffer again

To test this, I made NIOFSDirectory the default IMPL in FSDirectory.getDirectory and ran all tests. One test failed at first (because we were ignoring setBufferSize calls); with the new patch, all tests pass.

I also built first 150K docs of wikipedia and ran various searches using NIOFSDirectory and all seems good.

The class is quite a bit simpler now, however there's one thing I don't like: when you use CFS, the NIOFSIndexInput.readInternal method will wrap the CSIndexInput's byte[] (from it's parent BufferedIndexInput class) for every call (every 1024 bytes read from the file). I'd really like to find a clean way to reuse a single ByteBuffer. Not yet sure how to do that though...

[asfimport]

Michael McCandless (@mikemccand) (migrated from JIRA)

New version attached. This one re-uses a wrapped byte buffer even when it's CSIndexInput that's calling it.

I plan to commit in a day or two.

[asfimport]

Michael McCandless (@mikemccand) (migrated from JIRA)

I just committed revision 690539, adding NIOFSDirectory. I will leave this open, but move off of 2.4, until we can get similar performance gains on Windows...

[asfimport]

robert engels (migrated from JIRA)

SUN is accepting outside bug fixes to the Open JDK, and merging them to the commercial JDK (in most cases).

If the underlying bug is fixed in the Windows JDK - not too hard - then you fix this properly in Lucene.

If you don't fix it in the JDK you are always going to have the 'running out of file handles' synchronization, vs, the "locked position" synchronization - there is no way to fix this in user code...

[asfimport]

Yonik Seeley (@yonik) (migrated from JIRA)

Attaching new FileReadTest.java that fixes a concurrency bug in SeparateFile - each reader needed it's own file position.

[asfimport]

Uwe Schindler (@uschindler) (migrated from JIRA)

This issue was resolved a long time ago, but left open for the stupid Windows Sun JRE bug which was never resolved. With Lucene 3.x and trunk we have better defaults (use e.g. MMapDirectory on Windows-64).

Users should default to FSDirectory.open() and use the returned directory for best performance.