Add ExtrasFS to return "bonus" files/subdirs with directories. [LUCENE-6265]

[asfimport]

Lucene shouldn't care about files that don't match our index pattern. So if the operating system makes .DS_Store, thumbs.db, .nfsXXXXX, it shouldnt matter.

A few tests are sensitive to exact file names (they tend to be a pain for us anyway, when we change codecs and so on), these tests can use SuppressFileSystems annotation in #7326 to prevent this.

---

Migrated from LUCENE-6265 by Robert Muir (@rmuir), resolved Feb 21 2015 Attachments: LUCENE-6265.patch (versions: 4), TestDemoParallelReader-fix.txt

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

here is an initial patch. There are of course test failures, but i have not yet looked into them.

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

Here is a patch with test fixes. I am beasting lots of runs with nightly/slow.

[asfimport]

Michael McCandless (@mikemccand) (migrated from JIRA)

This is wonderful!

I will dig on TestDemoParallelReader...

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

just fixes another test bug found by beasting.

[asfimport]

Uwe Schindler (@uschindler) (migrated from JIRA)

Cool. Maybe instead of extraXXX as file name also use the list of known offenders like .DSStore to emulate more real-world examples. We can just have a list of them. Another idea would be to let it create "backup" files by adding another file with "\~" or ".bak" at end whenever an IndexOutput is created. This is also interesting, because those files could somehow match our file name patterns.

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

Well, i don't want to use "realistic" names because we are actually creating the files. So it could screw operating systems etc up :) I also like "extraNNNN" because it makes debugging easier. Finally, i don't want to test the case of someone being abusive here. This is about testing bonus files, which can happen by the OS. Very different from some abusive user who is opening files up with their editor...

[asfimport]

Chris M. Hostetter (@hossman) (migrated from JIRA)

Maybe instead of extraXXX as file name also use the list of known offenders like {{.DSStore}} to emulate more real-world examples.

woudn't that introduce a risk of the test trying to muck with actual files created by the OS on these types of filesystems?

[asfimport]

Michael McCandless (@mikemccand) (migrated from JIRA)

Here's a patch to fix TestDemoPLR "properly" ... it just restricts the namespace of its per-segment-per-gen subdirs.

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

Thanks Mike! I incorporated into this latest patch. I also removed the nocommit true.

I'm beasting a little bit more, but I think its ready.

[asfimport]

ASF subversion and git services (migrated from JIRA)

Commit 1661251 from @rmuir in branch 'dev/trunk' https://svn.apache.org/r1661251

LUCENE-6265: add extrasfs to return bonus files/subdirs with directories

[asfimport]

ASF subversion and git services (migrated from JIRA)

Commit 1661255 from @rmuir in branch 'dev/branches/branch_5x' https://svn.apache.org/r1661255

LUCENE-6265: add extrasfs to return bonus files/subdirs with directories

[asfimport]

ASF subversion and git services (migrated from JIRA)

Commit 1661285 from @rmuir in branch 'dev/branches/branch_5x' https://svn.apache.org/r1661285

LUCENE-6265: fix test bug

[asfimport]

ASF subversion and git services (migrated from JIRA)

Commit 1661286 from @rmuir in branch 'dev/trunk' https://svn.apache.org/r1661286

LUCENE-6265: fix test bug

[asfimport]

Timothy Potter (@thelabdude) (migrated from JIRA)

Bulk close after 5.1 release