apache / lucene

Apache Lucene open-source search software
https://lucene.apache.org/
Apache License 2.0
2.66k stars 1.03k forks source link

Extendable writer and reader of field data [LUCENE-662] #1737

Closed asfimport closed 14 years ago

asfimport commented 18 years ago

As discussed on the dev mailing list, I have modified Lucene to allow to define how the data of a field is writen and read in the index.

Basically, I have introduced the notion of IndexFormat. It is in fact a factory of FieldsWriter and FieldsReader. So the IndexReader, the indexWriter and the SegmentMerger are using this factory and not doing a "new FieldsReader/Writer()".

I have also introduced the notion of FieldData. It handles every data of a field, and also the writing and the reading in a stream. I have done this way because in the current design of Lucene, Fiedable is an interface, so methods with a protected or package visibility cannot be defined.

A FieldsWriter just writes data into a stream via the FieldData of the field. A FieldsReader instanciates a FieldData depending on the field name. Then it use the field data to read the stream. And finnaly it instanciates a Field with the field data.

About compatibility, I think it is kept, as I have writen a DefaultIndexFormat that provides some DefaultFieldsWriter and DefaultFieldsReader. These implementations do the exact job that is done today. To acheive this modification, some classes and methods had to be moved from private and/or final to public or protected.

About the lazy fields, I have implemented them in a more general way in the implementation of the abstract class FieldData, so it will be totally transparent for the Lucene user that will extends FieldData. The stream is kept in the fieldData and used as soon as the stringValue (or something else) is called. Implementing this way allowed me to handle the recently introduced LOAD_FOR_MERGE; it is just a lazy field data, and when read() is called on this lazy field data, the saved input stream is directly copied in the output stream.

I have a last issue with this patch. The current design allow to read an index in an old format, and just do a writer.addIndexes() into a new format. With the new design, you cannot, because the writer will use the FieldData.write provided by the reader.

enjoy !


Migrated from LUCENE-662 by Nicolas Lalevée, 1 vote, resolved Jul 01 2010 Attachments: ASF.LICENSE.NOT.GRANTED--generic-fieldIO.patch, entrytable.patch, generic-fieldIO-2.patch, generic-fieldIO-3.patch, generic-fieldIO-4.patch, generic-fieldIO-5.patch, indexFormat.patch (versions: 3), indexFormat-only.patch Linked issues:

asfimport commented 18 years ago

Nicolas Lalevée (migrated from JIRA)

I think I got it. What was disturbing on the last patch was the notion of FieldData I added. So I removed it. So let's summerize the diff between the trunk and my patch :

I have added an exemple of implementation in the patch that use this future : look at org.apache.lucene.index.rdf

I know this is a big patch but I think the API has not been broken, and I would appreciate comments on this.

asfimport commented 18 years ago

Grant Ingersoll (@gsingers) (migrated from JIRA)

Haven't looked fully at the patch, but one thing I am curious about is why remove the Fieldable interface?

asfimport commented 18 years ago

Nicolas Lalevée (migrated from JIRA)

It is due to lazy loading. A lazy field, when being retreived data, have to know how to read the stream. In the current trunk, a special implementation of Field is doing this. Here, we don't have control of the implemenation of Fieldable it will be. As I wanted to keep the lazy loading mechanism controlled internally in Lucene, being transparent to the user, I had to force every Fieldable implementation to know how about retreiving data lazily. So I switched the interface to an abstract class : in fact I have moved AbstractField to Fieldable. But as I already raised, I still have an issue about it : the lazy loading mechanism isn't totally internal. The function Fieldable.setLazyData() shouldn't be public but default.

asfimport commented 18 years ago

Nicolas Lalevée (migrated from JIRA)

I just realized reading the recent discussing on lucene-dev (LazyField use of IndexInput not thread safe) that the implementation I have done isn't thread safe at all. The input is not cloned at all...

asfimport commented 17 years ago

Nicolas Lalevée (migrated from JIRA)

Here is an update of the patch:

asfimport commented 17 years ago

Grant Ingersoll (@gsingers) (migrated from JIRA)

Hi Nicolas,

Have you run any benchmarks on this? Once I finish up some documentation stuff, my plan is to start digging into this.

-Grant

asfimport commented 17 years ago

Nicolas Lalevée (migrated from JIRA)

Not at all.

In fact we don't use a lucene modified with my patch in our system. I only start working with lucene this year, and our search engine is a too critical component to play with a patched trunk. So I have even not tested it in real condition.

asfimport commented 17 years ago

Nicolas Lalevée (migrated from JIRA)

Patch synchronized with the trunk. I also tried to minimize the diff. And in fact I just realized that there are two patchs in one there :

I will try to seperate them.

asfimport commented 17 years ago

Nicolas Lalevée (migrated from JIRA)

here it is : I have synchronized with the current trunk, and I have splited the patch in two parts.

asfimport commented 17 years ago

Nicolas Lalevée (migrated from JIRA)

Patch update: synchronized with the trunk and new features.

public interface PostingWriter { public void close() throws IOException; public long[] getPointers(); public int getNbPointer(); public long writeSkip(RAMOutputStream skipBuffer) throws IOException; public void write(int doc, int lastDoc, int nbPos, int[] positions) throws IOException; }

public interface PostingReader { public void close() throws IOException ; public TermDocs termDocs(BitVector deletedDocs, TermInfosReader tis, FieldInfos fieldInfos) throws IOException; public TermPositions termPositions(BitVector deletedDocs, TermInfosReader tis, FieldInfos fieldInfos) throws IOException; }

Furthermore this "generification" also allows an implementation invoked many times : http://wiki.apache.org/jakarta-lucene/FlexibleIndexing Note that it does not break the actual format. The .tis file is still managed internaly by Lucene and it holds pointers to some external files (managed by the indexFormat). The implementation of the PostingReader/PostingWriter specify how many pointers there are. The default one is 2 : .frq and .prx. The FlexibleIndexing would be 1.

asfimport commented 17 years ago

Michael Busch (migrated from JIRA)

Nicolas,

wow this looks like a lot of work!! There are good ideas in your patch. I have been and am currently very busy (moving to a new country...), so I probably won't have a chance to review it for another week or so.

Michael

asfimport commented 17 years ago

Nicolas Lalevée (migrated from JIRA)

Thanks Michael ! I will appreciate a review and feedbacks, as it will open a lot the API, this will go even further than just make Document public (#1853).

asfimport commented 17 years ago

Grant Ingersoll (@gsingers) (migrated from JIRA)

Hi Nicolas,

I tried applying indexFormat.patch and am getting: grantingersoll@britta[1016]$ patch -p 0 -i ../patches/indexFormat.patch --dry-run patching file src/test/org/apache/lucene/store/IndexInputTest.java patching file src/test/org/apache/lucene/index/DocHelper.java patching file src/test/org/apache/lucene/index/TestIndexFormat.java can't find file to patch at input line 369 Perhaps you used the wrong -p or --strip option? The text leading up to this was:


Property changes on: src/test/org/apache/lucene/index/TestIndexFormat.java
___
Name: svn:keywords
+ Date Revision Author HeadURL Id
Name: svn:eol-style
+ native
Index: src/test/org/apache/lucene/index/impl/TestSegmentTermDocs.java
===================================================================
— src/test/org/apache/lucene/index/impl/TestSegmentTermDocs.java (revision 0)
+++ src/test/org/apache/lucene/index/impl/TestSegmentTermDocs.java (working copy)

-----
File to patch:

Meaning, it doesn't know what to do with this diff. From the looks of it, TestSegmentTermDocs.java did not get move to the impl directory from the directory it was in.

I'm not sure how to handle this in SVN, but I suspect you have to do a local copy move first. Perhaps try applying this patch to a clean checkout to let me know if it works for you. Also, perhaps we can collaborate with Doron to write up some benchmarks or to at least make sure the existing benchmarks are covering this new way.

asfimport commented 17 years ago

Nicolas Lalevée (migrated from JIRA)

Hum... same here.... This is due to some svn mv, and I created the patch with svn diff. I can provide a patch with the complete diff, but you will loose the svn mv infos, so the svn history of the file will be lost. Any advise is welcomed. I will also ask on monday to my colleagues how they use to work with svn mv and patches.

asfimport commented 17 years ago

Nicolas Lalevée (migrated from JIRA)

Patch updated and synchornized with the trunk r517330. I have removed the "svn mv" I have done so now the patch is applying fine on a fresh trunk. The svn mv was just about creating the package impl. So everthing came back to o.a.l.index.

Note about the last commit in trunk I have merged : lazy loading of the "proxstream". The feature is lost within this patch. I didn't took time to merge it properly. I think this is hightly feasable, but not just done. So a new item on the TODO list.

asfimport commented 17 years ago

Nicolas Lalevée (migrated from JIRA)

Synchronized with the trunk, so with the payload feature. It allowed me to refactor in one class the payload writing which is in two places today : it is now in the DefaultPostingWriter class.

From my last update, the TODO list is still to do, nothing has been fixed. Furthermore there is a regression in the new patch : the ensureOpen() is not correctly handled for lazy loaded fields : a test fail. This is due to the fact that the FieldsReader doesn't handle it anymore in my patch. As the data struture can be customized, lazy loading is exported to the FieldData created by the FieldsReader. So the both instance have to communicate about the closing of the streams. So a new item in the TODO list.

As discussed in java-dev, here is a light patch with only the index format handling, without the possibility to redefine how data and postings are store/retreived.

asfimport commented 14 years ago

Grant Ingersoll (@gsingers) (migrated from JIRA)

Marking as won't fix, as I think the new flex indexing stuff takes care of this.