search

apache / lucene

Apache Lucene open-source search software
https://lucene.apache.org/
Apache License 2.0
2.62k stars 1.02k forks source link

FieldCache introspection API [LUCENE-1749] #2823

Closed asfimport closed 15 years ago

asfimport commented 15 years ago

FieldCache should expose an Expert level API for runtime introspection of the FieldCache to provide info about what is in the FieldCache at any given moment. We should also provide utility methods for sanity checking that the FieldCache doesn't contain anything "odd"...

Migrated from LUCENE-1749 by Chris M. Hostetter (@hossman), 1 vote, resolved Aug 12 2009 Attachments: fieldcache-introspection.patch, LUCENE-1749.patch (versions: 16), LUCENE-1749-hossfork.patch Linked issues:

asfimport commented 15 years ago

Yonik Seeley (@yonik) (migrated from JIRA)

How about, for queries that need to go and look up docFreq, their QueryWeight impls simply hold onto the [top-level] IndexSearcher that had been passed to their ctor, and then do the docFreq call against that, if explain is invoked?

Asking the searcher for the docFreq is the right thing to do... but people who rely on Weight being serializable might be in for a nasty surprise. Of course... one might wonder if we should bother supporting serializable in Lucene longer term at all - anyone dealing with distributed systems has found it to have too many shortcomings anyway.

asfimport commented 15 years ago

Chris M. Hostetter (@hossman) (migrated from JIRA)

General Comments on mark's latest patch...

Big Questions I Still Have....

Lastly: This patch feels unnecessarily large at this point. Several of the bugs/improvements we've uncovered here don't seem like belong in this patch, and should be tracked in separate Jira issues, which can be committed independently and enumerated in CHANGES.txt....

Mark: would you mind splitting up the latest patch you have (you mentioned some additional minor tweaks) and opening new issues for these peripheral changes and then attaching back what's left for this patch. Then I'll take the conch back and work on the missing javadocs.

(I'll happily commit once i get at least one thumbs up from someone on the "Big Questions" above ... we can always tweak the javadocs further in subsequent commits)

asfimport commented 15 years ago

Chris M. Hostetter (@hossman) (migrated from JIRA)

Hoss/Mark do you want to fold it in to the patch, here? Or I can open a new issue?

as i alluded to above, i'm in favor of individual issues for each "bug" uncovered by this issue so they can be tracked separately.

asfimport commented 15 years ago

Mark Miller (@markrmiller) (migrated from JIRA)

Mark: would you mind splitting up the latest patch you have (you mentioned some additional minor tweaks) and opening new issues for these peripheral changes and then attaching back what's left for this patch.

I've already got separate issues and patches up were it makes sense (not the last one Mike mentions) - I wanted to keep them here too though until the insanity tests were complete - the tests that the fixes are somewhat correct are in this patch, and I don't like to manage layers of patches. if we don't plan on doing anymore with the insanity tests here though, I'll spin them out of this patch now.

I'll put up one more version and then you can have it back.

asfimport commented 15 years ago

Yonik Seeley (@yonik) (migrated from JIRA)

Asking the searcher for the docFreq is the right thing to do... but people who rely on Weight being serializable might be in for a nasty surprise.

If we decide not to ask weight to hang onto it's searcher, then the other way to do it right is to change explain() to accept a Searcher as well as a IndexReader.

asfimport commented 15 years ago

Mark Miller (@markrmiller) (migrated from JIRA)

is it a bug that AverageGuessMemoryModel.getSize() will NPE on a non primitive class ... or should/will the docs for that API say it only works on primitives?

Its only meant to work with primitives. I'll change the name to getPrimitiveSize - on my last pass through, I'll also review the javadoc for the classes I added.

asfimport commented 15 years ago

Michael McCandless (@mikemccand) (migrated from JIRA)

Asking the searcher for the docFreq is the right thing to do... but people who rely on Weight being serializable might be in for a nasty surprise.

Argh, right.

If we decide not to ask weight to hang onto it's searcher, then the other way to do it right is to change explain() to accept a Searcher as well as a IndexReader.

+1

Of course... one might wonder if we should bother supporting serializable in Lucene longer term at all - anyone dealing with distributed systems has found it to have too many shortcomings anyway.

Yeah this was never really "settled" in #2547. Lucene currently supports live serialization, but not cross-version serialization... and we have moved RemoteSearchable to contrib and removed RMI from Searchable.

Does Solr ever rely on Lucene's "implements Serializable"?

asfimport commented 15 years ago

Michael McCandless (@mikemccand) (migrated from JIRA)

Hoss/Mark do you want to fold it in to the patch, here? Or I can open a new issue?

as i alluded to above, i'm in favor of individual issues for each "bug" uncovered by this issue so they can be tracked separately.

OK I'll open a new issue for this one (deprecate Scorer.explain).

asfimport commented 15 years ago

Yonik Seeley (@yonik) (migrated from JIRA)

Does Solr ever rely on Lucene's "implements Serializable"?

Nope - itra-node communications use the same mechanism as clients... a generic data structures (Map,List,Document,etc) that has custom serialization to XML,JSON,Python,Ruby or Binary (and binary is now used by default).

asfimport commented 15 years ago

Mark Miller (@markrmiller) (migrated from JIRA)

patch is coming soon - I've merged to trunk and pulled the separate issues - just want to look over some a bit later. Would have had this sooner, but Eclipse decided to start crashing every 5 minutes this morning because firefox brought in a new xulrunner and ... ugg - at least its not Windows ... coming though.

asfimport commented 15 years ago

Mark Miller (@markrmiller) (migrated from JIRA)

I still havn't looked at this in the detail that I want to, but time is my enemy at the moment, so take it back and at least you can finish up what you have planned.

Hopefully its all in good working order after all the extracting and to trunking - let me know if you see anything off and I'll spin another.

asfimport commented 15 years ago

Mark Miller (@markrmiller) (migrated from JIRA)

P.S. I'm not sure we want to go with the way I have changed the tests here.

I switched things to go per subreader rather than use the overall reader - this is how things happen in IndexSearcher now. But we lose the top level reader test. We might want to do it both ways, and when doing it by top reader, ignore the triggered insanity check?

asfimport commented 15 years ago

Chris M. Hostetter (@hossman) (migrated from JIRA)

Mark: I'll start working on improving the docs (and other things from my previous todo list)

P.S. I'm not sure we want to go with the way I have changed the tests here.

Are you talking about TestOrdValues and TestFieldScoreQuery ?

if we expect OrdValues and FieldScoreQuery to use subReader based field caches, then the test seems to be doing the correct thing (in your patch) .. inspecting the fieldcaches per subreader. Is there a code path where we expect those methods to get called on a MultiReader?

(Actually: that seems like a wroth while improvement to make to these classes: a MultiDocValues impl that all of the getValues(IndexReader) methods use when passed a MultiReader ... it uses getSequentialSubReaders to construct DocValue instances for each so you don't get FieldCache expolsions if code inadvertenly passes the wrong reader to getValues. What do you think? ... new issue?)

asfimport commented 15 years ago

Chris M. Hostetter (@hossman) (migrated from JIRA)

Hmmmm...

actually mark, testing our your latest patch against hte trunk i'm seeing (FieldCache sanity) failures from TestCustomScoreQuery, TestFieldScoreQuery, TestOrdValues, and TestSort ... have you seen these? did some other recent change on the trunk trigger these?

asfimport commented 15 years ago

Mark Miller (@markrmiller) (migrated from JIRA)

I think that TestCustomScoreQuery, TestFieldScoreQuery, and TestOrdValues all fail because the fix for them is now in another issue.

TestSort I didn't notice. It looks like its considering String[] and StringIndex the same for the two multi and parallel sort tests - merged to trunk, so perhaps something has gone awry there? I've looked over the patch and I don't see any obvious mistake - I don't know that I have time to dig more now, but since you are more familiar with that code anyway, perhaps you can tell me why its now considering them the same anyway? Otherwise I will look more before too long.

asfimport commented 15 years ago

Mark Miller (@markrmiller) (migrated from JIRA)

Here is the output - it appears to think String[] and StringIndex are both string:

VALUEMISMATCH: Multiple distinct value objects for org.apache.lucene.index.CompoundFileReader$CSIndexInput@56d73c7a+string 'org.apache.lucene.index.CompoundFileReader$CSIndexInput@56d73c7a'=>'string',class org.apache.lucene.search.FieldCache$StringIndex,3,null,null=>org.apache.lucene.search.FieldCache$StringIndex#279807577 (size =\~ 152 bytes) 'org.apache.lucene.index.CompoundFileReader$CSIndexInput@56d73c7a'=>'string',class java.lang.String,11,null,null=>[Ljava.lang.String;#647057258 (size =\~ 108 bytes)

asfimport commented 15 years ago

Mark Miller (@markrmiller) (migrated from JIRA)

(Actually: that seems like a wroth while improvement to make to these classes: a MultiDocValues impl that all of the getValues(IndexReader) methods use when passed a MultiReader ... it uses getSequentialSubReaders to construct DocValue instances for each so you don't get FieldCache expolsions if code inadvertenly passes the wrong reader to getValues. What do you think? ... new issue?)

Very interesting idea - def a new issue I think. Not sure its worth it if you can't protect general fieldcache access as well though ...

asfimport commented 15 years ago

Chris M. Hostetter (@hossman) (migrated from JIRA)

checkpoint: no functional change from mark's previous patch, just improved all the javadocs, including explanation of SanityCheckers purpose and experimental/expert warnings where appropriate.

asfimport commented 15 years ago

Mark Miller (@markrmiller) (migrated from JIRA)

Okay, sorry - I messed up when merging with trunk.

In TestSort you had moved the local sorting to the bottom in the multi sort test - I kept that, but I also kept them higher up. So thats the fail - they just have to be removed.

Line 953-957 it looks - sorry bout that - just didn't notice it fail with the other 3.

asfimport commented 15 years ago

Chris M. Hostetter (@hossman) (migrated from JIRA)

I think that TestCustomScoreQuery, TestFieldScoreQuery, and TestOrdValues all fail because the fix for them is now in another issue.

ah ... are you talking about #2845 ? (the jira dependency sems backwards in that case)

In TestSort you had moved the local sorting to the bottom in the multi sort test - I kept that, but I also kept them higher up. So thats the fail - they just have to be removed.

yeah .. i just caught that and was starting to reply ... the interestingthing is that the CacheEntry.toString() doesn't show the Local.US was used when getting the Strings[] FieldCache. .. i'm currently trying to figure out why (because that could confuse people as well)

asfimport commented 15 years ago

Mark Miller (@markrmiller) (migrated from JIRA)

Right - I set that up when the code was in this issue - reversed now!

asfimport commented 15 years ago

Chris M. Hostetter (@hossman) (migrated from JIRA)

the interestingthing is that the CacheEntry.toString() doesn't show the Local.US was used when getting the Strings[] FieldCache

I'm an idiot ... the Locale isn't used like a FieldCache Parser ... the same String[] is used regardless of the Localed, so it's never part of the CacheKey. the output is correct.

revised patch fixes TestSort as mark pointed out, and updates some javadocs where i missleading suggested different Locales might trigger InsanityType.VALUEMISMATCH

asfimport commented 15 years ago

Chris M. Hostetter (@hossman) (migrated from JIRA)

confirmed that patch in #2845 fixes the remaining broken tests in this issue.

asfimport commented 15 years ago

Michael McCandless (@mikemccand) (migrated from JIRA)

Maybe we should simply print a warning, eg to System.err, on detecting that 2X RAM usage has occurred, pointing people to the sanity checker? We could eg do it once only so we don't spam the stderr logs...

asfimport commented 15 years ago

Chris M. Hostetter (@hossman) (migrated from JIRA)

Maybe we should simply print a warning, eg to System.err, on detecting that 2X RAM usage has occurred, pointing people to the sanity checker? We could eg do it once only so we don't spam the stderr logs

I'm not really comfortable dumping anything to System.err without user requesting it ... but this is a really interesting idea. (I suppose we could add an infoStream type idea to FieldCache to expose this)

FieldCacheImpl.Cache.get could use the FieldCacheSanityChecker to inspect itself immediately after calling createValue, and could even test if any of the Insanity instances returned are related to the current call (by comparing the CacheEntry with the Entry it's using) ... it could even log a useful stack trace since the sanity check would be happening in the same call stack as at least one of the CacheEntries in the Insanity object.

I've opened #2873 to track implmenting somehting like this once the FieldCacheSanityChecker gets committed.

asfimport commented 15 years ago

Chris M. Hostetter (@hossman) (migrated from JIRA)

slightly revised patch based on java-dev@lucene discussion...

the sortFieldTYpe and Locale portions of Cache.Entry are never used by FieldCache – just a deprecated class that abuses the Entry api out of lazyiness... so the CacheEntry debugging abstraction shouldn't expose them (but i left in code to manifest them in the toString() if they are atypical just in case). Also added some deprecation notices so we remember to remove them once they are no longer needed.

asfimport commented 15 years ago

Chris M. Hostetter (@hossman) (migrated from JIRA)

updated patch to trunk (QueryWeight->Weight) and tweaked some FieldCacheImpl methods to use the non-deprecated Entry constructors (forgot that part before)

I'll commit as soon as my test run is finished.

asfimport commented 15 years ago

Chris M. Hostetter (@hossman) (migrated from JIRA)

one last updated: the Locale.US asserts in TestRemoteSort had the same problem as TestSort, they were suppose to be moved, but instead they were just copied (not sure how i missed that before)

asfimport commented 15 years ago

Chris M. Hostetter (@hossman) (migrated from JIRA)

Committed revision 803676.