Add Query Memory Estimation Ability in QueryVisitor [LUCENE-8864]

[mikemccand]

In LUCENE-8855, there is a discussion around adding memory accounting capabilities to QueryVisitor to allow estimation of memory consumption by queries.'

This Jira tracks the effort

---

Legacy Jira details

LUCENE-8864 by Atri Sharma (@atris) on Jun 15 2019, updated Jun 18 2019

[mikemccand]

What kind of API do you propose here? Theoretically, if a query tree contains queries that implement Accountable we can already use existing QueryVisitor API to walk through the query tree and collect ramBytesUsed() from each sub-query. And if a Query doesn't implement Accountable we have no way to predict this (apart from `RamUsageEstimator.shallowSizeOf(Object)`).

[Legacy Jira: Andrzej Bialecki on Jun 18 2019]

[mikemccand]

Right, the purpose of this Jira was twofold:

 

1) To throw out thoughts about making memory accounting a first class citizen within QueryVisitor. I think it would be good if we added a method which returned the overall size of the underlying query. This fits in nicely with QueryVisitor's model since queries can be nested, so it is good to get the "deep" memory usage of the parent query. As you said, the new method could return the Accountable's estimate or shallow size if Accountable is not supported.

 

2) Borrow ideas from QueryVisitor design to see if we can improve Accountable itself. While this is orthogonal and I have not really thought through every corner case, my instinct says that there might be opportunities to improve Accountable's APIs to be more recursive in nature. For eg, there are a ton of instanceof checks present today, for each Query type. Should we think about delegating some of that calculation to a visitor type model which localizes the per query calculation to the query's scope?

[Legacy Jira: Atri Sharma (@atris) on Jun 18 2019]