ENH: add ability to limit Entry searches bases on ancestor

[shilorigins]

Our search methods don't provide a way to limit results based on ancestor. For example, I have a PV in one snapshot and I want to find the PV with the same name in a second snapshot. There's no direct way to do this, because the search will return any Setpoints with the same name in the entire backend, and there aren't other search terms that will properly limit the search.

This may work as a new SearchTerm operator, or as an arg in Client.search().

[shilorigins]

I wanted to record some thoughts I had on this before I got started.

Relying on the backend to trace parentage is potentially very time consuming, since the backend will search all of its entries and doesn't necessarily store data hierarchically. It may be valuable to logically separate backend-level searching and client-level searching for when we want to filter on relationships between entries or don't want to search the whole database.

A few ways to separate backend and client searching:

• mix both types in SearchTerm and have Client.search filter out its SearchTerms before passing the rest to _Backend.search and then post-processing those results
• use kwargs to set client-level search criteria
  • useful for a small set of criteria
  • suffers some of the same deficiencies that caused us to implement SearchTerm in the first place
  • terms don't need re-implementation like backend search terms do
• separate client-level and backend-level search into separate methods

In this case, we could have the client traverse the Nestable, filling data from the backend as needed, which shouldn't take too much time since the ancestor term immediately limits the search space. Traversing Nestables is a repeating motif, so I may approach this by implementing a Client.traverse(visitor: Callable (or maybe Visitor?)) method to hopefully consolidate some duplicate code.

[tangkong]

Part of my intent behind keeping the search functionality in the backend was so that we could customize the search patterns / algorithms to suite the backend of choice. I expect the way we search a mongo backend to be very different from the way we search the filestore backend w.r.t caching, querying etc.

For this reason I'm hesitant to agree on splitting the search logic between the client and backend, unless we're absolutely certain that the optimization is valid for all the backends we support. In the end this is an optimization concern right? (Possibly pre-mature optimization? I won't call it that just yet)

[shilorigins]

It's sort of an optimization concern, but I'm not entirely sure how we'd implement this in a mongo backend, for example. It's relatively simple when the search terms only inspect attributes of the entries, but in abstract I'm not sure how we would tell a mongo database to trace parent-child relationships between entries. Can you give mongo custom functions? Would that part of the search end up in python, just in MongoBackend instead of Client? Maybe this is premature since we haven't touched a mongo backend yet.

[shilorigins]

I may have just talked myself into doing basically what I was thinking, but in the _Backend subclasses rather than Client.

[tangkong]

... but I'm not entirely sure how we'd implement this in a mongo backend, for example. ...

I think this is something we may want to tackle right after we finish all the MVP gui stuff. It'll be good to enable a real database and actually give the app some meaning.