Switch backend to use querying before filtering

[Sun-flow]

Right now we are querying the whole database, then manually filtering assets.

Consider changing the initial filter to utilize the settings of the user, and then filter as necessary from there manually. This would reduce request size, and allow us to export the query in the future for the user to do their own filtering using MongoDB or some such method.

• use lists from data-schema-models to determine search parameters, there might be a resource online already for generating querys from tables/lists.

Another future application would be directly applying a custom/user-defined query filter, then getting presence outputs on the results.

[dbirman]

Generating the queries is definitely a small add, but currently the server is caching the entire database so performance would take a hit if I changed it to actually use the queries to retrieve records. The caching is probably worth it, if you make a large query it can take 20-30 seconds to request everything.

Adding the models for search parameters I'll add in a separate ticket.

And allowing for custom query filters that get pasted would also be easy to add.

[dbirman]

After discussion w/ David I'm going to implement this

• [ ] Refactor to use queries on back-end
• [ ] Display queries to user
• [ ] Allow users to enter queries

[dbirman]

Some notes for myself:

• Aggregation pipelines with logic run very slowly on the full dataset, not feasible to use in realtime
• Can build fake pipelines to display to user, but run filters locally? Entering queries won't be possible though
• Need to silently update cache server-side with manual caching and a task https://panel.holoviz.org/how_to/callbacks/schedule.html
• Can probably reduce downloads by only triggering cache misses when _id and last-updated dates change

[dbirman]

With the validation code running on the Panel server I think this is too hard to implement, need to revisit in the future. We can re-open this as a discussion in the future if we want