PavlidisLab / Gemma

Genomics data re-analysis
Apache License 2.0
23 stars 6 forks source link

Cache individual clauses of a filtering query #870

Open arteymix opened 1 year ago

arteymix commented 1 year ago

To enhance GemBrow performance with complex filters, we can break down the query in individual clauses, query their IDs, intersect those and then retrieve entities by IDs.

In practice, this will be greatly beneficial because a user only modify a clause at a time. All others will be retrieved from the cache.

This could be done in the Cached filtering service implementation as it make no sense to do this for uncached filters.

arteymix commented 1 year ago

When retrieving entities by ID, we have an opportunity to load then from the cache instead of querying the database.

This is only applicable of the results are not sorted.

I wonder if it would be possible to sort individual queries and combine the results from the by-ID query. This would allow us to always lookup the cache and only retrieve missed entities.