search

agateblue / lifter

A generic query engine, inspired by Django ORM
ISC License
449 stars 16 forks source link

Optimize query matching when chaining filters #25

Closed agateblue closed 8 years ago

agateblue commented 8 years ago

The current pattern when chaining filters do not merge queries together:

manager.filter(query1).exclude(query2)

When evaluating the queryset, it will loop over available values, return the one that match query1, then loop other these remaining values and exclude the one that match query2.

It would be more efficient to loop only once on all values, and return only values that match all queries.

Mec-iS commented 8 years ago

You may find something interesting at Python Dee perhaps.

agateblue commented 8 years ago

Thanks for the link, I never heard about it but there are sure interesting ideas in there.

Regarding this specific issue, though, this is something I'm working on by compiling queries together, before running them against the dataset. You can see the feature/backend branch for more details.

Mec-iS commented 8 years ago

Cool. I thought it could be something to be done using sets' intersection. I will follow along on the branch. Maybe add the reference link to the solution you are trying then.

agateblue commented 8 years ago

You're right, I've opened a regular PR to show the branch state, please refer to #34 for more details.