Making querysets lazy

[agateblue]

This would lead to a serious performance improvements if querysets where lazily evaluated and all queries where combined in a single one before iterating on values.

This was first raised on reddit:

the idea is fun but the implementation looks bad. I supposed from a fast review that every time a filter asked, it is applied. It result with lots of copy and lots of iteration.

to avoid that every filters, sorting fields and limit or whatever should be store in the QuerySet and a method all() first() or aggregation method could optimize iterations and copy.

[agateblue]

I think one way to achieve this is to follow django's implementation and internally use Q objects to represent query arguments. Each queryset could then be reduced to a list of Q objects, and when evaluation would be need, values would be populated using these objects as matchers, meaning we iterate only once.

[asfaltboy]

Nice! While using Q objects may seem like overkill, something like this would have the advantage of enabling "advanced queries" such as supporting advanced filters with operators like these

• Not - ~Q(foo=bar)
• Or- Q(foo=baz) | Q(foo=bar)
• Nested - Q(Q(foo=baz) | Q(foo=bar)) & Q(foo=spam) - this is a single Q object

Internally, Q objects could be visualized as a dictionary of filter parameters, and can even be de/serialized as such. I did something similar here.

[agateblue]

I agree it may seems overkill right now, but I also think we should not reinvent the wheel and reuse existing implementations (such as Django's) to solve this issue.

Handling complex OR/AND queries as you described would definitely be great stuff!

[agateblue]

I'started the work on this feature, the first part being using Q-like objects internally.

[agateblue]

This was also fixed in #23