Use pagination to optimizing queries

[ryanuc]

Hi, we're using Django Rest Multiple Models to combine several model types into a single API call (the specific use case is an Gallery, where the different models are Images, Panoramas and Videos)

Unfortunately due to the use case of a gallery, there are typically thousands of entries in the database. Pagination is especially useful for filtering down the number of requests.

However, as noted in the documentation pagination is not implemented in a way that can be used to enhance performance, only API usability. Unfortunately this means our request times can be quite large.

I'm wondering, do you have any plans to implement this in a way that could improve performance?

[MattBroach]

Hey @ryanuc -- I'd love to be able to get this working, but there are some technical difficulties (which you can see discussed in this issue. The problem essential boils down to this: how do you establish ordering across models without evaluating the querysets first?

In some ways, it depends on your use case. If you were content with pagination happening per-model (i.e. get the first 20 Images, the first 20 Panoramas, the first 20 Videos ... and then the next 20 Images, the next 20 Panoramas, etc), it would be fairly trivial to implement. I'm assuming that your Panoramas, Images, and Videos are all mixed together with an unpredictable ordering, and that's where this get tricky. I'm open to re-investigating the problem, but I haven't come up with a satisfactory solution yet.

[MattBroach]

@ryanuc, it's 6 months after the fact, so probably not useful for your previous needs, but pagination has now been properly implemented -- the filtering happens at the query level rather than after the querysets have been evaluated, so large datasets are handled correctly. There are a few caveats to be aware of:

1. The only pagination that has been implemented so far is LIMIT/OFFSET pagination; others may follow in the future, but this is the only pagination style currently implemented.
2. The new 2.0 version of DjangoRestMultipleModels went through a substantial change in code and interface: the View classes changed, queryList has become querylist (removed the awkward camelCase), and querylist items are now dicts, rather than tuples. I'd recommend reading the 1.x to 2.0 guide in the docs for a full description of what changed and how to update your code.

Although this probably happened too late to be useful to you, let me know if you get a chance to play with the new version of the library. Would love to get some user feedback on the new structure and pagination.