Improve export

[dpapathanasiou]

Made some improvements to how the export logic checks the relational db/django ORM:

• Used Q objects to create an "OR" query rather than a sql subquery, which is expensive
• Instead of querying the export table twice, first to take a count, then to determine if the latest export timestamp is earlier than the latest XForm submission, just call the latter function instead

This logic flow is still very ugly: the exports_outdated() function really should not be a classmethod of Export, but doing it incrementally like this for now, until can do a more thorough overhaul later.

• like this for now, until can do a more thorough overhaul later.

[dpapathanasiou]

Using "latest()" like this in the ORM is efficient (especially now that the rest of the query uses Q models for the pending/success "or" condition rather than creating a sub query, which is what the '__in=[]' was doing), so the right place to add the time frame limitation is in the "query_mongo()" function, which is another beast entirely...

[prabhasp]

Not about query efficiency, but more about when to trigger exports (which unleashes the mongo beast). We can walk through it tomorrow or Monday. On Apr 24, 2014 6:14 PM, "Denis Papathanasiou" notifications@github.com wrote:

Using "latest()" like this in the ORM is efficient (especially now that the rest of the query uses Q models for the pending/success "or" condition rather than creating a sub query, which is what the '__in=[]' was doing), so the right place to add the time frame limitation is in the "query_mongo()" function, which is another beast entirely...

Reply to this email directly or view it on GitHubhttps://github.com/SEL-Columbia/formhub/pull/1322#issuecomment-41339245 .

[dpapathanasiou]

Right, that's what I meant about changing "query_mongo()" -- setting a date range or limit here, on the ORM query won't do that.

Changing "query_mongo()" to do just the diffs is also not that easy, for reasons too involved to explain here.