The current process of appending all pages in a query to a results object causes memory overflow for large queries. They should really be written to a disk location or database instead of just filling up in memory.
Perhaps write connectors to sqllite, mongoDB, etc.
The current process of appending all pages in a
query
to aresults
object causes memory overflow for large queries. They should really be written to a disk location or database instead of just filling up in memory.Perhaps write connectors to sqllite, mongoDB, etc.