Export documents fails completely for large collections (memory issue).
I'd guess this is related to the fact that the method tries to return a string and so at some point must put all the collection's documents into memory. The default max-memory-per-variable/max-string-length are fairly low in both JS/Node-JS.
Potential Fix
It'd be great if this method could return a ReadableStream instead?
Or maybe add a very similar sister function that returns a stream instead of a string?
Something like:
client.collections('companies').documents().exportAsStream()
Screenshot
Steps to reproduce
Export a sufficiently large collection. (Mine was 1.6 million records)
Description
Export documents fails completely for large collections (memory issue).
I'd guess this is related to the fact that the method tries to return a string and so at some point must put all the collection's documents into memory. The default max-memory-per-variable/max-string-length are fairly low in both JS/Node-JS.
Potential Fix
It'd be great if this method could return a ReadableStream instead?
Or maybe add a very similar sister function that returns a stream instead of a string? Something like:
client.collections('companies').documents().exportAsStream()
Screenshot
Steps to reproduce
Export a sufficiently large collection. (Mine was 1.6 million records)
Expected Behavior
Be able to export all documents in a collection.
Actual Behavior
Node-JS hits a memory related error
Typesense Version: Latest
OS: Linux/MacOS