typesense / typesense-js

JavaScript / TypeScript client for Typesense
https://typesense.org/docs/api
Apache License 2.0
414 stars 75 forks source link

Export documents fails for large data sets (1.6million records) #140

Closed Bug-Reaper closed 1 year ago

Bug-Reaper commented 1 year ago

Description

Export documents fails completely for large collections (memory issue).

I'd guess this is related to the fact that the method tries to return a string and so at some point must put all the collection's documents into memory. The default max-memory-per-variable/max-string-length are fairly low in both JS/Node-JS.

Potential Fix

It'd be great if this method could return a ReadableStream instead?

Or maybe add a very similar sister function that returns a stream instead of a string? Something like: client.collections('companies').documents().exportAsStream()

Screenshot

image

Steps to reproduce

Export a sufficiently large collection. (Mine was 1.6 million records)

Expected Behavior

Be able to export all documents in a collection.

Actual Behavior

Node-JS hits a memory related error

Typesense Version: Latest

OS: Linux/MacOS

jasonbosco commented 1 year ago

@Bug-Reaper There's already an exportStream method: https://github.com/typesense/typesense-js/blob/b0854ab5829b6819e82b6fa9d3304643b9d0772e/src/Typesense/Documents.ts#L255