would it consume too much memory to sort large data set?

[chrischjh]

Hi there, is the sorting being done purely in memory? for example if I need to sort 5 million data, would it consume too much memory? thanks

[nhachicha]

Hi @chrischjh the C++ core uses std::sort on B+tree, so roughly speaking it should require ~ 100MB, that's:

• 16 bytes per entry for sorting = 80MB
• The returned TableView stores row numbers as 32-bit integers, meaning that the memory requirements of storing 5 million entries in a TableView are about the same as storing 5 million 32-bit integers, = 20 million bytes = 20 MB

I would suggest filtering the query to limit the number of results, before sorting.

[chrischjh]

Hey thanks for the answer, then I think it would not consume the memory of my Node.js process, right? That's exactly what I'm looking for. By the way, I'm using levelUp as my local cache, however the sorting has to be done in node.js process memory, also asked the same question to PouchDB, looks all levelDown db clients work on the similar way.

[astigsen]

With Realm all the query processing (including sorting) happens off-heap (it is entirely managed by the C++ part of the library), so it does not affect node's internal memory use (and is for the same reason not something node has to spend time garbage collecting afterwards).