search

agmen-hu / node-datapumps

Node.js ETL (Extract, Transform, Load) toolkit for easy data import, export or transfer between systems.
MIT License
291 stars 38 forks source link

ExcelWriterMixin - Streaming / memory handling for very large files #35

Closed CreativeTransmissions closed 8 years ago

CreativeTransmissions commented 8 years ago

Hi there,

Could you please advise whether exporting a very large number of rows (100,000 +) from MySQL will cause the ExcelWriterMixin to build the whole file in memory before creating the file, or if it will stream the output row by row, freeing memory as it goes?

Many thanks.

novaki commented 8 years ago

Hi,

Unfortunately, we could not find any streaming Excel lib when we developed the ExcelWriterMixin, so it buffers the whole the file in the memory at the moment. Feel free to upgrade or replace the mixin to be able to stream results.

CreativeTransmissions commented 8 years ago

No problem. Looks like a great project anyway. Thanks for letting me know!