Open mut0u opened 8 years ago
I would also appreciate a tip how to handle big data.
Thanks for the feedback. It sounds like an interesting use cases.
The current stream story of Docjure is just to perform stream IO: the document is still built up in-memory.
I have not run into the memory-problem myself so I can offer no better advice than throwing more memory at it, or rolling up the sleeves and adding streaming to Docjure.
The underlying Apache POI library supports a limited streaming model for big datasets so please have a look at that and see if you can find a way to let Docjure leverage it.
You will find the documentation here: POI documentation - in particular the streaming API, SXSSF
I have a big data from database. I use the SQL limit offset and loop
to handle a litte part of data each time. But I have no idea that the data reference maybe exist all the time and the gc will never destroy the expired data. So I use 30G memory for create 20M xlsx file.
I am trying to rewrite my code to find out the way to solve my problem.
At the very beginning, I guess the create sheet will use lots of memory, so I want it can run like stream.
Finally, I figure out that it is due to my code .
I have a big data to save.
So I have to loop to call
add-rows!
tosheet
and then save thesheet
toworkbook
I guess the data is too big and the clojure throw
OutOfMemoryError GC overhead limit exceeded
Exception.So I have to change the sheet into stream and save the workbook with outputstream .
What should I do? Thanks.