search

monitorjbl / excel-streaming-reader

An easy-to-use implementation of a streaming Excel reader using Apache POI
Apache License 2.0
960 stars 345 forks source link

java.lang.OutOfMemoryError: GC overhead limit exceeded in FileBackedList.java #135

Closed edgarvonk closed 6 years ago

edgarvonk commented 6 years ago

Hi,

We are getting an 'OutOfMemoryError: GC overhead limit exceeded' in the FileBackedList class when reading very large (1GB) Excel files. This is in a Spring Boot app which currently has a max memory size of -Xmx1G.

We use the following settings for the excel streaming reader:

sst-cache-size=52428800 buffer-size=4096 row-cache-size=1000

Any thoughts? It might be that we are already consuming so much memory that not enough is left for the excel streaming reader.

Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
    at com.monitorjbl.xlsx.sst.FileBackedList.readFromFile(FileBackedList.java:107) ~[xlsx-streamer-1.2.1.jar!/:na]
    at com.monitorjbl.xlsx.sst.FileBackedList.getAt(FileBackedList.java:73) ~[xlsx-streamer-1.2.1.jar!/:na]
    at com.monitorjbl.xlsx.sst.BufferedStringsTable.getEntryAt(BufferedStringsTable.java:108) ~[xlsx-streamer-1.2.1.jar!/:na]
    at com.monitorjbl.xlsx.impl.StreamingSheetReader.unformattedContents(StreamingSheetReader.java:359) ~[xlsx-streamer-1.2.1.jar!/:na]
    at com.monitorjbl.xlsx.impl.StreamingSheetReader.handleEvent(StreamingSheetReader.java:205) ~[xlsx-streamer-1.2.1.jar!/:na]
    at com.monitorjbl.xlsx.impl.StreamingSheetReader.getRow(StreamingSheetReader.java:76) ~[xlsx-streamer-1.2.1.jar!/:na]
    at com.monitorjbl.xlsx.impl.StreamingSheetReader.access$200(StreamingSheetReader.java:37) ~[xlsx-streamer-1.2.1.jar!/:na]
    at com.monitorjbl.xlsx.impl.StreamingSheetReader$StreamingRowIterator.hasNext(StreamingSheetReader.java:412) ~[xlsx-streamer-1.2.1.jar!/:na]
edgarvonk commented 6 years ago

Never mind about this issue. I think it was caused by setting the sst cache size too large in combination with other processes that consumed a lot of memory in our case. I will close this issue.