search

e2ebridge / xslt4node

A XSLT package wrapping the XSLT interface of the Java API for XML Processing (JAXP)
Other
11 stars 4 forks source link

Processing very large XML file leaks gigabytes of memory #9

Open srodovinsky opened 8 years ago

srodovinsky commented 8 years ago

Hi I am trying to process a very large (1.8GB) XML file, with an XSLT to JSON style sheet. The process works fine with up to 5K records, however when I try to feed the full file with ~180K records - the system runs out of memory (even when I specify -Xmx=4096m)

I have split the XML file into 5K record chunks using an excellent https://github.com/remuslazar/node-xmlsplit package. Now I am able to process all 35 (or so) chunks one time - great! However at the end of the transformation my node.js process would be sitting at around 4 GB in memory size. When I try the transformation again (it's supposed to be a daily process) - it will run out of memory.

So my question is - is there a way to release all memory resources after I am done with my XSLT processing?

Right now I am using a pretty ugly workaround - calling process.exit() and PM2 restarts my program. However I would like to see if there is a more civilized approach.

p-muessig commented 8 years ago

I'll be on vacation the next weeks. Will look into it when I'm back. Regards, Peter

grantcv1 commented 5 years ago

Still on vacation?