out of memory when backing up with a large book

[fefe982]

Again with my large book, when I tried to delete all the transactions, the backup procedure triggered an out-of-memory exception. The stack trace is as follows:

08-28 22:50:39.100 25347-25347/org.gnucash.android E/dalvikvm-heap﹕ Out of memory on a 7543522-byte allocation.
08-28 22:50:39.100 25347-25347/org.gnucash.android I/dalvikvm﹕ "main" prio=5 tid=1 RUNNABLE
08-28 22:50:39.100 25347-25347/org.gnucash.android I/dalvikvm﹕ | group="main" sCount=0 dsCount=0 obj=0x416ab700 self=0x400be0b0
08-28 22:50:39.100 25347-25347/org.gnucash.android I/dalvikvm﹕ | sysTid=25347 nice=0 sched=0/0 cgrp=apps handle=1074958128
08-28 22:50:39.100 25347-25347/org.gnucash.android I/dalvikvm﹕ | schedstat=( 0 0 0 ) utm=85267 stm=4262 core=1
08-28 22:50:39.100 25347-25347/org.gnucash.android I/dalvikvm﹕ at java.lang.AbstractStringBuilder.enlargeBuffer(AbstractStringBuilder.java:~94)
08-28 22:50:39.100 25347-25347/org.gnucash.android I/dalvikvm﹕ at java.lang.AbstractStringBuilder.append0(AbstractStringBuilder.java:132)
08-28 22:50:39.100 25347-25347/org.gnucash.android I/dalvikvm﹕ at java.lang.StringBuffer.append(StringBuffer.java:126)
08-28 22:50:39.100 25347-25347/org.gnucash.android I/dalvikvm﹕ at java.io.StringWriter.write(StringWriter.java:135)
08-28 22:50:39.100 25347-25347/org.gnucash.android I/dalvikvm﹕ at org.apache.xml.serializer.ToStream.printSpace(ToStream.java:819)
08-28 22:50:39.100 25347-25347/org.gnucash.android I/dalvikvm﹕ at org.apache.xml.serializer.ToStream.indent(ToStream.java:796)
08-28 22:50:39.100 25347-25347/org.gnucash.android I/dalvikvm﹕ at org.apache.xml.serializer.ToStream.indent(ToStream.java:806)
08-28 22:50:39.100 25347-25347/org.gnucash.android I/dalvikvm﹕ at org.apache.xml.serializer.ToStream.startElement(ToStream.java:1938)
08-28 22:50:39.110 25347-25347/org.gnucash.android I/dalvikvm﹕ at org.apache.xalan.transformer.TransformerIdentityImpl.startElement(TransformerIdentityImpl.java:1073)
08-28 22:50:39.110 25347-25347/org.gnucash.android I/dalvikvm﹕ at org.apache.xml.serializer.TreeWalker.startNode(TreeWalker.java:359)
08-28 22:50:39.110 25347-25347/org.gnucash.android I/dalvikvm﹕ at org.apache.xml.serializer.TreeWalker.traverse(TreeWalker.java:145)
08-28 22:50:39.110 25347-25347/org.gnucash.android I/dalvikvm﹕ at org.apache.xalan.transformer.TransformerIdentityImpl.transform(TransformerIdentityImpl.java:390)
08-28 22:50:39.110 25347-25347/org.gnucash.android I/dalvikvm﹕ at org.gnucash.android.export.xml.GncXmlExporter.generateExport(GncXmlExporter.java:175)
08-28 22:50:39.110 25347-25347/org.gnucash.android I/dalvikvm﹕ at org.gnucash.android.export.xml.GncXmlExporter.createBackup(GncXmlExporter.java:192)
08-28 22:50:39.110 25347-25347/org.gnucash.android I/dalvikvm﹕ at org.gnucash.android.ui.settings.DeleteAllAccountsConfirmationDialog$2.onClick(DeleteAllAccountsConfirmationDialog.java:53)
08-28 22:50:39.110 25347-25347/org.gnucash.android I/dalvikvm﹕ at com.android.internal.app.AlertController$ButtonHandler.handleMessage(AlertController.java:166)
08-28 22:50:39.110 25347-25347/org.gnucash.android I/dalvikvm﹕ at android.os.Handler.dispatchMessage(Handler.java:99)
08-28 22:50:39.110 25347-25347/org.gnucash.android I/dalvikvm﹕ at android.os.Looper.loop(Looper.java:137)
08-28 22:50:39.110 25347-25347/org.gnucash.android I/dalvikvm﹕ at android.app.ActivityThread.main(ActivityThread.java:4914)
08-28 22:50:39.110 25347-25347/org.gnucash.android I/dalvikvm﹕ at java.lang.reflect.Method.invokeNative(Native Method)
08-28 22:50:39.110 25347-25347/org.gnucash.android I/dalvikvm﹕ at java.lang.reflect.Method.invoke(Method.java:511)
08-28 22:50:39.110 25347-25347/org.gnucash.android I/dalvikvm﹕ at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:808)
08-28 22:50:39.110 25347-25347/org.gnucash.android I/dalvikvm﹕ at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:575)
08-28 22:50:39.110 25347-25347/org.gnucash.android I/dalvikvm﹕ at dalvik.system.NativeStart.main(Native Method)
08-28 22:50:39.110 25347-25347/org.gnucash.android I/dalvikvm﹕ [ 08-28 22:50:39.110 25347:25347 W/dalvikvm ]

[fefe982]

Took a look at the code related to exporting. When exporting to XML, the whole DOM structure and the whole generated XML string (and maybe many other stuff) are kept in memory at the same time. This should have cause the out of memory problem with my big book, whose GNC XML file (unzipped) generated by GnuCash is 12M.

This may need a change of the export interface to change. The current interface returns the exported content as a String. We can let the interface take a Writer (or something alike) as a parameter and write to it. If we want a String, then we can pass in a StringWriter. If we are exporting to file, a FileWriter can be used to reduce memory cost.

Also, although using DOM to export XML is very intuitive, we do not have to generate the DOM at all when exporting to XML. We can take the database content and write the XML directly. Android has org.xmlpull.v1.XmlSerializer (API level I) to do the job. See http://stackoverflow.com/a/2296051/1058916 and http://stackoverflow.com/a/5186414/1058916 .

[fefe982]

The GncXmlExporter does not seem to check or set the exported status, is it right?

[codinguser]

I concur. This was an artifact of the former support for exporting only changed transactions. The export files tended to be small. But the XML export backs up everything which can be a bit heavy. I will change the interface to work accordingly. Or do you want to do it?

On 02.09.2014, at 16:19, Yongxin Wang notifications@github.com wrote:

Took a look at the code related to exporting. When exporting to XML, the whole DOM structure and the whole generated XML string (and maybe many other stuff) are kept in memory at the same time. This should have cause the out of memory problem with my big book, whose GNC XML file (unzipped) generated by GnuCash is 12M.

This may need a change of the export interface to change. The current interface returns the exported content as a String. We can let the interface take a Writer (or something alike) as a parameter and write to it. If we want a String, then we can pass in a StringWriter. If we are exporting to file, a FileWriter can be used to reduce memory cost.

Also, although using DOM to export XML is very intuitive, we do not have to generate the DOM at all when exporting to XML. We can take the database content and write the XML directly. Android has org.xmlpull.v1.XmlSerializer (API level I) to do the job. See http://stackoverflow.com/a/2296051/1058916 and http://stackoverflow.com/a/5186414/1058916 .

— Reply to this email directly or view it on GitHub.

[codinguser]

No it doesn't. XML exports are always the whole file.

On 02.09.2014, at 16:43, Yongxin Wang notifications@github.com wrote:

The GncXmlExporter does not seem to check or set the exported status, is it right?

— Reply to this email directly or view it on GitHub.

[fefe982]

Changing the interface would affect a lot of things. I can do the XML part, and I know something about QIF. But I never touched the OFX format.

The DB access strategy should also change to make the exporting of the whole book faster, one query for Accounts and one query for Transactions should be enough.

I think I would do the XML part first, add separate function in the XML exporter, using XMLSerializer and a new db access strategy and may be write directly to a zipped file (when this is a backup), and see how it works. I won't touch QIF or OFX export or change the abstract interface function until this is finished.

[fefe982]

See #193 for new XML export, #198 for QIF export.

I didn't find enough documents on OFX format to do a full rewrite for OFX export. Changing only the interface is easy, as a StringWriter is used inside OFXExporter.generateExport, just replace it with the argument writer would be enough. However, to get a faster exporter a full rewrite would be needed, which requires knowledge about the OFX format.

I'll leave OFX export there and try what I can do with the problems introduced by #188 . Will create a new issue to discuss the details. Are you already working on this?

[codinguser]

Yes, I think we can leave OFX as-is. It doesn't support double entry transactions anyway and is a hassle to use to import transactions into the desktop GnuCash.

[codinguser]

So this can be closed now right?

[fefe982]

Sure, I'll close it.