search

loiclacombe / dbdeploy

Automatically exported from code.google.com/p/dbdeploy
0 stars 0 forks source link

Cannot handle large files with data. ~150MB file give heap space error #37

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
I have a data file that is about 150MB, has 15 tables and the largest 
table has about 6 million rows. I get heap space error when I try to run 
it against mysql.

BUILD FAILED
java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:2882)
        at java.lang.AbstractStringBuilder.expandCapacity
(AbstractStringBuilder.java:100)
        at java.lang.AbstractStringBuilder.append
(AbstractStringBuilder.java:390)
        at java.lang.StringBuilder.append(StringBuilder.java:119)
        at com.dbdeploy.scripts.ChangeScript.getFileContents
(ChangeScript.java:82)
        at com.dbdeploy.scripts.ChangeScript.getContent
(ChangeScript.java:56)
        at com.dbdeploy.appliers.DirectToDbApplier.apply
(DirectToDbApplier.java:29)
        at com.dbdeploy.Controller.processChangeScripts(Controller.java:40)
        at com.dbdeploy.DbDeploy.go(DbDeploy.java:111)
        at com.dbdeploy.AntTarget.execute(AntTarget.java:35)
        at org.apache.tools.ant.UnknownElement.execute
(UnknownElement.java:288)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.tools.ant.dispatch.DispatchUtils.execute
(DispatchUtils.java:106)
        at org.apache.tools.ant.Task.perform(Task.java:348)
        at org.apache.tools.ant.Target.execute(Target.java:357)
        at org.apache.tools.ant.Target.performTasks(Target.java:385)
        at org.apache.tools.ant.Project.executeSortedTargets
(Project.java:1337)
        at org.apache.tools.ant.Project.executeTarget(Project.java:1306)
        at org.apache.tools.ant.helper.DefaultExecutor.executeTargets
(DefaultExecutor.java:41)
        at org.apache.tools.ant.Project.executeTargets(Project.java:1189)
        at org.apache.tools.ant.Main.runBuild(Main.java:758)
        at org.apache.tools.ant.Main.startAnt(Main.java:217)
        at org.apache.tools.ant.launch.Launcher.run(Launcher.java:257)
        at org.apache.tools.ant.launch.Launcher.main(Launcher.java:104)

Would like to see this fixed. Or, is there a way to run load file?

Original issue reported on code.google.com by roshan.s...@gmail.com on 8 Jan 2010 at 9:52

GoogleCodeExporter commented 9 years ago 
Hmm, it was never expected that dbdeploy would be run with such big files, and 
as you see from the stack trace 
it loads the contents of the files into an in-memory string.  This is probably 
fixable, especially in the case where 
you're applying directly to the database.

As a workaround, give ant more memory using -Xmx to give ant more memory -- see 
http://www.mail-
archive.com/dev@ant.apache.org/msg20970.html for an example.

Original comment by gtack...@googlemail.com on 10 May 2010 at 8:15