search

ghmo / google-refine

Automatically exported from code.google.com/p/google-refine
0 stars 0 forks source link

Large tsv files raise "Java heap space" error #145

Open GoogleCodeExporter opened 8 years ago

GoogleCodeExporter commented 8 years ago 
What steps will reproduce the problem?
1. Attempt to create a project with large file (88mb was sufficient to cause 
error)

What is the expected output? What do you see instead?
Slow running, but completed import. HTTP 500 Error.

Traceback:

HTTP ERROR 500

Problem accessing /command/core/create-project-from-upload. Reason:

    Java heap space

Caused by:

java.lang.OutOfMemoryError: Java heap space
    at java.util.ArrayList.<init>(ArrayList.java:112)
    at com.google.gridworks.model.Row.<init>(Row.java:29)
    at com.google.gridworks.importers.TsvCsvImporter.read(TsvCsvImporter.java:128)
    at com.google.gridworks.importers.TsvCsvImporter.read(TsvCsvImporter.java:41)
    at com.google.gridworks.commands.project.CreateProjectCommand.internalInvokeImporter(CreateProjectCommand.java:443)
    at com.google.gridworks.commands.project.CreateProjectCommand.load(CreateProjectCommand.java:311)
    at com.google.gridworks.commands.project.CreateProjectCommand.internalImportFile(CreateProjectCommand.java:297)
    at com.google.gridworks.commands.project.CreateProjectCommand.internalImport(CreateProjectCommand.java:139)
    at com.google.gridworks.commands.project.CreateProjectCommand.doPost(CreateProjectCommand.java:79)
    at com.google.gridworks.GridworksServlet.service(GridworksServlet.java:120)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
    at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
    at org.mortbay.servlet.UserAgentFilter.doFilter(UserAgentFilter.java:81)
    at org.mortbay.servlet.GzipFilter.doFilter(GzipFilter.java:132)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
    at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
    at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
    at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
    at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
    at org.mortbay.jetty.Server.handle(Server.java:326)
    at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
    at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:938)
    at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:755)
    at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
    at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
    at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:619)

Powered by Jetty://

Original issue reported on code.google.com by mcnamara.tim@gmail.com on 30 Sep 2010 at 10:33

GoogleCodeExporter commented 8 years ago 
You can allocate more memory to Refine by editing the Refine.ini file* in the 
installation directory.  The line to edit is: #REFINES_MEMORY=1024M

*it appears you're running an older version of Refine, so the file would be 
named Gridworks.ini

Original comment by iainsproat on 30 Sep 2010 at 11:23

GoogleCodeExporter commented 8 years ago 
Possibly could be related to Issue 147

Original comment by iainsproat on 14 Oct 2010 at 4:51

GoogleCodeExporter commented 8 years ago 
I had the same error with a 100mb xls file as well.

Original comment by little...@gmail.com on 12 Nov 2010 at 2:08

Attachments:

GoogleCodeExporter commented 8 years ago 
littlemog - check the FAQ 
http://code.google.com/p/google-refine/wiki/FaqAllocateMoreMemory on how to 
increase memory for Google Refine.

Original comment by thadguidry on 12 Nov 2010 at 2:40

GoogleCodeExporter commented 8 years ago 
While allocating more memory works fine, it would be really nice if the system 
handled this better. Throwing an OOM exception and 500ing is a pretty bad 
experience.

Original comment by andy.war...@gmail.com on 20 Nov 2010 at 8:55