search

pattersonkl / google-refine

Automatically exported from code.google.com/p/google-refine
0 stars 0 forks source link

Can't load large datasets #270

Closed GoogleCodeExporter closed 8 years ago

GoogleCodeExporter commented 8 years ago 
Can't load large datasets only loads 90,539 rows of 500k + dataset

google-refine-2.0-r1836

Original issue reported on code.google.com by jonathan...@gmail.com on 1 Dec 2010 at 8:47

GoogleCodeExporter commented 8 years ago 
Despite this it looks like it processed up to about 300k records in the file, 
but the remainder is unchanged by my edits

Original comment by jonathan...@gmail.com on 1 Dec 2010 at 9:04

GoogleCodeExporter commented 8 years ago 
Jonathan, are you seeing any Java exception in the console? What's the 
"symptom" of "can't load large datasets"? What do you observe?

Have you tried this?
http://code.google.com/p/google-refine/wiki/FaqAllocateMoreMemory

Original comment by dfhu...@gmail.com on 2 Dec 2010 at 6:49

GoogleCodeExporter commented 8 years ago 
no exceptions

symptoms: the file opens eventually, but shows 90,539 rows at the top of the 
screen, I make changes using "edit cells - cluster and edit" and as stated in 
my 2nd post I can see in the exported file that the edits have only effected 
the top half of the file.

Original comment by jonathan...@gmail.com on 3 Dec 2010 at 6:11

GoogleCodeExporter commented 8 years ago 
What format is the file in?  (is it CSV?)

Is there anything special about the 90,540th line?

Are you able to upload the data somewhere (you can attach it to the bug report 
if you don't mind the data being made publicly visible) so we can take a look 
at what's happening?

Original comment by iainsproat on 3 Dec 2010 at 9:07

GoogleCodeExporter commented 8 years ago 
I also hoped to manipulate large datasets with refine and failed, but my 
testing happened a few months ago, i dont have very many details.  I can tell 
you that increasing memory size allocated didnt fix the problem, and i was 
attempting to load 200mb files.

Original comment by jimfli...@gmail.com on 14 Mar 2011 at 2:38

GoogleCodeExporter commented 8 years ago

Original comment by tfmorris on 18 Sep 2012 at 7:35