Open GoogleCodeExporter opened 8 years ago
What errors, if any, are being reported on the console that the Refine server
was started from?
There are no hardcoded limits. It's much more likely that you're hitting an
error which isn't being handled and reported properly.
Original comment by tfmorris
on 8 Feb 2011 at 4:23
Does the console keep an error log by chance or do I need to scroll through it
immediately when it fails?
Original comment by vinnygof...@gmail.com
on 8 Feb 2011 at 5:13
I don't know yet if Refine keeps any kind of console log history but I was
watching the console when it failed during my last test. At line "16:24:18.429"
is when the reconcile stopped, the percentage completed window disappeared and
no error messages popped up.
Console window contents attached.
Original comment by vinnygof...@gmail.com
on 9 Feb 2011 at 4:11
Attachments:
I don't think any logging is done to a file, although a little log4j
configuration magic could probably change that pretty easily.
At first glance, it looks to me like it thinks things completed successfully.
Anything special about the data near where the operation stops?
Unless David has some other ideas, it'll probably take getting a copy of the
data file and the operation(s) that you are attempting to be able to debug this.
Original comment by tfmorris
on 9 Feb 2011 at 4:22
How can I tell what row the operation stopped at?
Original comment by vinnygof...@gmail.com
on 9 Feb 2011 at 4:49
I don't see anything in the log which would indicate, but I thought you had an
idea since you said "about halfway through."
Original comment by tfmorris
on 9 Feb 2011 at 5:36
I was just going according to the status box at the top of the page during the
reconcile, it was at 53% completed when it suddenly disappeared. The file I was
working with at that time was about 20,000 rows and was hoping it would have
logged where it stopped for debugging purposes.
I'm currently trying to apply the exact same JSON code to a project containing
141,160 rows. This is the largest file I have so I'm curious to see if it
successfully completes.
Thanks for your help with this.
Original comment by vinnygof...@gmail.com
on 9 Feb 2011 at 6:45
Hmm....I'm wondering if he's not having a div problem with Chrome 9 as in Issue
102 ? And that it actually was continuing the reconcile process, but he didn't
know it ? I typically have just looked and followed along watching in the
Command window to be extra sure.
Original comment by thadguidry
on 9 Feb 2011 at 7:12
When the reconciles fail the command window stays open and there isn't any
activity.
I needed to stop the process on my biggest file (141,160). It got to about 24%
complete however.
I've decided to be more methodical with my approach. I have 15 files (projects)
that range from 123 rows to 141,160 rows. I'm going to apply the JSON code to
the smallest first and work my way up. I've attached the JSON code I've been
working with.
Below is what I've done so far, how long it took to complete and if completed
successfully:
Project containing 123 rows - successful in 4 minutes
Project containing 1,171 rows - successful in 15 minutes
Project containing 4,797 rows - successful in 1 hour and 5 minutes
Project containing 5,403 rows - successful in 1 hour and 14 minutes
Project containing 9,716 rows - FAILED at 2 hours and 5 minutes (console
message attached)
Project containing 12,493 row - currently running (started at 11:54 am -
2/10/11)
So far the project size really has had a noticeable difference on the resources
used. CPU usage peaks at about 20% and memory usage at about 1.5 GB.
Thanks again guys.
Original comment by vinnygof...@gmail.com
on 10 Feb 2011 at 5:05
Attachments:
Thanks for the additional info. The failure looks to be related to HTTP 500
Server Errors on the reconcile operation. I wonder if a) they're getting
retried by Refine and b) this is a transient or hard error.
The processing times strike me as being very long. I'm guessing that most of
this is network/reconciliation latency, but still an hour to reconcile 10,000
items (5,000 rows x 2 columns) seems like a lot.
What Java heap size are you using? If you've got the memory available, you
should make it generous enough that it's not an issue.
Original comment by tfmorris
on 10 Feb 2011 at 7:06
The project I was running (12,493 rows) just stopped at about 2hours and
3minutes into it. I've attached the contents of the console window.
The Java heap size is whatever default was for the install. I've got 4GB of
memory and only about 1.5 GB are being used during the processing, so what do
you suggest I raise it to? Also, where is that setting changed?
Thanks.
Original comment by vinnygof...@gmail.com
on 10 Feb 2011 at 7:20
Attachments:
The default heap size is 1024M (1 GB). Try going up to 1536 or 2048, depending
on what other memory demands you have on the system.
You can find instructions here
http://code.google.com/p/google-refine/wiki/FaqAllocateMoreMemory
It definitely should be reporting the out of memory error though (presuming
that's what's happening), so we'll need to look into that more.
Original comment by tfmorris
on 10 Feb 2011 at 8:04
I installed the java development kit and increased the heap size to 2048M. I
tried to apply the saved operation to a file that been failing for me
containing 9,716 rows and the reconciliation stopped again about 2 hours in on
the "Title" column. I've attached the contents of the console window, the time
of failure was 17:01:58.144.
Thanks.
Original comment by vinnygof...@gmail.com
on 12 Feb 2011 at 2:16
Attachments:
Are there any know characters within a data set that can cause issues?
Original comment by vinnygof...@gmail.com
on 12 Feb 2011 at 2:19
tfmorris, looking back at my code, the run() method doesn't catch and log any
exception
http://code.google.com/p/google-refine/source/browse/trunk/main/src/com/google/r
efine/operations/recon/ReconOperation.java
So memory exceptions would go unnoticed.
vinnygoffin, if you're comfortable with Java code, would you mind trying to
modify that file locally, wrap the run() method's content in a try catch and
see if any exception shows up? Thank you for your patience.
Original comment by dfhu...@gmail.com
on 12 Feb 2011 at 4:58
Hi,
I'm not versed in Java so unfortunately i'm not sure how to do what your
asking, sorry.
I have continued with some other testing using a file named(RED425 with 9,716
rows)this consistently stops and doesn't complete. I've attached the most
recent console window contents of this failure.
Trying to see if it was a piece of data within the file that is giving me the
headaches, I split the file into 2 pieces (RED425-1 with 4,858 rows and
RED425-2 also with 4,858 rows).
Both completed fine, so I'm not thinking it's a weird character or bad data or
something like that. Would you agree? I've attached these files as well, in
case you interested.
Original comment by vinnygof...@gmail.com
on 15 Feb 2011 at 8:52
Attachments:
Thank you for trying again. I think you're right: it's not a weird character
that's causing problems. It looks to me--and I'm not surprised--that the
reconciliation service just got overloaded. So for now, is it OK for you to
tend the reconciliation process somewhat manually (e.g., splitting it up into
batches)? By the way, you could do this in a single project by creating a
custom facet on any column with the expression
ceil(row.index / 4000)
Substitute 4000 for however large a batch you want. Select one value at a time
in the facet and invoke the reconcile command.
Original comment by dfhu...@gmail.com
on 15 Feb 2011 at 10:53
Yup, that's what I'm going to do. I'll split the files into 5000 row segments.
Thanks for all your help with this.
Original comment by vinnygof...@gmail.com
on 17 Feb 2011 at 3:37
Hi again,
One more quick question.
Is there a way to automatically invoke the ceil(row.index / 4000) command in
succession?
I was thinking of seeing if I could have it somehow run it on the first 4000
rows, stop, then continue on the next 4000 and so on until complete.
Thanks.
Original comment by vinnygof...@gmail.com
on 1 Mar 2011 at 3:09
It's not really possible to say for sure from the information provided, but
this could be related to issue 440 which would effectively prevent completion
of any single operation (one step in the undo history) which took longer than
one hour.
The fix for that is in SVN and will be included in the upcoming v2.5 release.
Original comment by tfmorris
on 21 Oct 2011 at 4:09
Original issue reported on code.google.com by
vinnygof...@gmail.com
on 8 Feb 2011 at 3:54