search

calvez / xcoaitoolkit

Automatically exported from code.google.com/p/xcoaitoolkit
0 stars 0 forks source link

Harvest fail, out of memory #25

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
Test conducted 8/3 and 8/4 resulted in an MST harvest failing due to the 
OAI Toolkit throwing an out of memory error.   Initial data leads 
developers to believe that the long running harvest was "stomped" for some 
reason when the daily chron job started in the middle of the night.

Possible areas to explore are true running out of memory, configure and 
managing the memory usage better by process, etc.   

One possibility to consider is that the job failed due in some way to the 
insertion of new records into the repository by the convert and load 
process when a harvest was already underway.   Make sure time stamps are 
addressed appropriately to handle these scenarios.

Original issue reported on code.google.com by rc...@library.rochester.edu on 5 Aug 2009 at 4:39

GoogleCodeExporter commented 9 years ago

Original comment by rc...@library.rochester.edu on 6 Aug 2009 at 5:31

GoogleCodeExporter commented 9 years ago 
As discussed in meeting today, need to derive a plan for this, more than just 
adding
more memory.   We need to have some directions created on how to parse out and 
use
the available memory so that jobs don't just fail.

Also discussed that we need a plan for how to hand different scenarios, such as 
the
number of harvests the OAI toolkit should respond to.  For example, if multiple
harvest request come in at once, does the OAI Toolkit respond to all of them, 
just
perhaps more slowly, or do we set a number based on certain conditions.

Original comment by rc...@library.rochester.edu on 10 Aug 2009 at 5:31

GoogleCodeExporter commented 9 years ago 
Shrey will write the results and any advice in here and in the documentation re.
memory allocation and data set size. More testing for the handling of different
scenarios as mentioned above will need to be done in the future.

Original comment by rc...@library.rochester.edu on 18 Feb 2010 at 4:44

GoogleCodeExporter commented 9 years ago 
 Per C Delis, no longer an issue after rev. 145

Original comment by rc...@library.rochester.edu on 11 May 2011 at 8:23