search

FePhyFoFum / otu

nexson / git interactions
0 stars 0 forks source link

memory on indexing #3

Closed blackrim closed 10 years ago

blackrim commented 10 years ago

need to do some transactions during the indexing. my laptop (4GB RAM) got overwhelmed. it finished, but wouldn't take much more.

chinchliff commented 10 years ago

Sounds good. Will do that today

On Thursday, September 12, 2013, Stephen Smith wrote:

need to do some transactions during the indexing. my laptop (4GB RAM) got overwhelmed. it finished, but wouldn't take much more.

— Reply to this email directly or view it on GitHubhttps://github.com/FePhyFoFum/otu/issues/3 .

chinchliff commented 10 years ago

Hmmm. Each study is indexed individually, so it is probably not an issue we can solve with transactions. I think what might be happening is that your neo4j was trying to do too many things at once. Right now it just uses a timer to stagger the indexing request, which is currently set to 10 requests per second. This works ok on my machines but it is definitely not something we can count on. I am thinking a recursive callback would probably let us do the indexing one study at a time.

chinchliff commented 10 years ago

Recursive indexing is set up. We will see how that works...