search

natverse / neuprintr

R client utilities for interacting with the neuPrint connectome analysis service
http://natverse.org/neuprintr
3 stars 3 forks source link

break down queries that are too large #89

Closed romainFr closed 4 years ago

romainFr commented 4 years ago

Really large queries result in a timeout error from the server. It should be possible to break down those and rebuild the results. Either through a tryCatch or by setting thresholds on query sizes for various functions.

schlegelp commented 4 years ago

Batching queries is one way of mitigating time-outs but Stuart also found that the neoj4 planner appears to misbehave when using large lists of bodyIds. You can trick the planner into behaving by changing the query. See here:

https://github.com/connectome-neuprint/neuprint-python/issues/7

romainFr commented 4 years ago

Progress made in #98 #99 and #100 . Any other places we'd like to chunk ?

jefferis commented 4 years ago

Thanks @romainFr. I think that's good for now. See also 7921b21f67e40e59af971479058914089c4ba4c4 and 4c6b814d6ab56c983ca0c1bd2659d5a291859e71 for some points to bear in mind if we add chunking to any more functions.