Significant lag reading/writing data to ndstore

neurodata / ndstore

code for storing neurodata images and image annotations

http://neurodata.io

Apache License 2.0

38 stars 12 forks source link

Significant lag reading/writing data to ndstore #332

Closed kkutten1 closed 8 years ago

kkutten1 commented 8 years ago

Steps to reproduce:

wget http://openconnecto.me/ocp/ca/kharris15apical/info/

Result: The command hangs for 5 to 10+ seconds at "HTTP request sent, awaiting response..."
Expected behavior: Server should respond immediately

kkutten1 commented 8 years ago

This problem also means that uploading images is slow. A 1GB volume has ~1 billion voxels and might be broken up into 1,000 blocks by ndio. If there's a 10 second lag in uploading every cube, the upload might take 10,000 seconds (nearly 3 hours)

jovo commented 8 years ago

@kunallillaney i like ndtoe we should have such a thing :)

kunallillaney commented 8 years ago

@jovo and what would this "toe" do?

jovo commented 8 years ago

the possibilities are limitless!

On Mon, Aug 8, 2016 at 11:42 AM, Kunal Lillaney notifications@github.com wrote:

@jovo https://github.com/jovo and what would this "toe" do?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/neurodata/ndstore/issues/332#issuecomment-238278012, or mute the thread https://github.com/notifications/unsubscribe-auth/AACjcpAEih1CXxQAm5477UB_9W7nEziMks5qd05agaJpZM4JfBwV .

the glass is all full: half water, half air. neurodata.io, jovo calendar https://calendar.google.com/calendar/embed?src=joshuav%40gmail.com&ctz=America/New_York

kunallillaney commented 8 years ago

@kkutten1 The issue was with Hopkins networking and this has now been resolved. If your reads and writes are still slow, please check if they are cube aligned. Also use the latest version of ndio which has some performance improvements, if you are on an older version.