Large data requests

[wrgr]

There are several really great potential collaborators who would like to be able to pull down entire datasets (especially Kasthuri11 and annotations). Currently, the recommendation is to use ndio, but server latency can make this difficult, and even in the best case, is much slower (and intensive to the backend) than make prepopulated downloads.

Doing this client side is difficult because then we have to curate a bunch of different datasets ourselves and those humans are already pretty swamped! And we've had trouble with them going stale.

Could we make a channel export in the near future? What's the best near-term advice for folks? Could use some wisdom and help. @j6k4m8 @kunallillaney @jovo @Aeusman @randalburns

[jovo]

in the cloud data model, the slices live in S3, so there is no need for an 'export' function, people can download directly from the bucket if they want.

currently,

http://cloud.neurodata.io/nd/ca/public_tokens/

yields:

[ "blaze1", "kasthuri11" ]

so, in theory, they can get a version right now using that strategy.

note that i could be totally wrong.

On Friday, June 10, 2016, William Gray notifications@github.com wrote:

There are several really great potential collaborators who would like to be able to pull down entire datasets (especially Kasthuri11 and annotations). Currently, the recommendation is to use ndio, but server latency can make this difficult, and even in the best case, is much slower (and intensive to the backend) than make prepopulated downloads.

Doing this client side is difficult because then we have to curate a bunch of different datasets ourselves and those humans are already pretty swamped! And we've had trouble with them going stale.

Could we make a channel export in the near future? What's the best near-term advice for folks? Could use some wisdom and help. @j6k4m8 https://github.com/j6k4m8 @kunallillaney https://github.com/kunallillaney @jovo https://github.com/jovo @Aeusman https://github.com/Aeusman @randalburns https://github.com/randalburns

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/neurodata/ndstore/issues/285, or mute the thread https://github.com/notifications/unsubscribe/AACjctk5HQNCA8-6Z4R79dH4FEC-AJgUks5qKWLJgaJpZM4Iy7_N .

the glass is all full: half water, half air. neurodata.io, jovo calendar https://calendar.google.com/calendar/embed?src=joshuav%40gmail.com&ctz=America/New_York

[wrgr]

yea - this is only image data in tiles/cuboids, so not in a format easy for new folks to use. (e.g., tiff/png)

[kunallillaney]

@willgray An email would help me help others. :)

[jovo]

better yet: anybody who wants anything from kunal please encourage them to create an issue for their request, and we will put it in the priority queue.

On Tue, Jun 28, 2016 at 2:19 PM, Kunal Lillaney notifications@github.com wrote:

@willgray https://github.com/willgray An email would help me help others. :)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/neurodata/ndstore/issues/285#issuecomment-229136477, or mute the thread https://github.com/notifications/unsubscribe/AACjcrtUJdoEAY_z7GRS_8_uywRCCFNlks5qQWWXgaJpZM4Iy7_N .

the glass is all full: half water, half air. neurodata.io, jovo calendar https://calendar.google.com/calendar/embed?src=joshuav%40gmail.com&ctz=America/New_York

[wrgr]

done. i encouraged issue creation and forwarded to kunal. to my knowledge i am now off the critical path on these requests, or at least not the bottleneck. :)

[kunallillaney]

Thanks.