search

janelia-flyem / dvid

Distributed, Versioned, Image-oriented Dataservice
http://dvid.io
Other
195 stars 33 forks source link

Need GET /indices #309

Closed stuarteberg closed 5 years ago

stuarteberg commented 5 years ago

I can post many label indexes at once via POST .../indexes, but I can't GET many label indexes at once. When copying all label indexes from one database to another, fetching them one at a time is slow. The copy I'm performing right now (using 1-by-1 GET and batched POST) is going to take 8 hours, whereas with batch GET I think it ought to take ~1 hour or less.

DocSavage commented 5 years ago

I was thinking we would've wanted a streaming GET but error conditions like too large a return payload (> 2 GB) can't really be returned once data is coming back in HTTP. So I suggest you specify a JSON array of a number of labels and I return a protobuf for repeated label indices like we currently do for POST /indices. If you ask for too many labels and the cumulative size is greater than some threshold, I return an error. Sound OK?

stuarteberg commented 5 years ago

Sure