search

janelia-flyem / dvid

Distributed, Versioned, Image-oriented Dataservice
http://dvid.io
Other
197 stars 33 forks source link

Ingesting dataset where the dimension is not a multiple of block size #322

Closed ak0ska closed 5 years ago

ak0ska commented 5 years ago

Hello,

I was experimenting with ingesting a dataset into DVID using just the REST API and I was trying the following two endpoints:

Both seem to require that the data sent is always a full block size, but this is not the case when when the dimension of the dataset is not a multiple of blocksize. If I have blocksize 64x64x64 and a dataset of 150x150x150 then the edges of the dataset are not going to make full blocks.

Is it expected that the user "pads" the dataset so its dimension will be a multiple of blocksize?

Thank you!

Akos

DocSavage commented 5 years ago

Akos, yes, padding the border with zeros or whatever is the background voxel data is the desired approach.

Early on, DVID did allow mutations using non-block dimensions. It's not a problem for single DVID servers since we could use a mutex and prevent race conditions. But we thought it better to reinforce the idea that the underlying storage was really these blocks or chunks and with a little thought upfront in how the REST APIs were called, the actual operations would be much more efficient. We think it will also be more manageable if we eventually support distributed DVID servers for mutating data. (We already use a DVD server per proofreader for our immutable grayscale since we don't worry about POSTs.)

ak0ska commented 5 years ago

I see. Thank you for the clarification!