william-silversmith
commented
5 years ago Thanks for investigating this issue! I've been working on several libraries and the most dire issue I've been faced with, more than the difficulty of any algorithm, has been the variation in strides, C, and Fortran ordering in arrays. Thank you for the tip about ndarray.strides, I didn't know about it.
Some of the most recent work I've done on this issue is located here, where I rely on the user to specify C or Fortran order.
cdef int sx = data.shape[2]
cdef int sy = data.shape[1]
cdef int sz = data.shape[0]
cdef float ax = anisotropy[2]
cdef float ay = anisotropy[1]
cdef float az = anisotropy[0]
if order == 'F':
sx, sy, sz = sz, sy, sx
ax = anisotropy[0]
ay = anisotropy[1]
az = anisotropy[2]This is in part because not only is there XYZC vs CZYX in the array ordering, numpy's indexing is [ row, col, depth ] which could be considered YXZ. Additionally, a programmer can simply imagine within the hermetic seal of their brain an array to be in a different orientation and wish to process it that way (or actually address the array as ZYX when the underlying memory appears to by XYZ).
Amusingly, as you noted, the "C" orientation that numpy outputs is in what is usually termed "Fortran" order, while "F" orientation outputs bytes that are ordinarily considered to be "C" order.
x = np.zeros((2,2,2), dtype=np.uint8, order='C')
x[0,0,0] = 1
x[1,0,0] = 2
x[0,1,0] = 3
x[1,1,0] = 4
x[0,0,1] = 5
x[1,0,1] = 6
x[0,1,1] = 7
x[1,1,1] = 8
print(x.tobytes('C'))
print(x.tobytes('F'))b'\x01\x05\x03\x07\x02\x06\x04\x08'
b'\x01\x02\x03\x04\x05\x06\x07\x08'I think in the case of uncorrelated (or weakly correlated in the case of the anisotropic data my lab uses) 2D arrays, we could add a flag image_stack=True to the compress function that would invert the third and fourth dimension iff the given third dimension is > 1 and the fourth dimension is 1.
shangmu
lindstro
First, a big thanks for making these fpzip wrappers available! Skimming the code and seeing user comments about the need to transpose arrays, I wanted to get to the root cause of this issue.
My concerns are with this piece of code:
fpz_ptr[0].nx = data.shape[0] fpz_ptr[0].ny = data.shape[1] fpz_ptr[0].nz = data.shape[2] fpz_ptr[0].nf = data.shape[3]
fpzip expects an array layout with x varying faster than y, y varying faster than z, and z varying faster than f. In other words, nf 3D arrays of dimensions nx ny nz should have this C layout:
float array[nf][nz][ny][nx];
AFAIK, shape[3] is the fastest varying dimension in a numpy ndarray; shape[0] is the slowest varying dimension. If the array is not reshaped, this will result in the data being interpreted incorrectly, leading to poor compression.
Furthermore, fpzip assumes that the data is correlated along x, y, and z, but not along f (nf denotes the number of independent fields/variables/components). Aside from this, fpzip handles single (nf = 1) or multiple (nf > 1) arrays that may be 1D (nx > 1, ny = nz = 1), 2D (nx > 1, ny > 1, nz = 1), or 3D (nx > 1, ny > 1, nz > 1).
Finally, if the ndarray data to be compressed is a stack of F 2D images of dimensions M * N (with M varying fastest), then some remapping of dimensions would be needed so that nx = M, ny = N, nz = 1, nf = F. This would instruct fpzip to make F separate calls to compress 2D arrays. As is, this code would be invoked
if len(data.shape) == 3: data = data[:,:,:, np.newaxis ]
which results in correct behavior for truly 3D data (modulo transposition) but incorrect behavior in case of a stack of unrelated 2D arrays.
The caller would need to know how to reshape the ndarray to make it 4D with the expected layout.
Absent a more elaborate API that allows specifying which dimensions are correlated and what the underlying layout of the array is, it would seem best to map the last ndarray dimension to nx, the second to last to ny, etc. A more robust solution would be to examine the ndarray strides to determine how to map the dimensions. This still does not address the issue of uncorrelated dimensions.