search

NSLS-II-CSX / csxtools

Useful python tools for CSX (23-ID)
http://nsls-ii-csx.github.io/csxtools
Other
4 stars 13 forks source link

Add C Modules for fast ccd image manipulation #3

Closed stuwilkins closed 8 years ago

danielballan commented 8 years ago

Neat! Even if np.rot90 and np.subtract can be made to be parallel by default (which would be, I think, a more straightforward approach if it can be done) this pattern will be useful for other more specific image processing functions.

stuwilkins commented 8 years ago

Hi @danielballan . I found some curious things. ''np.rot90'' only changes how you view the array in memory, not actually rotate it. I think that can lead to some performance issues unless you are accessing elements randomly.

When I played with this, the order in which you access elements in the array matters. That was one of the biggest things that changed the execution time which AFIK it is due to either hitting the memory cache or not.

This does point to the fact that we may want to be sensitive to how these image stacks are actually shown in memory, which might depend on if you are pixel centric or image centric.

I agree that standard libraries are usually better. I mostly did this to learn OpenMP and play. Enjoyed actually coding something for a change!

For the rest the speed up is quite impressive. When you have time I wanted to touch base with you and @tacaswell about the return format of ''get_images''

danielballan commented 8 years ago

That's good to know. I actually expected that but had not checked it myself. A numpy array is a set of instructions for interpreting a contiguous block of memory. If any operation can be accomplished by modifying those instructions -- not rearranging the bytes -- it probably should be. However, that can make later computations seem to be take longer, as they may be making less-than-optimal strides through memory.

Yes let's talk about get_images. I have some ideas for improving this -- ideas I can implement very quickly -- but I'd like some input from you and other experienced users before I commit to something.

stuwilkins commented 8 years ago

Thanks @danielballan I could see a number of different options I would like to see for our use cases.

danielballan commented 8 years ago

I've got an ffmpeg conda package on my channel (also on the soft-matter channel) if that would help.

tacaswell commented 8 years ago

This code all assumes c-ordered arrays, that should probably be enforced during the input -> numpy conversion process.

stuwilkins commented 8 years ago

Hi @tacaswell. C ordered arrays are enforced with the macro NPY_ARRAY_IN_ARRAY that is a short macro for many options including enforcing c-ordered.

stuwilkins commented 8 years ago

Merging to continue, Attn @vivekthampy. Note travis is broken due to dataportal / data broker issues.