search

OpenPIV / openpiv-python

OpenPIV is an open source Particle Image Velocimetry analysis software written in Python and Cython
http://www.openpiv.net
GNU General Public License v3.0
229 stars 141 forks source link

How well openpiv-python deals with the upcoming huge PIV cameras? #209

Open alexlib opened 2 years ago

alexlib commented 2 years ago

Is your feature request related to a problem? Please describe. Here is a link to the issue on PIVLab: https://github.com/Shrediquette/PIVlab/issues/68#issue-973313073

The message is about differences in RAM and disk usage when 25 megapixel images are analysed for PIV. I wonder how does openpiv-python performs and where are the bottlenecks.

Describe the solution you'd like A volunteer is needed to create a good test environment based on debuggers and memory profilers, create different increasing size PIV test images up to 100 Mp and test performance. The main goal is to find the bottlenecks and understand them, looking for possible future enhancements of the code.

ErichZimmer commented 2 years ago

FYI, scipy FFTs operate better with dtype of float32 compared to other dtypes because some reason scipy processes in float64 on everything except float32. Additionally, it is faster by 10-112% and takes half as much RAM (good for multiprocessing).

alexlib commented 2 years ago

Good point. We need to check everywhere what dtype is used.

ErichZimmer commented 2 years ago

On possible bottlenecks, scipy/numpy's use of vectorized FFT algorithms could be one due to its high memory usage. A general rule of thumb is 1GB of free RAM per core to be used mostly eliminates it, but it still could be an issue for images >3 MP with window sizes <32 pixels and using multi/parallel-processing. This seems to be a python/matlab issue as nearly all c++ softwares using FFTW + OpenMP use very little memory and are orders of magnitude faster on large images. Another possible bottleneck is local median validation and second peak searching algorithms as it can take some time when there is a large amount of vectors (>50,000).

alexlib commented 2 years ago

Probably you are right. We need some profiling tools to learn the issue and prepare some heavy load tests. Meanwhile it is only a wishful future extension.

ErichZimmer commented 2 years ago

Have we tried chunking the 3D stack of arrays and use the multiprocessing.Process module for parallel processing? This is what Fluere uses and it's as fast as some commercial softwares. It also lowers the RAM requirements.

alexlib commented 2 years ago

Windef has some multiprocessing use. Not sure that it is more than that. Please try

From: Erich Zimmer @.> Sent: Monday, November 29, 2021 7:56:45 PM To: OpenPIV/openpiv-python @.> Cc: Alex Liberzon @.>; Author @.> Subject: Re: [OpenPIV/openpiv-python] How well openpiv-python deals with the upcoming huge PIV cameras? (#209)

Have we tried chunking the 3D stack of arrays and use the multiprocessing.Process module for parallel processing? This is what Fluere uses and it's as fast as some commercial softwares. It also lowers the RAM requirements.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/OpenPIV/openpiv-python/issues/209#issuecomment-981875222, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAFWMZU57FWIDCYGTIV5VN3UOO5F3ANCNFSM5CLKOV5Q.

ErichZimmer commented 2 years ago

For the testing of large images, my personal GUI now has a synthetic image generator based on synimagegen and PIVlab. It should give us a good idea of the bottlenecks. Additionally, if you like, I could make it a standalone GUI.