New GUI idea based on H5Py

ErichZimmer commented 3 years ago

Background/Issue

The current GUI stores data in separate files which can make it hard to do more thorough data processing. To combat this, an already suggested solution was to store all results in a single dictionary of the dataset and export the results in a manner the user deems sufficient. However, on large processing sessions (>60,000 images), the GUI can become quite slow especially on lower performing laptops. Furthermore, the performance of the GUI starts to decrease with these large sessions. This can cause a disruption to efficient work flows and an increase in glitches (mostly applies lower performing computers).

Proposed solution

After exploring different ways of storing huge amounts of data, H5Py was found to perform pretty well even on underperforming computers (e.g. my laptop 😢). When properly configured, most data is stored in the hard drive, leaving RAM mostly unused unlike dictionary style designs. Additionally, the structuring of an HDF5 file makes it very simple to load specific sections of data/results which has its advantages. Taking advantage of these features, the HDF5 file is structured like the following;

session: The main file which everything is stored in
session\images: Group containing all image filenames and frames for the GUI
session\images\files_a: Dataset of full path lengths of A frames
session\images\files_b: Dataset of full path lengths of B frames
session\images\frames: Dataset of Frame names used for GUI frame list and file names for results
session\results: Group containing all results
session\results\frame_...: Group containing components of a result
session\results\frame_...\x: Dataset containing x component (there would be a dataset for each component in the group)
session\results\frame...Attrs: Attributes containing processing time, ROI and mask coordinates for further processing and display

Possible downfalls

It takes somewhat careful planning to get H5Py to work as needed. If structured wrong, the advantages of a HDF5 based storage format is waivered and performances are similar to that of a dictionary style storage format. However, this requires some major mistakes and shouldn't really happen.
Extra dependency (H5Py)
To get multiprocessing to work, there will most likely be an extra dependency

PS, I'm back 😁 (got medically discharged from an injury) and ready to relearn everything/hopefully not be so ill-informed on testing methods like I was back then -_-. Additionally, your inputs on using HDF5 or others for storage would be helpful for further research and designing.

alexlib commented 3 years ago

@ErichZimmer is back :) @eguvep - what do you think?

alexlib commented 3 years ago

@ErichZimmer can you please take a look at NetCDF files? the xarray project and our sister pivpy project uses it (as a competitor for HDF5) and xarray provides some great extension over pandas to get easy things e.g. data.piv.average or data.piv.mean(dim='t')

ErichZimmer commented 2 years ago

@alexlib I made a mostly functional GUI built around HDF5 and is parallel capable through some work arounds with current limitations on dependencies. So far, the only extra dependency for this GUI is H5Py. I'll try other styles to see their performances, but HDF5 is performing pretty well so far...

Some GUI screenshots: Prototype_HDF5_GUI

extensive_preproc

advanced_settings

PS, please ignore spelling errors as I am low on time and my mobile hotspot won't let me edit previous posts for some reason..

eguvep commented 2 years ago

Dear Erich!

I am very happy to read that you are back and it is great to see your immediate productive postings! Besides the performance advantage, it would be great to put the whole parameter object into the data-set. When loading a data-set, the associated parameters should also be loaded directly. This would be a big advantage for people who are switching between data-sets or when reevaluating older data. What do you think, @ErichZimmer?

Another thing – mentioned in our previous discusson – is compatibility. Our simple and stupid CSV files are the lowest common denominator with almost every other code (like awk or other command line stuff; they are even human readable) and we follow the UNIX philosophy by using text files. I would strongly vote for an CSV import and export option to not destroy this compatibility. Or are there any command line tools for extracting HDF5 data (I am a novice in HDF5)?

Can we be shure, that changes in the HDF5 code do not break the GUI? As far as I can see, HDF5 seems to be fairly mature, right?

I had a quick look at the other data-formats, @alexlib. And I worked with NetCDF before (there is a JPIV extension for generating synthetic PIV images based on that format and the SIG project). It is hard to tell – if not impossible – which format is best. HD5 seems to be slightly more flexible, so it seems possible to put really everything into the files. Everything that is hard to decide, can be decided randomly in my opinion ;-) So lets give HDF5 a try!

Regards!

Peter

alexlib commented 2 years ago

I think we need to split the two topics: A) whether we want a single database (a single binary or ASCII file, or a group of files that are bonded together) for everything - probably we do. PIVLAb has a MAT file that contains all the session details and then if you do export, exports multiple data files. BTW, MAT files are HDF5 files AFAIK. B) Choice of the file format. For performance the binary formats are obviously the solution, although now pandas have very fast CSV reader that I believe can be our solution. If we work with pandas - we get also built in HDF5 support (through h5py) and we can also convert it to xarray format.

My suggestion is to try pandas with CSV first and then if the performance is not sufficient, keep working with pandas and HDF5.

Regarding HDF5 - they're fast and flexible, and there are some things like HDFView or h5dump that help to see their content.

NetCDF - the only benefit is the straightforward continuation and connection with the pivpy - after all, we probably want to have GUI also for the post-processing, colorful images, vorticity, strain, etc.

eguvep commented 2 years ago

For clarity, the PIV database could even be a separate project. A PIV-database object could provide methods

for creating such an object from a set of images, CSV-files and settings,
reading it and providing the data in different formats, and
exporting the data. We could than use this object in the GUI, but also in Jupyter notebooks or other places.

alexlib commented 2 years ago

Great idea.

What is the structure of this project? For pivpy we use xarray DataSet - it's a pandas-dataframe-on-steroids with metadata attached to it. I didn't find another similar solution that provides me an option to average along a "named" dimension and have a lot of underlying mechanics of all kind of possible numerical operators.

ErichZimmer commented 2 years ago

Do you know any way of chunking PIV data so the user doesn't have to load too much stuff into memory? The reason why I chose a HDF5 format was because at most, there is only two complete PIV results (2 frames) loaded into memory and the rest is stored on the hard drive. In my case, I analyzed ~3,000 images to get ~3000 results of which I only have to load and work on one result at a time on the GUI. Since I am not using chunking, the results can have different sizes for whatever reason (different size windowing/overlap). Perhaps, NetCDF partnered up with xarray would be the way to go, but for now, I'll stick with a HDF5 format until I learn more about NetCDF (I like xarray a lot though so I'll try..)

By the way, there is an export page on the GUI to export our results in multiple different ways and file types.

alexlib commented 2 years ago

The parallel or chunked reading is not from xarray, but from dask

http://xarray.pydata.org/en/stable/user-guide/dask.html

ErichZimmer commented 2 years ago

Well, this got a little more confusing... But I'll see what I can do as the GUI is currently made to switch internal formats relatively easy.

For HDF5, the GUI is setup like this: Session: the file that contains everything Session\images: group that contains all image related stuff Session\images\img_list: datasets that contains all images loaded into the GUI Session\images\files_a: dataset of A frames list Session\images\filesb: dataset of B frames list Session\images\frames: dataset of frames list for display Session\images\settings: group for image settings Session\images\settings\frame{i}: dataset of settings for frame i Session\results: group for all results Session\results\frame{i}: results group for frame i Datasets in group: Session\results\frame{i}\xraw: raw x component dataset Session\results\frame{i}\yraw: raw y component dataset Session\results\frame{i}\uraw: raw u component dataset Session\results\frame{i}\vraw: raw v component dataset Session\results\frame{i}\tpraw: raw vector type dataset More will be added once postprocessing is working up to 'standard' Attributes in group; Session\results\frame{i}.attrs['processed']: Boolean if frame was processed Session\results\frame_{i}.attrs['processtime']: Time it took to process frame Session\results\frame{i}.attrs['units']: list of units for GUI purposes Ex: [px, px, px/dt, px/dt] Session\results\frame_{i}.attrs['roipresent']: Boolean if roi is present Session\results\frame{i}.attrs['roicoords']: roi coords in x min, x max, y min, y max Session\results\frame{i}.attrs['maskcoords']: mask coords, set to [] if no mask present Session\results\frame{i}.attrs['window_size']: Used for GUI purposes

PS, nearly flipped out hitting the close with comment button since that thing is HUGE on my phone :(

alexlib commented 2 years ago

some simple facts: NetCDF4 = HDF5 with some extra limitations and it's own API. same performance there is h5netcdf to read/write NetCDF through h5py - so no more dependenices there are newer format called zarr - very good for Python, might be an issue for other languages. does everything that HDF5 does and a bit better for the cloud storage (natural)

which branch are you at @ErichZimmer ? I'll try to see if I understand if there is a point to use xarray + netcdf file for it.

ErichZimmer commented 2 years ago

@alexlib , I haven't uploaded it yet mainly because of lack of internet and having to relearn everything. I plan on making my version of the GUI a separate repository. This is to keep the original GUI and format the same (has some really good perks) and make a separate GUI for more in depth and complicated analysis. @eguvep , What do you think of this idea?

eguvep commented 2 years ago

Hi @ErichZimmer, now (v0.4.11) the new Add-In infrastructure is working (File → Select Add-Ins). Also see the examples in: https://github.com/OpenPIV/openpiv_tk_gui/tree/master/openpivgui/AddIns It should be fairly easy now, to implement H5P as an add-in, instead of a complete separate project. This might lead to a bit of double code (but less double code than in two projects)., Altogether, the code should be more separate and self-consistent within the add-ins than ever before. In this way, it should be possible to have two (or more) GUIs in one. A simple one (e.g. for teaching or beginners), and one ore more with more complex or special features, if the add-ins are selected. What do you think?

ErichZimmer commented 2 years ago

This is a good idea. I'll see what I can do and clean up the code (when I have time) so I can push it up a separate branch for more testing. However, this might take a while along with having to figure out how to use GitHub command line without messing around with the wrong branches. Hopefully, I can do this soon so we can test NetCDF with the advanced GUI and merge to create a nice GUI system/ecosystem. Additionally, I'll work on the present simple GUI as the spatial and temporal pre-processing needs to be updated along with a few other minor things. The add-ins system looks nice :)

Some pictures of the GUI: default_GUI_size

masking

preproc

preview_grid_size

advanced_algs

validation

modify

plotting

test2

You can load external results, settings, or another session external_results

Or export the current figure (not from the basic figure generator), settings, or results. export_figure

All of this comes with the cost of 23 new functions and 3000+ lines of code. However, it is simple since I have very little programming skills by programming standards :P

eguvep commented 2 years ago

That looks very impressive! Regarding the add-in structure, there is also a documentation for a quick start: https://openpiv-tk-gui.readthedocs.io/en/latest/usage.html#add-in-handler

alexlib commented 2 years ago

@ErichZimmer looks very nice. We need to figure out how to merge this into the existing one. Through AddIn or otherwise, by some coding.

ErichZimmer commented 2 years ago

@eguvep Currently, the advanced GUI is not compatible with the add-ins system. However, an option can be selected in the add-ins panel to enable the advanced GUI and all its features. I just have to figure out how the list boxes are going to be coded as they are completely incompatible with the simple GUI. On another note, support for PyFFTW would be nice, but optional, for faster computation speeds for large batches (on a 4 core laptop, I can get about 250 frames processed every 10 minutes with HD images and windowing of 128>64>32>16>12 with 50% overlap on an Intel Pentinum N3710 running at ~2.25 GHz and 1 GB unused RAM) This might be an interesting feature for OpenPIV in the future (but lets stay away from arbitrary windowing :) ). It reintroduces Cython though...

alexlib commented 2 years ago

Computational speed is important but only if we find a way to install the package as simple as it is now. We shall probably first try numba. We can always create a professional version with a different name and installation instructions, eg openpiv-Python-pro code the advanced users

eguvep commented 2 years ago

I think, we could make it compatible by making the code more modular with the help of the add-in system. In my dreams ;-) every user can compos her or his individual GUI by selecting or deselecting the features they need or do not need.

ErichZimmer commented 2 years ago

@alexlib numba works great on the correlation_to_displacement method, which is somewhat slow. However, numba limits the officially supported operating systems to something like windows, MacOS, and Linux. I'm trying a vectorized implementation, however it gives off wonky results.

@eguvep That would be very nice and is a great idea. Some GUI's have good control on what features are needed and what isn't. The advanced GUI is starting to incorporate this in an attempt to make the main code similar enough to the the simpler version, but it is hard to combine the two as there isn't much double coding (the only similar function is initializing the widgets). I probably wasn't thinking about the Add-Ins system until I was done with the main functions.

ErichZimmer commented 2 years ago

This might take a little more work than I thought. When spending my spare time playing around and trying two merge the two projects, The advanced GUI loses some of its functionality due to it being built around H5Py. For instance, an entire extractions rider would not be feasible to implement ins the simple GUI's format. However, I'm trying to incorporate the simple GUI into the advanced one, which seems to go a little more smoother.

If I were to take away the H5Py core, only the scatter plot and histogram plot would remain functional. GUI_statistics

alexlib commented 2 years ago

@eguvep @ErichZimmer please also take a look on the way the GUI for this tracker is arranged. Seems quite simple in terms of uncluttered environment with multiple options. I think this is the same concept as for napari https://www.youtube.com/watch?v=ajEp18opM-Y&list=PL56zLBbX0yZZw18yyMM9tD0fLrobmdbJG&index=1

ErichZimmer commented 2 years ago

@eguvep I toyed around with different merging ideas and found that it may be best to remove h5py and only temporary store data when analyzing the current frame. This would allow the user to find the optimal settings before batch processing. Furthermore, it would make ensemble correlation MUCH more easier to implement and expand for more advanced features. In conclusion, you get most of the benefits of the h5py GUI with no additional dependencies and still have most of GUI features. However, this would require a massive change to the current GUI to something similar to the h5py GUI and the json file would be much larger if manual object masking is used. Calibration might be interesting too.

@alexlib That video is very interesting. Insight 4G (I think that's right) uses an identical system for preprocessing, analysis algorithms, and postprocessing. It is very flexible and can be easily fine tuned for advanced users. With all the new preprocessing algorithms and a surplus of advanced algorithms on the h5py GUI, this will most definitely be helpful. Just have to wait until I have time and finish merging my "mega" GUI that takes advantage of nearly all OpenPIV functions except for 3d PIV. The GUI has a total of 6,000 lines when all files are summed up. That's a lot of work ;)

Regards, Erich

PS, maybe we can create an executable with an embedded python interpreter for users that don't want to bother with installing python. If we do go this route, an executable would have to made in each operating system. Just stay away from the ones that attempt to transcribe python to c or c++. It'll make you lose your hair at the end of the day ;)

ErichZimmer commented 2 years ago

Should we try to keep h5py or netCDF based GUIs? Using them allow for a huge amount of opportunities before exporting files, but at the cost of complexity and some additional computation costs.

alexlib commented 2 years ago

Should we try to keep h5py or netCDF based GUIs? Using them allow for a huge amount of opportunities before exporting files, but at the cost of complexity and some additional computation costs.

I agree that one of those would be great. I think the main part here is a fast I/O and if possible, access from outside of the GUI, e.g. from a Jupyter notebook - allowing interaction with the data from a post-processing package. I do not mind h5py or netcdf - as long as we in the future interface them, i.e. we will add h5data.to_necdf() and netcdfdata._to_h5() later on. @eguvep ?

ErichZimmer commented 2 years ago

access from outside of the GUI FWI, all data stored in the HDF5 file can be accessed from a notebook. I've done it many times when I was still working on the GUI for debugging and more efficient structuring. Postprocessed and other data can also be stored in the HSF5 file and depending on the name of the groups and databases, the GUI can read and process them.

eguvep commented 2 years ago

Should we try to keep h5py or netCDF based GUIs? Using them allow for a huge amount of opportunities before exporting files, but at the cost of complexity and some additional computation costs.

In my point of view, one of the main design-goals of the GUI is simplicity, so that none-programmers can easily understand and contribute. The add-in system is structuring the code even more to make it even more accessible. On the other hand, I see the advantages of an efficient binary file format. Do you really see no way of using h5py in the scope of a plug-in? This would be the most desirable solution, in my opinion.

ErichZimmer commented 2 years ago

With how the advanced GUI is built around h5py, it would not be feasible to incorporate it in any other way. However, with the add-in system, I can get the main features incorporated and the GUI becomes something like Fluere and PIVview. The GUI looks the same as the h5py GUI, except it operates in a similar way to the simple GUI. In the analysis tab, there is a menu to help find the optimal settings which I managed to keep functioning. Here is how the riders are structured.

General -general

preprocessing -transformations -phase separation -temporal filters -spatial filters -exclusions

Analysis -PIV settings/analyze -advavanced settings -ensemble settings -first pass validation -other pass validation -postprocessing -data probe

Postprocessing -validate results -modify results -exclusions

Calibration -calibration

Plotting (still working on this) -vector settings -contour settings -streamline settings -statistics -extractions -preferences

The menus can be toggled with the add-in system and soon, may have individual parameters in the menus toggled with a similar system that @alexlib mentioned is the video he mentioned.

Note, for calibration to work, the units are stored in the .vec (or .txt or .dat) in the third line. The first line stores the GUI and OpenPIV-python version and the second line stores the image filenames processed. This is for debugging and other purposes.

ErichZimmer commented 2 years ago

Additionally, the GUI can extract many derivatives and works well with most files compatible with the pandas read_csv (except for PIVview ASC-II .dat files, which requires some reformatting).

derivatives and extras: vorticity enstrophy shear strain normal strain divergence convergence (questionable) acceleration (questionable) kinetic energy total kinetic energy fluctuations gradient du/dx gradient du/dy gradient dv/dx gradient dv/dy

For convergence and acceleration, I think I messed up on the equation somewhere because the results are not matching the article I read.

Finally, an option could be made to store all files in netCDF, but the GUI won't be able to read it as of yet.

ErichZimmer commented 2 years ago

I think not using h5py is the way to go for the sake of simplicity. H5py had some limitations, especially on arrays of dtype object, and required some crazy out of the box thinking to get working efficiently (Ex: converted arrays to UTF-8 strings and stored them as attributes or .npy files). When removing h5py, ensemble correlation now has advanced features inspired from microPIV articles and chapters.

alexlib commented 2 years ago

@ErichZimmer where is the repo of this new GUI? I think we need to take a closer look - to see whether we need to consider it as another GUI option without forcing it into the existing openpiv_tk_gui - if it's so far from each other.

ErichZimmer commented 2 years ago

@alexlib I'll push it to my fork when I get access to internet and my laptop (hopefully soon).

alexlib commented 2 years ago

@alexlib I'll push it to my fork when I get access to internet and my laptop (hopefully soon).

Thanks. if it's very different from the original source, maybe it's best to push it to another, not a fork, but a new repo?

ErichZimmer commented 2 years ago

The non-h5py GUI is sort of like an extension to the original GUI and with more functions in the main code. If we were to make a separate repository, it would most likely be for the h5py GUI since it doesn't have much double coding compared to the original. I think @eguvep wants a single repository for the GUI, right?

ErichZimmer commented 2 years ago

Going back to the Add-Ins system, I am pretty sure that it unpacks the widgets instead of restarting the GUI. At the very least, it hides the selected menus in the riders.

ErichZimmer commented 2 years ago

I haven't really done much with the GUIs as of late because of work and SARS-CoV-2 delta and mu variants, so I forgot some of the features and changes I made to the original GUI. I wished I motivated my self to get more done last month when I had longer free time. :(

eguvep commented 2 years ago

I think @eguvep wants a single repository for the GUI, right?

If possible, yes. Multiple GUIs will certainly confuse new users. Which one should they use? What are the differences? Should we document and explain every difference? Advancing in time, some people will develop features for one GUI but not for the other. Maintenance of more than one GUI is anyway a nightmare.

This holds at least for GUIs that are using the same technology. We can certainly have a C++ GUI, a Python GUI and a web-based one.

I see three options:

We merge manage to merge the Python based GUIs again.
We keep the original one with the add-in system.
We keep the new one with h5py support.

I would prefer the first option, of course. If this is not possible, it is also fine for me to keep the new GUI with the h5py support. It would be good to review the code and the documentation, first, of course.

Regards!

eguvep commented 2 years ago

What I mean is, I would rather drop development on the original GUI, than confusing users with two GUIs.

alexlib commented 2 years ago

Agree completely. Only one Python GUI that OpenPIV will support officially. Let’s take a closer look at the h5py version and see how to move on. Of course we cannot prevent competition or tell others what to do, so if there are repositories that fork and change our work, we would only be able to decide if we want to incorporate it. Open source communities divide, split and merge all the time :)

On Wed, 15 Sep 2021 at 9:12 Peter Vennemann @.***> wrote:

What I mean is, I would rather drop development on the original GUI, than confusing users with two GUIs.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/OpenPIV/openpiv_tk_gui/issues/33#issuecomment-919729984, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFWMZWXO55ZLAE7KRF4LNLUCA2NXANCNFSM47PHOLCQ .

ErichZimmer commented 2 years ago

FWI, I found a very early version of the h5py GUI on my phone. The concept is the same as the current h5py GUI, except it is less buggy, there is no disabling widgets (toggled by each checkbox), ensemble correlation isn't developed yet, and a few other things. It has about the same complexity as the current h5py GUI, so if you don't like it, I can stop developing it. The non-h5py GUI has a similar amount of complexity to the h5py GUI, except it doesn't have to deal with storing results in h5py. https://github.com/ErichZimmer/openpiv_tk_gui/tree/GUI_2.0_prototype

eguvep commented 2 years ago

Open source communities divide, split and merge all the time :)

That is completely true and a good thing! If we drop the original GUI officially, I may still move it to a repository outside OpenPIV and maintain it for teaching and laboratory purposes. It will probably still be available, but more in a private sense, not in the »official« repository.

But let's wait until Erichs internet connection is back again ;-)

alexlib commented 2 years ago

FWI, I found a very early version of the h5py GUI on my phone. The concept is the same as the current h5py GUI, except it is less buggy, there is no disabling widgets (toggled by each checkbox), ensemble correlation isn't developed yet, and a few other things. It has about the same complexity as the current h5py GUI, so if you don't like it, I can stop developing it. The non-h5py GUI has a similar amount of complexity to the h5py GUI, except it doesn't have to deal with storing results in h5py. https://github.com/ErichZimmer/openpiv_tk_gui/tree/GUI_2.0_prototype

@ErichZimmer

let's try to see the differences between the present master branch of the GUI and yours and then start merging things one by one, with @eguvep in the loop for every step. For that, please, first using Github - do "fetch upstream" into your repo and equalize it with the present state. Then, press Compare or Contribute and create a pull request for one-two features that you want to add and we'll review it.

In parallel, please create a repo of your full, new shiny GUI with h5py and we'll look at it as well. Please give the repo a name that will be clearly distinguish it from the existing state - not to confuse the browsing user. We'll also look at it together and decide together which features we can copy from it, one feature at a time - into your fork of the single, official GUI and from there a pull request to the upstream, official repo.

How does it sound for you as a plan?

alexlib commented 2 years ago

Open source communities divide, split and merge all the time :)

That is completely true and a good thing! If we drop the original GUI officially, I may still move it to a repository outside OpenPIV and maintain it for teaching and laboratory purposes. It will probably still be available, but more in a private sense, not in the »official« repository.

But let's wait until Erichs internet connection is back again ;-)

Yep, btw, I was on a hype to make a Streamlit app, but as usual, I never have time to implement things, only the first bit :) Now, I'm on another hype - to make a napari plugin :) it's so easy and straightforward and the image viewer is so fast (based on OpenGL VIspy technology) that it's simply amazing. But, as much as I know myself, I'll never be able to complete this work in the same way as you did for the OpenPIV GUI. So dreams apart and practicality apart :)

eguvep commented 2 years ago

How does it sound for you as a plan?

Perfect!

ErichZimmer commented 2 years ago

I STEAL'd (Strategically Transport Equipment to Alternative Location) my laptop back and looked at the advanced GUIs. The h5py and non-h5py versions are nearly identical in complexity and structure, so I'll focus on the h5py version a little more. The advanced GUI uses a version of the Add-Ins system to separate pre/post-processing functions from the main code. In the Add-In files, there is two additional (and optional) dictionary and list to allow for toggled widgets attached to Boolean check button widgets (functions not in use are set to "disabled" state). There is still a lot to be finished (figure generator was removed?), so I'll try to get the GUI looking "nice" and hopefully without any bugs. As for combining the simple and advanced GUIs, I wasn't able to do it, but I did enhance the simple GUI with more pre-processing and post-processing functions, so I'll hopefully have pull requests by next week.

ErichZimmer commented 2 years ago

Additionally, a second order image dewarping function is being developed, but it is going slow due to my lack of expertise in mathematics (took too long of a break :P )

alexlib commented 2 years ago

Additionally, a second order image dewarping function is being developed, but it is going slow due to my lack of expertise in mathematics (took too long of a break :P )

Great. Where is it? We had another repo by Theo with a similar development - better is we learn from both

ErichZimmer commented 2 years ago

My internet is a little too slow to push the GUIs to my fork, so I'll try again later. The theory is based on the article, Distortion correction of two-component - two-dimensional PIV using a large imaging sensor with application to measurements of a turbulent boundary layer flow at Reτ = 2386 where the normalized autocorrelation of the calibration image is used to find the peaks. Invalid peak locations can be manually removed so only valid peaks are left. After that, the object plane peaks need to be found, but I'm having trouble with that currently. Finally, to solve and warp the image, scikit-image ProjectiveTransform() is used to get the warp matrix which is then used to warp the image with the warp() function. For the sake of performance, two application methods should be featured; one for images and one for vectors. Both have similar RMS and bias errors, but correcting vectors is much faster and can be used on any vector field.

alexlib commented 2 years ago

please see https://github.com/TKaeufer/Open_PIV_mapping we tried also to use scipy and scikit-image, but eventually Theo's code was the most robust

ErichZimmer commented 2 years ago

Is the repository public?

OpenPIV / openpiv_tk_gui