search

maxthemouse / CrystalMIR

Analysis of data for Multiple Image Radiography
BSD 3-Clause "New" or "Revised" License
0 stars 0 forks source link

Multi-threading #9

Open maxthemouse opened 3 years ago

maxthemouse commented 3 years ago

Look at multi-threading or multi-processing to improve performance.

maxthemouse commented 3 years ago

Not sure if this makes sense. The overhead of pulling out the data from the array and then putting the results into the new array could overtake the benefit of threading. Note that the main use of asyncio is for i/o where you have to wait for something from a drive or network. In this case, everything is is memory and the main thing is to process everything as quickly as possible. I.e. the process is cpu intensive.

maxthemouse commented 3 years ago

This might be something that would addressed if the fitting code is written in Julia which has specific methods for tasks like this.

maxthemouse commented 2 years ago

Using numba might be a simple was to do this. I don't know how the jit interacts with numpy.polyfit.

maxthemouse commented 2 years ago

Trying numba @jit and it is working somewhat. However, it looks like the only part that really helps is on the r-squared difference calculation which is composed of basic math functions. Adding to the main function with the fitting doesn't seem to work and throws warnings about classes. Using polyfit doesn't work with the jit. Even so, the time goes from 1:30 to :59 which is consistent with turning off the r-squared calculation. Dropping the r-squared drops the time to :57. In other words, the jit makes the r-squared calc only 2 seconds. That is a huge help as I prefer to have it on by default. Parallel did not work as written due to classes as Numba doesn't work with arbitrary classes such as the progress bar or polyfit.

maxthemouse commented 10 months ago

There is a GPU version of numpy that could be an option.