Ptychography-4-0 / ptychography

Code repository for Ptychography 4.0 project.

https://ptychography-4-0.github.io/ptychography/

GNU General Public License v3.0

27 stars 14 forks source link

Use multithreading that was merged into LiberTEM master #35

Closed uellue closed 3 years ago

uellue commented 3 years ago

With this change, using the CPU is faster than the GPU.

This can be merged and released after LiberTEM is released with the new threading feature.

Contributor Checklist:

[x] I have added myself to the creators.json file
[N/A] I have added a changelog entry for my contribution
[x] I have added/updated documentation for all user-facing changes
[N/A] I have added/updated test cases

SimeonEhrig commented 3 years ago

Is this parallelization used for steps before and after the UDF?

sk1p commented 3 years ago

Is this parallelization used for steps before and after the UDF?

The idea is that we can, in addition to the more coarse-grained data-parallelism, also support fine-grained parallelism inside of the UDF. For example, one could allow two threads per process, and possibly gain something from hyperthreading, or more than two threads if it matches the CPU/NUMA topology better.

In the extreme case, one would allow to use nthreads=ncores inside the UDF and use a single process - we (ab)use this for our live processing prototype, where the multi-process version is not done yet.

uellue commented 3 years ago

The commit history now contains a version that uses multithreading on transposed data. Wall clock wise it is 40 % slower, but CPU time wise it is 2x as fast as the current state. Is this useful for anything?

uellue commented 3 years ago

Rerunning CI, should work now with current LiberTEM release 0.7

codecov[bot] commented 3 years ago

Codecov Report

Merging #35 (a517b1e) into master (f64e89a) will decrease coverage by 2.81%. The diff coverage is 55.88%.

@@            Coverage Diff             @@
##           master      #35      +/-   ##
==========================================
- Coverage   66.90%   64.08%   -2.82%     
==========================================
  Files           8        8              
  Lines         278      323      +45     
  Branches       29       38       +9     
==========================================
+ Hits          186      207      +21     
- Misses         81      104      +23     
- Partials       11       12       +1

Impacted Files	Coverage Δ
src/ptychography40/reconstruction/ssb/udf.py	`63.20% <50.81%> (-10.61%)`	:arrow_down:
src/ptychography40/reconstruction/ssb/__init__.py	`100.00% <100.00%> (ø)`
src/ptychography40/reconstruction/ssb/trotters.py	`65.88% <100.00%> (+1.68%)`	:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update f64e89a...a517b1e. Read the comment docs.

uellue commented 3 years ago

Tests passed, merging! Other open issues before the release can be addressed in separate PRs.