search

ahay / src

Main Madagascar source
Other
264 stars 173 forks source link

Parallelize basic utilities with OMP #70

Open sfomel opened 13 years ago

sfomel commented 13 years ago

Parallelize basic utilities like sfattr and sftransp with OMP. For sfattr, test using a large dataset if it is worth the code duplication to move the tests for types outside the for loops.

Reported by: ivlad

sfomel commented 13 years ago

Let's make a list of which utilities we'd like so we can check them off:

sffft1 sffft3 sfadd sfmath sftransp sfattr

Others?

Original comment by: godwinj

sfomel commented 13 years ago

sfbandpass, sfstretch, sfstack would also come to mind. With some types of RAID disks (for file servers) or SSDs (for laptops) some types of simple data-parallel problems that were I/O bound cease to be so, parallel I/O becomes possible, and it's worth trying to put the CPU to work.

When it comes to the types of problems in these common programs, I see two classes only:

(1) the simple data-parallel ones, for which parallelization is trivial, and OMP parallelization works well in tandem with distributed-memory parallelization with an external tool. MapReduce at shared-memory level.

(2) sftransp, which is a much more complex problem. Doing an optimized general-case transpose block-by-block and in parallel, taking into account transfer times between cache, RAM, and disk (let alone distributed-memory issues) is quite something. There are some optimized procedures for some particular transposition cases in some linear algebra packages (ATLAS? LAPACK?), it might make sense to use them if they are available, for those cases, and fall back to a general-case algorithm if they are not available or if this is one of the cases not covered by them. A transpose can be quite an issue computationally -- think just the "simple" case of transforming 5-D, wide-azimuth data (tens of terabytes in size) from some type of gathers to frequency slices...

Original comment by: ivlad