efficient parallel list-list solver

[bertsky]

In rapidfuzz there's a cdist function that computes a matrix of alignment scores between each pair of two collections in parallel.

Is there something similar in pyalign, too?

[poke1024]

There's a so-called batch mode, though I need to remind myself how it works. It makes use of SIMD calls to speed up computing multiple alignments at the same time. This is the code at the end of the example notebook - it does not seem to work right now. I need to check.

[bertsky]

Ok, so in order to utilise multiple cores, I therefore have to wrap the solver (batched or not) in some Pythonic multiprocessing paradigm, right? Since builtin multithreading suffers from GIL and multiprocessing restricts sharing data to pickle-able objects, is there any one you can recommend in particular? (Like builtin concurrent.futures, or Joblib, Ray, MPIRE?)

[poke1024]

Yes, exactly. In theory, multithreading should already give good concurrency, since pyalign will release the GIL once it is inside the C++ code and does the heavy computations - I have never tested this out though. I have no suggestion to multiprocessing libs, I only have used the builtin multiprocessing so far.