array_parmap

[JuliaLawall]

The documenation (mli file) for array_parmap says:

      If the optional [chunksize] parameter is specified,
      the processes compute the result in an on-demand fashion
      on blochs of size [chunksize]; this provides automatic
      load balancing for unbalanced computations, but the order
      of the result is no longer guaranteed to be preserved. 

So that means that if the chunksize argument is used, f(a[27]) might appear at position 42 in the result array?

[rdicosmo]

Yes, that's the case indeed. One can work around this limitation by tagging the array elements with their index, and sorting the result. E,g,: let taggeda = Array.mapi (fun n x -> (n,x)) a and f' (_,x) = f x in let b = Parmap.array_parmap f' tagged a in let _ = Array.fast_sort (fun (n,_) (m,_) -> n-m) b in Array.map (fun (_,x) -> x) b

[JuliaLawall]

On Sun, 27 Dec 2020, Roberto Di Cosmo wrote:

Yes, that's the case indeed. One can work around this limitation by tagging the array elements with their index, and sorting the result. E,g,: let taggeda = Array.mapi (fun n x -> (n,x)) a and f' (_,x) = f x in let b = Parmap.arrayparmap f' tagged a in let = Array.fastsort (fun (n,) (m,) -> n-m) b in Array.map (fun (,x) -> x) b

OK. It's hard to see the point of using an array in this case...

julia

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, orunsubscribe.[AAD2ZGSY2K2ORG6HMFQOEMLSW3UEVA5CNFSM4VJ2QD22YY3PNVWWK3TUL52HS4 DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOFTFAYGA.gif]

[rdicosmo]

If you do not force a chunksize, the order is preserved...

More seriously, it is definitely possible to improve the inner routines in parmap.ml to preserve the order even with a fixed chunksize, by tagging the result chunks, and avoiding the extra sort step.

This could be done by replacing result with (result, lo, hi) in this line, and using these values to reconstruct the array in order at the end and dropping the collect functions passed to the generic mapper driver.

If somebody wants to have a look at this, it is a nice exercise.

[rdicosmo]

I just pushed an updated version of Parmap on the branch sorted, latest commit d95e26897a0f4984513a3ffd7ce6f774037dd4fd, to fix the ordering issue. This version preserves order for all the mapping functions even when chunksize is used, and from my early tests it seems that one does not pay any significant penalty in performance.

@JuliaLawall would you mind checking this out and testing with your applications? If everything seems fine, I'll merge it into master and make a new major release.

[rdicosmo]

Closing this leftover issue (solved in https://github.com/rdicosmo/parmap/releases/tag/1.2)