chunks

[lmiq]

In this PR I implement the chunks iterator, which is useful to distribute the workload in threaded loops.

The iterators are implemented in this package, for the moment: https://github.com/m3g/ChunkSplitters.jl

The idea is to facilitate the distribution of workloads in custom threaded loops, with, for example:

julia> using IterTools

julia> Threads.@threads for (xrange, ichunk) in chunks(1:7, 3)
           @show xrange, ichunk
       end
(xrange, ichunk) = (6:7, 3)
(xrange, ichunk) = (4:5, 2)
(xrange, ichunk) = (1:3, 1)

and this can be used to split the workload in each range of indices of the array, and also to accumulate in thread-safe temporary variables, by indexing by the ranges and chunk numbers. For instance:

julia> function sum_parallel(f, x; nchunks=Threads.nthreads())
           s = fill(zero(eltype(x)), nchunks)
           Threads.@threads for (xrange, ichunk) in chunks(x, nchunks)
               for i in xrange
                   s[ichunk] += f(x[i])
               end
           end
           return sum(s)
       end
sum_parallel (generic function with 1 method)

julia> sum_parallel(x -> log(x)^7, rand(10^7); nchunks=128)
-5.039255477681504e10

[codecov-commenter]

Codecov Report

Merging #95 (92f52de) into master (831bbd2) will increase coverage by 2.10%. The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master      #95      +/-   ##
==========================================
+ Coverage   73.39%   75.49%   +2.10%     
==========================================
  Files           1        2       +1     
  Lines         233      253      +20     
==========================================
+ Hits          171      191      +20     
  Misses         62       62              

Impacted Files	Coverage Δ
src/IterTools.jl	73.39% <ø> (ø)
src/chunks.jl	100.00% <100.00%> (ø)

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

[sylvaticus]

Hello, any update concerning this pull request ?

[lmiq]

To be truth, I don´t think it is a good idea to merge it now, as the ChunkSplitters.jl package went already has been tested more carefully. If it was the case, the PR should be updated to include the latest version of it (maybe make it a dependency?).