mapConcurrently is not reliable

[mwotton]

I wrote a little wrapper:

mapConcurrentlyBounded n f args = AsyncPool.withTaskGroup n $ \tg -> do
  AsyncPool.mapConcurrently tg f args

but it doesn't seem to work as expected

> mapConcurrentlyBounded 10 (\x -> threadDelay (x*100000) >> print x) [1..100]

This fails with an STM error.

mapConcurrentlyBounded :: Traversable t => Int -> (a1 -> IO a) -> t a1 -> IO (t a)
mapConcurrentlyBounded n f args = AsyncPool.withTaskGroup n $ \tg -> do
  AsyncPool.mapTasks tg (fmap f args)

works fine.

[expipiplus1]

This is still an issue.

Thank you for the workaround, @mwotton!

[jwiegley]

I'd say we redefine mapConcurrently to this version, then, rather than trying to use the Concurrently abstraction.

[jwiegley]

@mwotton PR opened.

[jwiegley]

A user on #haskell has claimed that this still breaks with the current version on Hackage, but is fixed by the suggested change, so more testing is needed.

[infinisil]

(me being that user):

For reference, this simple Main makes it fail:

module Main where

import           Control.Concurrent.Async.Pool

main :: IO ()
main = do
  pool <- createPool
  group <- createTaskGroup pool 1
  mapConcurrently group return [0]
  return ()

with

hashsearch: thread blocked indefinitely in an STM transaction
Received ExitFailure 1 when running

The async-pool version is confirmed to be 0.9.0.2, the latest one on hackage.

With nix you can reproduce it with

$ nix-shell -p 'haskellPackages.ghcWithPackages (p: [ p.async-pool ])' --command ghci -I nixpkgs=https://github.com/nixos/nixpkgs-channels/archive/nixos-unstable.tar.gz
GHCi, version 8.2.2: http://www.haskell.org/ghc/  :? for help
Prelude> import Control.Concurrent.Async.Pool
Prelude Control.Concurrent.Async.Pool> pool <- createPool
Prelude Control.Concurrent.Async.Pool> group <- createTaskGroup pool 1
Prelude Control.Concurrent.Async.Pool> mapConcurrently group return [0]

<hangs indefinitely>

[jwiegley]

@Infinisil Your problem is different from the original report. The difference here is that you're not doing anything in a thread. Just calling createTaskGroup and then mapTasks is very different from what withTaskGroup is doing.

[jwiegley]

Namely, at some point you need to runTaskGroup for the tasks within it to execute.

[infinisil]

Oh, sorry for that! Thanks for the information though, I didn't know that (I only read the functions docs, so maybe I've missed that)

[rycee]

Ran into this issue today 🙁

The fix for this never seem to have gotten into a release, would be great if one could be arranged and put on Hackage!

[jwiegley]

@rycee What was that fix exactly?

[rycee]

@jwiegley The latest version of async-pool on Hackage is 0.9.0.2 and its implementation of mapConcurrently is still the problematic one causing errors:

λ> import Control.Concurrent.Async.Pool
λ> withTaskGroup 1 $ \tg -> mapConcurrently tg print [1, 2, 3]
1
*** Exception: thread blocked indefinitely in an STM transaction

The implementation of commit 96589a8baffc4aa697d906f51016cfecfd20d818 fixes the issue for me as well

λ> mapConcurrently' tg f = mapTasks tg . fmap f
λ> withTaskGroup 1 $ \tg -> mapConcurrently' tg print [1, 2, 3]
1
2
3
[(),(),()]

so I would love to see a new release being made that includes this commit.

[jwiegley]

@rycee Got it; will do!