search

utdemir / distributed-dataset

A distributed data processing framework in Haskell.
BSD 3-Clause "New" or "Revised" License
114 stars 5 forks source link

Replace Control.Concurrent.Throttled with Control.Concurrent.QSem? #13

Closed axman6 closed 5 years ago

axman6 commented 5 years ago

With a small API change, you could take advantage to of the pre-existing QSem type which is designed for this sort of job:

throttled :: MonadUnliftIO m => Int -> m (m a -> m a)
throttled n | n <= 0 = pure id
throttled n = do
  sem <- liftIO $ newQSem n
  run <- askRunInIO
  return $ \ma -> liftIO $ bracket_ (waitQSem sem) (signalQSem sem) (run ma)

Neither way seems obviously better than the other - I use this method in amazonka-s3-streaming to limit the number of current threads.

utdemir commented 5 years ago

Thanks! Only reason I rolled my own is because I wasn't aware of this; so I have no reason to believe that mine is better.

I will definitely use this; however I was thinking about also throttling localProcessBackend as I do with lambdaBackend; so I want to apply this change in a slightly bigger refactor where I move this module to distributed-dataset package and expose it.