search

adrn / schwimmbad

A common interface to processing pools.
MIT License
115 stars 18 forks source link

Speed difference to emcee.utils.MPIPool? #17

Closed adrn closed 2 months ago

adrn commented 7 years ago

Is there a huge speed difference?

See: https://github.com/gnarayan/WDmodel/commit/a65c676e0b2d055939d54ebc5d1744c0e1f25780

dfm commented 7 years ago

I'd be surprised if this is true in general. If so, we should definitely figure it out! The emcee pool can do some load balancing, but it doesn't look like that's what's happening here. I know that @rodluger managed to get a significant speedup by specializing the pool to pass only numpy objects, but that wasn't ever added to emcee. It would be interesting to know where the bottleneck is.