search

HuwCampbell / grenade

Deep Learning in Haskell
BSD 2-Clause "Simplified" License
1.44k stars 84 forks source link

Concurrency? #33

Closed cpennington closed 6 years ago

cpennington commented 6 years ago

I'm trying to speed up a program that's using Grenade, and a) ran into MVar errors (my fault) and b) noticed that it seems to only run single-threaded, even if I'm using GHC concurrency. Is there something under the hood that would limit network execution to a single core at a time?

HuwCampbell commented 6 years ago

Cool. I hope other than speed issues it's working well for you.

Concurrency is something I'm interested in adding. I'll explain the state of play.

At the moment, all operations for a network run on a single thread. In some ways this is good, because it means that concurrency at a high level (running examples in parallel) works well, and one should be able to get a decent speed up from doing so – accumulating the results at the end and applying them.

The other issue tied to this is explicit minibatching. For most layers I believe the gains are marginal at best. But for layers where most operations are matrix vector multiplication (LSTM and fully connected are like this), just going to minibatching with matrix matrix multiplications over a minibatch could be a big win.

So I guess the reason I haven't added it is because there are number of levels where it could be added, and they bash against each other. So experimentation will be required to get it to work.

cpennington commented 6 years ago

Huh. I was trying to work at the high level (multiple threads running over multiple executions of the same network), and wasn't seeing it use more than one CPU simultaneously (or at least, I wasn't having any luck saturating my cpus). That said, it's quite possible that the problem was higher up. I'll have to run it through ThreadScope and see what I can find.

Do you know what parts of the network execution use an MVar? (I was getting an MVar error when I accidentally was feeding the output of a network run in as its own input).

HuwCampbell commented 6 years ago

I don't use any MVars or import Control.Concurrent anywhere. So not sure about that one.

cpennington commented 6 years ago

Ok, the issues w/ high level concurrency were definitely on my end. I've got it working concurrently now. (I'm still not sure where the MVar errors were coming from).