how to speed up the simulation of GCAL model ?

[windform2015]

Hi, I am wondering how to accelerate the simulation of GCAL model, since I need to run the model for a large time scale and this may use several days for the original speed. Are there some parts we can modify to push the simulation to run faster ? Thanks for your attention !

[jbednar]

Sure. One thing you can do is change some of the parameters to get a smaller memory footprint, which will give major speedups. E.g. you can reduce cortical density, which if you reduce by some multiple x will cut down the memory and simulation time by x^4. You can instead reduce only some of the connection field lengths, if they aren't important for your particular application. E.g. if you aren't interested in specific lateral connectivity, those connections dominate the memory and computation time, so you should make them a good bit smaller. You can probably even safely replace all the lateral connections with a fixed, shared kernel, for a dramatic speedup, but I haven't actually tested to see how well that works or what parameters you'd need to adjust.

If you don't want to change any of your parameters, the easiest thing is to try a machine with more cores; Topographica should be able to use all of them to good effect with no coding changes, giving a major speedup.

There are also GPU and MPI versions of Topographica, which allow you to run the most computationally intensive parts of the model on either GPU hardware or on clusters of machines, if you need more computing power or memory than you have on one machine. Alas, I haven't needed either one, so I can't tell you how to get started with them. @philippjfr could point you to how to use the GPU support, but he's very busy for the next few weeks, and I think it still has some rough edges that need filling in. For MPI support I can send you the thesis from the student who implemented it, but I think it has rough edges as well.

So there are definitely things you can do...

[windform2015]

Thanks a lot for your nice comments ! I will think about your suggestions thoroughly.

[windform2015]

Hi, how about batch mode processing of GCAL model, can it be faster ? Recently I read the user Manual and it mentions the batch processing, but I am not sure that means the running speed can become faster. For one of your suggestions, I reduced the cortex density and indeed, the running become faster and simulation time become a fraction as previous time. But I would like to run the model even faster than that, so I am testing other ways of your suggestions, maybe the batch mode is also an option for testing ?

[windform2015]

Now I understand what the batch mode means, and it is not supposed to be used for the speedups. Thanks for your attention.

[jbednar]

I wrote this before your last comment, but it doesn't look like it got posted:

Depends what you're doing. If you run one step at a time in the Tk GUI, then batch mode will be faster, because it won't be updating the display all the time. But if you run 10000 steps in one go in the GUI, it will be exactly the same as batch mode. The only difference between batch mode and GUI mode is the display. In fact, when I run things in batch mode they are usually slower, because I have it automatically run various analyses so that the results are ready when I need them. The big advantage of batch mode is simply that you are not sitting and watching, so that it doesn't much matter how long it takes in most cases.

[windform2015]

Okay, I see, thank you very much !

[windform2015]

Dear Bednar,

could you tell me some information about the MPI version of GCAL model, please ? For example, the student' thesis implementing this version.

With best regards

[jbednar]

Here is the thesis describing the MPI implementation. In nearly all cases in our own work, we've ended up needing to run multiple independent jobs rather than the single large jobs that MPI makes possible, and so we've not really been keeping up the MPI support. But it shouldn't be hard to revive it if it's gone stale. Good luck!

[mjabri]

i have been running large models on gpus (titan x) and i get about 10x to 100x speed up over 36 phys core machine. So definitely worth it to consider investing in a gpu board...

[windform2015]

@jbednar Thanks !

[windform2015]

@mjabri Can you tell further about how to implement the gpu version of Topograhica, please ?

[mjabri]

I actually started from http://homepages.inf.ed.ac.uk/s1137931/thesis.pdf and https://github.com/Tasignotas/topographica/ and added some functions to load fetch the connections weights from/to GPU. See also https://github.com/ioam/topographica/issues/635. I also had to fix a bug that makes it work with magnify_mapper > 0. I am waiting for a problem with the GUI to to be fixed so I can merge with latest version and put a pull request.

[philippjfr]

@mjabri Could you remind me what exact GUI functionality you need restored in master before applying submitting a PR?

[mjabri]

Here it is. https://github.com/ioam/topographica/issues/611#issuecomment-150275418

[windform2015]

@mjabri Hi, did you run GPU version of topographica in cluster ? How much is its speed up like ?

[mjabri]

I run 1 instance on 1 GPU (Titan X with 12GB RAM). Speed up depends on network size, but for me it is about 10 to 100x.

[windform2015]

@mjabri Sounds great ! Look forward to your problem to be fixed and your pull request for the GPU version.

[windform2015]

@mjabri Hi, currently it is kind of emergent for me to need a speed-up version of topographica to run some simulations and up to now the GPU version is supposed to be one of fastest versions. So I would like to ask, how is the problem you mentioned being processed ? If it is fixed already, can you share that runnable version with us ? I will really appreciate this, :)

[sea2016]

@mjabri Hi, I would like to ask, did you implement GPU processing of Topographica with MPI ? I am also very interested in the implementions to speed up the simulations.