berthelm
commented
8 years ago I remember having a discussion a long time ago about pneumatk where the way the data was stored was important to the efficiency of the analysis. I guess this is a general problem. As I don’t know python, I cannot contribute solutions, but it seems clear that a discussion of the points is important.
From my perspective, and if I imagine myself as a user, I would want this discussion to emphasize the following:
- speed of loading preferably up to 1000s of cells
- speed of analysis
- ease of extending the analysis
- consistency with other BBP tools that I have heard mentioned relating to issues like the apical point
Who should be at this meeting and will someone make a set of concrete items for the agenda?
Julian
Julian Shillcock Algorithm Development Section Manager
julian.shillcock@epfl.chmailto:julian.shillcock@epfl.ch phone: +41 (0)21 69 39679
On 15 Apr 2016, at 12:15, Eleftherios Zisis notifications@github.com<mailto:notifications@github.com> wrote:
I would like to discuss some things that I have noticed in NeuroM. The suggestion that I present requires a lot of changes in order to be applied, thus I would like make clear that this post is more on thinking into the future than making any changes in the current structure of NeuroM. Let's begin:
We have adopted a tree representation of our data which is used throughout the modules via iterators, maps etc.
I understand that in languages where the trees are expressed with pointers, creating iterators and recursively generating stuff is super fast. In python however, generators are not fast, and functional programming will always lag behind the not so cheap function calls (although as far as we do not put them in loops, it's not that bad).
Iterators in Python are memory efficient because they do not store everything in memory at once when iterative calculations are yielded, but in our case that we first create the tree in memory and then return the parent node, we are in a lose lose situation. Plus, we always need to apply an O(n) traversal for operations where this could be avoided by a lookup table structure.
Our loaders and processing are slow when the order of magnitude of number of cells is hundred and up, needing a lot of minutes to load or process a population. When I tried to profile and optimize the loading I stumbled upon the scary truth: Our bottleneck is mainly the make_tree function which iteratively generates a tree from the data. Unfortunately, the recursive alternative is even worse due to the unoptimized recursion in Python.
I also noticed that in a lot of functions we convert the tree to a list of elements (as_elements), and I am wondering what would be the ratio of the times that we need a tree as a tree over the times that we need it as a array of points. Maybe we would need to write less code if we had the points to begin with.
Being stuck on the python level, we will always find these overheads in front of us, which kills the performance of NeuroM and causes serious problems when populations need to be analyzed.
You will ask: what alternatives do we have? Well, we can always go full numpy and exploit the speed of light (c). Instead of creating trees we can load the data through numpy, and generate a connectivity matrix for the "parenting" (I think that Pneumatk uses this design, not sure though). I suspect that this will eliminate the majority of the overheads.
What do you think?
— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHubhttps://github.com/BlueBrain/NeuroM/issues/301
juanchopanza
eleftherioszisis
lidakanari
I would like to discuss some things that I have noticed in NeuroM. The suggestion that I present requires a lot of changes in order to be applied, thus I would like make clear that this post is more on thinking into the future than making any changes in the current structure of NeuroM. Let's begin:
We have adopted a tree representation of our data which is used throughout the modules via iterators, maps etc.
I understand that in languages where the trees are expressed with pointers, creating iterators and recursively generating stuff is super fast. In python however, generators are not fast, and functional programming will always lag behind the not so cheap function calls (although as far as we do not put them in loops, it's not that bad).
Iterators in Python are memory efficient because they do not store everything in memory at once when iterative calculations are yielded, but in our case that we first create the tree in memory and then return the parent node, we are in a lose lose situation. Plus, we always need to apply an O(n) traversal for operations where this could be avoided by a lookup table structure.
Our loaders and processing are slow when the order of magnitude of number of cells is hundred and up, needing a lot of minutes to load or process a population. When I tried to profile and optimize the loading I stumbled upon the scary truth: Our bottleneck is mainly the make_tree function which iteratively generates a tree from the data. Unfortunately, the recursive alternative is even worse due to the unoptimized recursion in Python.
I also noticed that in a lot of functions we convert the tree to a list of elements (
as_elements), and I am wondering what would be the ratio of the times that we need a tree as a tree over the times that we need it as a array of points. Maybe we would need to write less code if we had the points to begin with.Being stuck on the python level, we will always find these overheads in front of us, which kills the performance of NeuroM and causes serious problems when populations need to be analyzed.
You will ask: what alternatives do we have? Well, we can always go full numpy and exploit the speed of light (c). Instead of creating trees we can load the data through numpy, and generate a connectivity matrix for the "parenting" (I think that Pneumatk uses this design, not sure though). I suspect that this will eliminate the majority of the overheads.
What do you think?