More efficient loading of sparse activity files with python

[haydn-jones]

Significantly speeds up loading of sparse activity files in python (~25x from an SSD). Memory usage when loading in the PVP is also significantly reduced (maybe up to 4x or more? hard to figure out, massif is lying to me about memory usage). Haven't been able to do extensive performance/whether-or-not-it-works-properly testing because I'm using a laptop.

When building the frame lookup for a sparse pvp file, I save the number of active neurons in each frame. When loading it in I then allocate 3 arrays:

• A row array to store row indices for active neurons (row values will be duplicated, 1 index per active neuron in that row).
• A col array to store col indices for active neurons
• A value array to store the activation such that matrix[row[i], [col[i]] = value[i]

[sumtopmus]

Significantly speeds up loading of sparse activity files in python (~25x from an SSD). Memory usage when loading in the PVP is also significantly reduced (maybe up to 4x or more? hard to figure out, massif is lying to me about memory usage). Haven't been able to do extensive performance/whether-or-not-it-works-properly testing because I'm using a laptop.

When building the frame lookup for a sparse pvp file, I save the number of active neurons in each frame. When loading it in I then allocate 3 arrays:

• A row array to store row indices for active neurons (row values will be duplicated, 1 index per active neuron in that row).
• A col array to store col indices for active neurons
• A value array to store the activation such that matrix[row[i], [col[i]] = value[i]

I think it's better to return the sparse matrix in COO format since you are already using this format while creating a sparse matrix in the memory. CSR format is efficient for storing data on disk because it just takes less space, but COO is easier to use after reading the file from disk. There is no need for back and forth transformations between formats.

[haydn-jones]

Significantly speeds up loading of sparse activity files in python (~25x from an SSD). Memory usage when loading in the PVP is also significantly reduced (maybe up to 4x or more? hard to figure out, massif is lying to me about memory usage). Haven't been able to do extensive performance/whether-or-not-it-works-properly testing because I'm using a laptop. When building the frame lookup for a sparse pvp file, I save the number of active neurons in each frame. When loading it in I then allocate 3 arrays:

• A row array to store row indices for active neurons (row values will be duplicated, 1 index per active neuron in that row).
• A col array to store col indices for active neurons
• A value array to store the activation such that matrix[row[i], [col[i]] = value[i]

I think it's better to return the sparse matrix in COO format since you are already using this format while creating a sparse matrix in the memory. CSR format is efficient for storing data on disk because it just takes less space, but COO is easier to use after reading the file from disk. There is no need for back and forth transformations between formats.

COO does not directly support slicing which I would consider to be a major disadvantage over CSR.

[sumtopmus]

You are right, in order to split the data from PVP into different frames, CSR works better than COO.

[sumtopmus]

I am trying to understand how such significant speed-up was achieved. The profiler used to show ~13% of the total time for the list.extend method and ~50% of the time for the calls to np.array. These values are based on my particular PVP file and computer and they dropped to negligible values after your commit.

It is clear that calling list.extend for each frame was inefficient but it is quite surprising for me that the calls to np.array (which I believe were coming from conversion of python lists to numpy arrays) were taking that long. Do you know, by any chance, why list->np.array is so time-consuming?