search

SpikeAI / 2020-12_brainhack_Project7

Optimized Pipeline for the modelling of Spiking Neural Networks (SNNs)
GNU Affero General Public License v3.0
2 stars 0 forks source link

fast and efficient recording of data #3

Open laurentperrinet opened 3 years ago

laurentperrinet commented 3 years ago

At the moment, we are using pickle files to store neo objects, which is overkill when dealing with simple outputs from experiments

There are multiple alternatives

The solution should take into account efficiency (=> benchmark) vs ease of use (=> notebook showing usage) vs generality .

JuliaSprenger commented 3 years ago

I checked the PyNN requirements and it looks like it should be straight forward to extend this also to the currently recommended output format nix (Nix), e.g. in these lines. This does not require the usage of nix directly as it is used only indirectly via Neo.

I think we should dig deeper into this during the next project meeting.

laurentperrinet commented 3 years ago

doh! not supported on my ubuntu 20.04LTS... will try to compile from source

laurentperrinet commented 3 years ago

I began with a typical usecase to get an ouput neo object... work in progress (after lunch!)

JuliaSprenger commented 3 years ago

@laurentperrinet You can use nix from neo directly if you install it via pip install neo[nixio]. No need to install the C++ implementation of nix.

laurentperrinet commented 3 years ago

some documentation from pyNN on writing data, using nixio in particular

laurentperrinet commented 3 years ago

this saved me quite a lot of time - and works very fine!

$ python3 -m pip install --user neo[nixio]
Requirement already satisfied: neo[nixio] in /usr/local/lib/python3.8/dist-packages/neo-0.8.0-py3.8.egg (0.8.0)
Requirement already satisfied: numpy>=1.7.1 in ./.local/lib/python3.8/site-packages (from neo[nixio]) (1.19.2)
Requirement already satisfied: quantities>=0.9.0 in /usr/local/lib/python3.8/dist-packages/quantities-0.12.4-py3.8.egg (from neo[nixio]) (0.12.4)
Requirement already satisfied: numpy>=1.7.1 in ./.local/lib/python3.8/site-packages (from neo[nixio]) (1.19.2)
Requirement already satisfied: quantities>=0.9.0 in /usr/local/lib/python3.8/dist-packages/quantities-0.12.4-py3.8.egg (from neo[nixio]) (0.12.4)
Collecting nixio>=1.5.0b2
  Downloading nixio-1.5.0b6-py3-none-any.whl (122 kB)
     |████████████████████████████████| 122 kB 4.8 MB/s
Requirement already satisfied: six in /usr/lib/python3/dist-packages (from nixio>=1.5.0b2->neo[nixio]) (1.14.0)
Requirement already satisfied: numpy>=1.7.1 in ./.local/lib/python3.8/site-packages (from neo[nixio]) (1.19.2)
Collecting h5py
  Downloading h5py-3.1.0-cp38-cp38-manylinux1_x86_64.whl (4.4 MB)
     |████████████████████████████████| 4.4 MB 8.5 MB/s
Requirement already satisfied: numpy>=1.7.1 in ./.local/lib/python3.8/site-packages (from neo[nixio]) (1.19.2)
Installing collected packages: h5py, nixio
Successfully installed h5py-3.1.0 nixio-1.5.0b6
laurentperrinet commented 3 years ago

hi @albertoarturovergani !

I re-organized the structure of the folder to separate the different sub-tasks -> https://github.com/SpikeAI/2020-11_brainhack_Project7/commit/e56b56008d46aa8d11c3d87c64928a75edb742ee

I suggest your rename dev to output and remove the prepended 3_ on the notebooks to make things simpler

laurentperrinet commented 3 years ago

I recommend pushing regularly your latest notebooks