pyrates-neuroscience / PyRates

Open-source, graph-based Python code generator and analysis toolbox for dynamical systems (pre-implemented and custom models). Most pre-implemented models belong to the family of neural population models.
https://pyrates.readthedocs.io/en/latest/
GNU General Public License v3.0
73 stars 8 forks source link

Add to_file/from_file methods #17

Closed Richert closed 2 years ago

Richert commented 4 years ago

To make it easier to work with large, densely connected networks, it would be helpful to have to_file(filename) and from_file(filename) methods implemented at the level of the CircuitIR class.

The file format could just be a simple pickle file in my opinion, since the purpose is not to just save a definition or configuration file, but to actually save progress that has been made due to the compilation/vectorization of the network, which can take very long for large systems.

This method should check, whether the circuit has already been compiled, i.e. whether the backend is already fully set up. In this case, pickling cannot be done right away, currently. The reason for this is that the backend creates and imports functions to simulate the network behavior, that cannot be pickled.

To solve this, I suggest to implement NumpyBackend.to_file() and .from_file methods as well that are called by the CircuitIR that write/load each of those functions to/from a python file. The strings for this are already generated in the backend.

dafrose commented 4 years ago

I would like to extend this proposal and state the goal that every object from frontend to backend have file i/o methods. Where possible, these should include yaml readers/writers, but pickle should be available everywhere. Additionally I would like to introduce a mode attribute to the to_file/from_file methods to state whether the target/source file is in yaml/pickle or whatever other format we might introduce in the future. Dedicated methods for different file formats might still exist, but this way we have a single point of entry for saving and loading progress.

dafrose commented 4 years ago

commits 572f127c293a811dd2d7007f08d2ba3dd41652fb and fd9892bcc4f5315c4291535ed7da73f911583aec fix the current implementation of IR pickling and allow to pickle IR instances. In principle all kinds of IR instances are supported, but only CircuitIR is actively tested.

dafrose commented 4 years ago

It might be useful to implement dependent tests, where test 1 creates a network (with backend) and saves the result to file; and test 2 loads that from file and computes. That might cover some relevant use cases.

Richert commented 2 years ago

I implemented yaml-based i/o methods that allow to save AbstractTemplate instances to a yaml file such that it can be loaded back into Python via AbstractTemplate.from_yaml again. Further I/O methods do not make much sense after the backend restructuring, in my opinion, since intermediate representation always have to be build on the fly during a new simulation/file generation call.