Closed Richert closed 2 years ago
I would like to extend this proposal and state the goal that every object from frontend to backend have file i/o methods. Where possible, these should include yaml
readers/writers, but pickle
should be available everywhere. Additionally I would like to introduce a mode
attribute to the to_file
/from_file
methods to state whether the target/source file is in yaml/pickle or whatever other format we might introduce in the future. Dedicated methods for different file formats might still exist, but this way we have a single point of entry for saving and loading progress.
commits 572f127c293a811dd2d7007f08d2ba3dd41652fb and fd9892bcc4f5315c4291535ed7da73f911583aec fix the current implementation of IR pickling and allow to pickle IR instances. In principle all kinds of IR instances are supported, but only CircuitIR is actively tested.
It might be useful to implement dependent tests, where test 1 creates a network (with backend) and saves the result to file; and test 2 loads that from file and computes. That might cover some relevant use cases.
I implemented yaml-based i/o methods that allow to save AbstractTemplate
instances to a yaml file such that it can be loaded back into Python via AbstractTemplate.from_yaml
again. Further I/O methods do not make much sense after the backend restructuring, in my opinion, since intermediate representation always have to be build on the fly during a new simulation/file generation call.
To make it easier to work with large, densely connected networks, it would be helpful to have
to_file(filename)
andfrom_file(filename)
methods implemented at the level of theCircuitIR
class.The file format could just be a simple pickle file in my opinion, since the purpose is not to just save a definition or configuration file, but to actually save progress that has been made due to the compilation/vectorization of the network, which can take very long for large systems.
This method should check, whether the circuit has already been compiled, i.e. whether the backend is already fully set up. In this case, pickling cannot be done right away, currently. The reason for this is that the backend creates and imports functions to simulate the network behavior, that cannot be pickled.
To solve this, I suggest to implement
NumpyBackend.to_file()
and.from_file
methods as well that are called by theCircuitIR
that write/load each of those functions to/from a python file. The strings for this are already generated in the backend.