search

nismod / smif

Simulation Modelling Integration Framework
http://www.itrc.org.uk
MIT License
22 stars 6 forks source link

Performance: Trying to read model parameter default from key <*>.csv seems slow #332

Open willu47 opened 5 years ago

willu47 commented 5 years ago

With the large number of parameters in the digital comms model, reading from multiple columns of a single default.csv file is very slow.


DEBUG    Trying to read model parameter default from key ../digital_comms/parameters/defaults.csv
DEBUG    Trying to read model parameter default from key ../digital_comms/parameters/defaults.csv
DEBUG    Trying to read model parameter default from key ../digital_comms/parameters/defaults.csv
DEBUG    Trying to read model parameter default from key ../digital_comms/parameters/defaults.csv
DEBUG    Trying to read model parameter default from key ../digital_comms/parameters/defaults.csv
DEBUG    Trying to read model parameter default from key ../digital_comms/parameters/defaults.csv
DEBUG    Trying to read model parameter default from key ../digital_comms/parameters/defaults.csv
DEBUG    Trying to read model parameter default from key ../digital_comms/parameters/defaults.csv
DEBUG    Trying to read model parameter default from key ../digital_comms/parameters/defaults.csv
DEBUG    Trying to read model parameter default from key ../digital_comms/parameters/defaults.csv
DEBUG    Trying to read model parameter default from key ../digital_comms/parameters/defaults.csv
DEBUG    Trying to read model parameter default from key ../digital_comms/parameters/defaults.csv
DEBUG    Trying to read model parameter default from key ../digital_comms/parameters/defaults.csv
DEBUG    Trying to read model parameter default from key ../digital_comms/parameters/defaults.csv
DEBUG    Trying to read model parameter default from key ../digital_comms/parameters/defaults.csv
DEBUG    Trying to read model parameter default from key ../digital_comms/parameters/defaults.csv```
willu47 commented 5 years ago

Could cache the csv file or call to store.read_parameter_file using functools.lru_cache