Reading large OSeMOSYS datafiles is very slow

OSeMOSYS / otoole

OSeMOSYS Tools for Energy

https://otoole.readthedocs.io

MIT License

25 stars 18 forks source link

Reading large OSeMOSYS datafiles is very slow #44

Closed willu47 closed 4 years ago

willu47 commented 4 years ago

Reading in large OSeMOSYS datafiles is very slow (otoole.preprocess.datafile_to_datapackage). This could be due to the otoole.longify_data.check_datatypes, but some profiling would help clarify where the bottleneck is.

This could also be the pulp.amply library which is written using pyparsing. Note sure how this deals with large files (e.g. TEMBA has datafiles of around 191,000 lines).

willu47 commented 4 years ago

Profiling confirms that the issue is with Amply and pyparsing. I have now split Amply out of the PuLP library and will be maintaining this on the coin-or organisation. In the meantime, an independent version of Amply is available here for testing.

willu47 commented 4 years ago

Closing as a duplicate of #10