search

trynthink / scout

A tool for estimating the future energy use, carbon emissions, and capital and operating cost impacts of energy efficiency and demand flexibility technologies in the U.S. residential and commercial building sectors.
https://scout.energy.gov
Other
61 stars 23 forks source link

Find more efficient pickling solution for writing/reading ECMs #170

Closed jtlangevin closed 7 years ago

jtlangevin commented 7 years ago

The use of pickle.dump() (and possibly pickle.load()) in a for loop appears to be running into memory issues for more complex ECMs:

pickle.dump(meas_prepped_compete[ind], zp, protocol=4) MemoryError

It's possible that clearing the pickled variable before each iteration of the loop would help: http://stackoverflow.com/questions/16288936/how-do-i-prevent-memory-leak-when-i-load-large-pickle-files-in-a-for-loop

Or rather than doing a loop with a pickle.dump() in each iteration, the ECM data could all be stored in a list and then pickled only once: http://stackoverflow.com/questions/20716812/saving-and-loading-multiple-objects-in-pickle-file

jtlangevin commented 7 years ago

Upon further investigation, the pickling operation was not itself the primary cause for memory issues; rather, ECM definitions that indicated a wide range of applicable baseline market segments (e.g., 'Plug-and-Play Sensors (Prospective)') required significant memory resources to prepare in 'ecm_prep.py' for the analysis engine.

This problem has been mitigated to a large degree in commit 2c8d854 by removing from further analysis any applicable baseline market segment without complete technology cost, performance, and lifetime data. For now, this issue can be closed.