Closed BenPortner closed 2 years ago
I would prefer to drop this size to 2 or 3 - if you have 5 different giant databases, then do all operations on them one at a time. This cache will live for the whole time the process is open, so we should be conservative.
Note that LRU caching will break some tests. We use the pytest plugin antilru, but this still produced problems on Windows (I think). We used to cache some databases lookups but stopped because I couldn't get it to not break tests.
This is a fantastic effort, and I am not going to merge it. Instead, I will take the magic of pd.DataFrame(self)
(still can't believe that this actually works), and add some options, like limiting fields and sorting. We also want two methods, one for activities and the other for exchanges (see https://github.com/brightway-lca/brightway2-data/issues/106). We also want to be compatible, or at least be in line with, the work to convert premise to use DataFrames.
Implements a cached helper method
to_dataframe()
forSQLiteBackend
(and hence also for theIOTable
backend, because of inheritance). Uses standardfunctools.lru_cache
with a size of 5. Would be neat to have for https://github.com/brightway-lca/brightway2-analyzer/pull/16.Requesting review by @cmutel.
Speed improvement
On my machine, the improvement is approx. 8-fold. Using
get_labeled_inventory
from the correspondingbw2analyzer
branch:Memory usage
Ecoinvent 3.7.1 APOS takes approximately 60 MB of RAM. Assuming a user would call
Database(name).to_dataframe()
on five different ecoinvent versions, the total cache size would be 5x60 = 300 MB. On the next call to a new database, the first entry from the cache would be dropped, so memory usage would stay constant.