search

brightway-lca / brightway2-data

Tools for the management of inventory databases and impact assessment methods. Part of the Brightway LCA framework.
https://docs.brightway.dev/
BSD 3-Clause "New" or "Revised" License
11 stars 24 forks source link

Resolve "Too Many Open Files" Error by Managing ZipFS Resources #158

Closed Haitham-ghaida closed 1 year ago

Haitham-ghaida commented 1 year ago

I've modified the datapackage method to ensure proper management of ZipFS resources. The change involves using a with statement to handle the opening and closing of ZipFS instances. This aims to resolve the "Too Many Open Files" error encountered during extensive operations, such as Monte Carlo simulations in LCA calculations.

Modified code

  def datapackage(self):
        return load_datapackage(ZipFS(self.filepath_processed()))

to

def datapackage(self):
    zip_file_path = self.filepath_processed()
    with ZipFS(zip_file_path) as zip_fs:
        return load_datapackage(zip_fs)

I conducted an extensive manual testing process rather than traditional unit testing. I ran various LCA calculations using the entire library to validate the integrity and functionality of the modified method. The tests confirmed that the "Too Many Open Files" error was resolved without impacting the performance or output.

157

cmutel commented 1 year ago

Thanks Haitham, very much appreciated!

This won't be published as a new release for a bit, as I am working with @Cerebrovinny to clean up the current state of this repo, but I will push it out as soon as we have a stable release!