search

brightway-lca / brightway2-data

Tools for the management of inventory databases and impact assessment methods. Part of the Brightway LCA framework.
https://docs.brightway.dev/
BSD 3-Clause "New" or "Revised" License
11 stars 24 forks source link

data_store method opening too many files #157

Closed Haitham-ghaida closed 1 year ago

Haitham-ghaida commented 1 year ago

Hi,

For a project I'm working on, I have to do Monte Carlo LCAs, a large list of activities.

It crashes when I have more than 40 activities for just 20 iterations. The same happens if I also increase iterations to a large number.

~/N/c/Building_LCA  /home/haithamth/anaconda3/envs/lcdna/bin/python /home/haithamth/Nextcloud/code/Building_LCA/testing.py
Performing LCA for treatment of used motor scooter: 100%|██████████████████████████████████████████████████████| 50/50 [00:06<00:00,  7.98it/s]
Performing LCA for market for magnesium, for reuse in hafnium sponge production iterations 9 / 20:  40%|███▏    | 8/20 [07:12<10:49, 54.09s/it]
Traceback (most recent call last):
  File "/home/haithamth/anaconda3/envs/lcdna/lib/python3.11/site-packages/fs/errors.py", line 125, in new_func
  File "/home/haithamth/anaconda3/envs/lcdna/lib/python3.11/site-packages/fs/zipfs.py", line 353, in __init__
  File "/home/haithamth/anaconda3/envs/lcdna/lib/python3.11/zipfile.py", line 1284, in __init__
OSError: [Errno 24] Too many open files: '/home/haithamth/.local/share/Brightway3/Bhorizionv2.184c2f48/processed/ef-v31-en15804hc.379b26b972ec9166bb95834289b397d7.zip'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/haithamth/Nextcloud/code/testing.py", line 13, in <module>
  File "/home/haithamth/Nextcloud/code/calculation_handler.py", line 81, in cache_LCA
  File "/home/haithamth/Nextcloud/code/LCA_handler.py", line 217, in monteCarloLCA
  File "/home/haithamth/anaconda3/envs/lcdna/lib/python3.11/site-packages/bw2calc/lca.py", line 547, in switch_method
  File "/home/haithamth/anaconda3/envs/lcdna/lib/python3.11/site-packages/bw2calc/lca.py", line 524, in _switch
  File "/home/haithamth/anaconda3/envs/lcdna/lib/python3.11/site-packages/bw2data/compat.py", line 123, in prepare_lca_inputs
  File "/home/haithamth/anaconda3/envs/lcdna/lib/python3.11/site-packages/bw2data/data_store.py", line 157, in datapackage
  File "/home/haithamth/anaconda3/envs/lcdna/lib/python3.11/site-packages/fs/zipfs.py", line 240, in __new__
  File "/home/haithamth/anaconda3/envs/lcdna/lib/python3.11/site-packages/fs/errors.py", line 129, in new_func
fs.errors.CreateFailed: unable to create filesystem, [Errno 24] Too many open files: '/home/haithamth/.local/share/Brightway3/Bhorizionv2.184c2f48/processed/ef-v31-en15804hc.379b26b972ec9166bb95834289b397d7.zip'

It might be a problem affecting only linux users, i haven't tried on windows. because by default linux allows for 1024 files to be open.

ulimit -n
1024

I'm using: Python V 3.11.6 bw2data V (4, 0, 'DEV33')