Update functions present error

[romainsacchi]

I'm using premise 1.6.1, with python 3.9. I also have all libraries instaled, imported to the script (do I need to?) and with the correct versions.

When I run the code for databses imported with escopold2 files (cut-off), everything's fine

BUT when I try to rune any transformation function (updates) the same error keeps showing, with both ecospold2 3.8 and 3.9

update_electricity(), update_cement(), update_steel(), update_fuels(): "IndexError: list index out of range" in the "get" function in the "pool" script (line771). update_dac(): "TypeError: '<' not supported between instances of 'str' and 'NoneType'" in the "get" function in the "pool" script (line771). update_two_wheelers(): "AttributeError: 'IAMDataCollection' object has no attribute 'trsp_two_wheelers'" in the "get" function in the "pool" script (line771). In the functions update_emissions(), update_cars(), update_trucks() and update_buses() all work. (I'm implementing these functions as ndb.update_electricity())

[romainsacchi]

Can you write the exact script you tried to run?

[romainsacchi]

Also, can you update to v.1.6.3 and let me know if the issue persists?

[KamikoNana]

I've updated premise to v1.6.3 and still have the same issue in he same update functions. The exact script I have is: import numpy import pandas import bw2io import bw2data import wurst import xarray import prettytable import pycountry import cryptography import premise_gwp import yaml import sparse import schema import datapackage import requests import bottleneck import constructive_geometries import arrow

import premise

print (premise.version)

from premise import * from datapackage import Package

CREATE DATABSE FROM ECOSPOLD2 - CUT-OFF

ndb = NewDatabase( scenarios=[ {"model":"image", "pathway":"SSP1-Base", "year":2030}, ],

source_type="ecospold",

source_file_path=r"C:\Users\marta.tejada-ext\Desktop\databases\ecoinvent_cutoff_3.9",

source_version="3.9",

key='xxxxxxxxxxxxxxxxxxxxxxx',  #I have an actual key

)

Later, in the console, I use the function ndb.update_electricity()

[romainsacchi]

Hi @KamikoNana Ok, I'll look into this. Question: why do you import all these libraries? You should just need to import premise (which then imports all these libraries in the background).

[romainsacchi]

Can you try again but pointing to the "datasets" folder of the decompressed 7z archive? Like so: source_file_path=r"C:\Users\marta.tejada-ext\Desktop\databases\ecoinvent_cutoff_3.9\datasets"

[KamikoNana]

I wasn't sure if I needed to import all the libraries one by one, and thought that maybe the error could be related to that. Thanks for clarifying that for me!

[KamikoNana]

The file path is the one I retrieve from the properties of the databse file The "ecoinvent_cutoff_3.9" is a folder with only the database inside - if I try to add something more to the source_file_path it does not recognize any database

[romainsacchi]

Are we both talking about the decompressed archive 7z file one downloads from the ecoinvent website? Because it should look like this:

ecoinvent 3.9.1_cutoff_ecoSpold02

• datasets
• MasterData
• ...

[KamikoNana]

Indeed you were correct, I didn't unzip de folder so that wasn't an option However, after correctly unzip an now with a correct path (adding the \datasets), the same erros in the same update functions keep happening

[KamikoNana]

Just to inform, I've been trying with the following versions of ecoinvent ecospold2 files (and the errors are the same for all of them): 3.8 3.9 3.9 lci 3.9.1 3.9.1 lci And I'm changing the parameter "source_version" accordingly

[romainsacchi]

Hey, in a few minutes, v.1.6.4 will be available.

It should allow you to do:

from premise import *

#CREATE DATABSE FROM ECOSPOLD2 - CUT-OFF
ndb = NewDatabase(
    scenarios=[
        {"model":"image", "pathway":"SSP1-Base", "year":2030},
    ],

    source_type="ecospold",
    source_file_path=r"/Users/romain/Documents/ecoinvent 3.9.1_cutoff_ecoSpold02/datasets",
    source_version="3.9",
    key='xxxx',  #I have an actual key
)
ndb.update_all()
ndb.write_db_to_simapro()

[KamikoNana]

Thank you so much for your help, and for how fast you attend to the community issues!

However I'm encountering a new situation with using of premise 1.6.4., using the same ecoinvent 3.9.1 cut-off ecospold2 (with the correct path on my end) - althought the same is happening for the other versions that I previously mentioned In the step EXTRACTING SOURCE DATABASE the console shows "Cannot find cached database. Will create one now for next time... Extracting XML data from 21238 datasets"

Is this a normal behaviour? (I'm asking because it never happened before)

[romainsacchi]

Yes, upon first use, premise will cache the database and inventories, so that this saves times on subsequent uses.

[KamikoNana]

Okay, thank you once more for your help!

[romainsacchi]

Hi, any improvement?

[KamikoNana]

Thanks once again for all the help you’ve been providing me regarding the use of premise.

I’m using the same code as before the premise new update 1.6.4 with python 3.9.17. But using all the versions of EcoInvent I have access to (3.8, 3.9, 3.9lci, 3.9.1, 3.9.1lci) the code takes more than 15min to perform this step (noticing that wilt v3.9.1lci it spend more than 4hours in this process). “ //////////////////// EXTRACTING SOURCE DATABASE //////////////////// Cannot find cached database. Will create one now for next time... Extracting XML data from 21255 datasets “ Also, the anaconda kernel crashes, having to shut down all the spyder setup to be able to stop the code. Do you have any more suggestions of what I might be doing wrong?

In annex I'm sending a printscreen of how I'm downloading the ecospol2 files from EcoInvent website, just to check if there are any issues regarding this step

[romainsacchi]

I see. I could have been more specific here: you should only use the process-based database, meaning the one you indicate with the arrow. The other ones cannot be used.

[KamikoNana]

By changing the file directory I could solve this issue! However, the same problem that premise 1.6.3 had (related with the error in the update functions) is still on. When applying the update_all() function, no changes happen - acording with the reports that premise allows to retrieve:

• generate_scenario_report(): this report stays excacty
• generate_change_report(): this file shows up with only the first line, regarding the data used, but no information about any changes

To recall, the error are: update_electricity(), update_cement(), update_steel(), update_fuels(): "IndexError: list index out of range" in the "get" function in the "pool" script (line771). update_dac(): "TypeError: '<' not supported between instances of 'str' and 'NoneType'" in the "get" function in the "pool" script (line771). update_two_wheelers(): "AttributeError: 'IAMDataCollection' object has no attribute 'trsp_two_wheelers'" in the "get" function in the "pool" script (line771). In the functions update_emissions(), update_cars(), update_trucks() and update_buses() all work.

[romainsacchi]

In the change report, you have several tabs: is there really no logs in them?

[KamikoNana]

Not at all, the change report only has 1 page, with the following information:

[romainsacchi]

OK, I'll look into it. Will get back to you.

[romainsacchi]

I cannot reproduce the issue. The change report appears correctly populated. Can you maybe give it another try? Also, we've released v.1.6.5, which fixes another bug, try to get it.

[romainsacchi]

Hi, any news? Or is the issue persisting?

[Lars-Nun]

I think I have the same issue using premise version 1.6.7. When calling newdb.update_all(), I get a list index out of range error:

IndexError Traceback (most recent call last) Cell In[62], line 3

  2 #newdb.update_electricity()

----> 3 newdb.update_all() 4 Audio(sound_file, autoplay=True)

File ~\mambaforge\envs\brightwayeco38\lib\site-packages\premise\ecoinvent_modification.py:1162, in NewDatabase.update_all(self) 1149 with ProcessPool(processes=multiprocessing.cpu_count()) as pool: 1150 args = [ 1151 ( 1152 scenario, (...) 1160 for scenario in self.scenarios 1161 ] -> 1162 results = pool.starmap(_update_all, args) 1164 for s, scenario in enumerate(self.scenarios): 1165 self.scenarios[s] = results[s][0]

File ~\mambaforge\envs\brightwayeco38\lib\multiprocessing\pool.py:375, in Pool.starmap(self, func, iterable, chunksize) 369 def starmap(self, func, iterable, chunksize=None): 370 ''' 371 Like map() method but the elements of the iterable are expected to 372 be iterables as well and will be unpacked as arguments. Hence 373 func and (a, b) becomes func(a, b). 374 ''' --> 375 return self._map_async(func, iterable, starmapstar, chunksize).get()

File ~\mambaforge\envs\brightwayeco38\lib\multiprocessing\pool.py:774, in ApplyResult.get(self, timeout) 772 return self._value 773 else: --> 774 raise self._value

IndexError: list index out of range

I also had 'IAMDataCollection' object has no attribute 'trsp_two_wheelers' when calling newdb.update_two_wheelers() and an empty change log...

[romainsacchi]

Hi @Lars-Nun, could you provide the script you tried to run?

[Lars-Nun]

Hi @romainsacchi, I sent the entire script to you via email. The last line here (update_all()) is where the error occurs.

import bw2data
import premise
import brightway2 as bw
import numpy as np
import scipy as sp
from brightway2 import *
from bw2data.parameters import ActivityParameter, DatabaseParameter, ProjectParameter, Group

scenarios = [{"model":"remind", "pathway":"SSP2-Base", "year":2050}]

from premise import *

fpei38cut = r"...\Ecoinvent\EcospoldFiles\ecoinvent_3.8_cutoff\ecoinvent_3.8_cutoff_ecoSpold02\unpacked\ecoinvent_3.8_cutoff_ecoSpold02\datasets"
#fpei38cut = r"...\Ecoinvent\EcospoldFiles\ecoinvent_3.8_cutoff\ecoinvent_3.8_cutoff_ecoSpold02\datasets"
newdb = NewDatabase(
    scenarios = scenarios,
    source_type="ecospold",
    source_file_path= fpei38cut, 
    source_version="3.8",
    key='************' #  I have a key
)
eidb38_rem_SSP2base2050 = bw.Database('ecoinvent_cutoff_3.8_remind_SSP2-Base_2050')

newdb.update_all()

[romainsacchi]

Hi @Lars-Nun ,

I did not receive it. In the meanwhile, you could update to v.1.6.9 and try again? (you probably know that, but you need to restart your notebook's Python kernel after the update).

[BastienPais]

Hi @romainsacchi, I think I have the same issue than @Lars-Nun (using premise 1.7.2, python 3.9 and ecoinvent 3.8): from datapackage import Package import brightway2 as bw ndb = NewDatabase( scenarios=[ {"model":"remind", "pathway":"SSP2-PkBudg1150", "year":2020}, ], source_type="ecospold", source_file_path=r"D:\b.pais\Documents\Ecoinvent\3.8\datasets", source_version="3.8", key='######', ) ndb.update_all(

The error is: `--------------------------------------------------------------------------- RemoteTraceback Traceback (most recent call last) RemoteTraceback: """ Traceback (most recent call last): File "C:\Users\ba.pais\AppData\Local\miniconda3\envs\premise\lib\multiprocessing\pool.py", line 125, in worker result = (True, func(*args, **kwds)) File "C:\Users\ba.pais\AppData\Local\miniconda3\envs\premise\lib\multiprocessing\pool.py", line 51, in starmapstar return list(itertools.starmap(args[0], args[1])) File "C:\Users\ba.pais\AppData\Local\miniconda3\envs\premise\lib\site-packages\premise\ecoinvent_modification.py", line 458, in _update_all scenario, modified_datasets, cache = _update_fuels( File "C:\Users\ba.pais\AppData\Local\miniconda3\envs\premise\lib\site-packages\premise\fuels.py", line 299, in _update_fuels fuels.generate_fuel_markets() File "C:\Users\ba.pais\AppData\Local\miniconda3\envs\premise\lib\site-packages\premise\fuels.py", line 2340, in generate_fuel_markets self.generate_fuel_supply_chains() File "C:\Users\ba.pais\AppData\Local\miniconda3\envs\premise\lib\site-packages\premise\fuels.py", line 2026, in generate_fuel_supply_chains self.generate_hydrogen_activities() File "C:\Users\ba.pais\AppData\Local\miniconda3\envs\premise\lib\site-packages\premise\fuels.py", line 573, in generate_hydrogen_activities energy_suppliers = self.find_suppliers( File "C:\Users\ba.pais\AppData\Local\miniconda3\envs\premise\lib\site-packages\premise\fuels.py", line 447, in find_suppliers locations=possible_locations[counter], IndexError: list index out of range """

The above exception was the direct cause of the following exception:

IndexError Traceback (most recent call last) Cell In[12], line 1 ----> 1 ndb.update_all()

File ~\AppData\Local\miniconda3\envs\premise\lib\site-packages\premise\ecoinvent_modification.py:1279, in NewDatabase.update_all(self) 1266 with ProcessPool(processes=multiprocessing.cpu_count()) as pool: 1267 args = [ 1268 ( 1269 scenario, (...) 1277 for scenario in self.scenarios 1278 ] -> 1279 results = pool.starmap(_update_all, args) 1281 for s, scenario in enumerate(self.scenarios): 1282 self.scenarios[s] = results[s][0]

File ~\AppData\Local\miniconda3\envs\premise\lib\multiprocessing\pool.py:372, in Pool.starmap(self, func, iterable, chunksize) 366 def starmap(self, func, iterable, chunksize=None): 367 ''' 368 Like map() method but the elements of the iterable are expected to 369 be iterables as well and will be unpacked as arguments. Hence 370 func and (a, b) becomes func(a, b). 371 ''' --> 372 return self._map_async(func, iterable, starmapstar, chunksize).get()

File ~\AppData\Local\miniconda3\envs\premise\lib\multiprocessing\pool.py:771, in ApplyResult.get(self, timeout) 769 return self._value 770 else: --> 771 raise self._value

IndexError: list index out of range`

[romainsacchi]

@BastienPais could you try again with version 1.8.0?

[BastienPais]

It seems to work with ecoivent 3.8, but with 3.9 the output is: `RemoteTraceback Traceback (most recent call last) RemoteTraceback: """ Traceback (most recent call last): File "C:\Users\ba.pais\AppData\Local\miniconda3\envs\premise\lib\site-packages\premise\electricity.py", line 931, in create_new_markets_high_voltage locations=possible_locations[counter], IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\Users\ba.pais\AppData\Local\miniconda3\envs\premise\lib\multiprocessing\pool.py", line 125, in worker result = (True, func(*args, **kwds)) File "C:\Users\ba.pais\AppData\Local\miniconda3\envs\premise\lib\multiprocessing\pool.py", line 51, in starmapstar return list(itertools.starmap(args[0], args[1])) File "C:\Users\ba.pais\AppData\Local\miniconda3\envs\premise\lib\site-packages\premise\ecoinvent_modification.py", line 444, in _update_all scenario, modified_datasets, cache = _update_electricity( File "C:\Users\ba.pais\AppData\Local\miniconda3\envs\premise\lib\site-packages\premise\electricity.py", line 201, in _update_electricity electricity.update_electricity_markets() File "C:\Users\ba.pais\AppData\Local\miniconda3\envs\premise\lib\site-packages\premise\electricity.py", line 2170, in update_electricity_markets self.create_new_markets_high_voltage() File "C:\Users\ba.pais\AppData\Local\miniconda3\envs\premise\lib\site-packages\premise\electricity.py", line 965, in create_new_markets_high_voltage raise IndexError( IndexError: Couldn't find suppliers for Gas CHP when looking for set(). """

The above exception was the direct cause of the following exception:

IndexError Traceback (most recent call last) Cell In[8], line 1 ----> 1 ndb.update_all()

File ~\AppData\Local\miniconda3\envs\premise\lib\site-packages\premise\ecoinvent_modification.py:1330, in NewDatabase.update_all(self) 1317 with ProcessPool(processes=multiprocessing.cpu_count()) as pool: 1318 args = [ 1319 ( 1320 scenario, (...) 1328 for scenario in self.scenarios 1329 ] -> 1330 results = pool.starmap(_update_all, args) 1332 for s, scenario in enumerate(self.scenarios): 1333 self.scenarios[s] = results[s][0]

File ~\AppData\Local\miniconda3\envs\premise\lib\multiprocessing\pool.py:372, in Pool.starmap(self, func, iterable, chunksize) 366 def starmap(self, func, iterable, chunksize=None): 367 ''' 368 Like map() method but the elements of the iterable are expected to 369 be iterables as well and will be unpacked as arguments. Hence 370 func and (a, b) becomes func(a, b). 371 ''' --> 372 return self._map_async(func, iterable, starmapstar, chunksize).get()

File ~\AppData\Local\miniconda3\envs\premise\lib\multiprocessing\pool.py:771, in ApplyResult.get(self, timeout) 769 return self._value 770 else: --> 771 raise self._value

IndexError: Couldn't find suppliers for Gas CHP when looking for set().`

[romainsacchi]

@BastienPais OK. What script/scenarios did you try to run?

[BastienPais]

from datapackage import Package import brightway2 as bw ndb = NewDatabase( scenarios=[ {"model":"remind", "pathway":"SSP2-PkBudg1150", "year":2020}, {"model":"remind", "pathway":"SSP2-PkBudg1150", "year":2030}, {"model":"remind", "pathway":"SSP2-PkBudg1150", "year":2040}, {"model":"remind", "pathway":"SSP2-PkBudg1150", "year":2050}, ], source_type="ecospold", source_file_path=r"D:\b.pais\Documents\Ecoinvent\3.9\datasets", source_version="3.9", key='tUePmX_S5B8ieZkkM7WUU2CnO8SmShwmAeWK9x2rTFo=', )

and ndb.update_all()

[romainsacchi]

@BastienPais This looks fine. Can you print what conda list gives you?

[BastienPais]

@romainsacchi ab-plugin-scenariolink 0.0.6 pypi_0 pypi activity-browser 2.6.9 pyhd8ed1ab_1 conda-forge anyio 4.0.0 pyhd8ed1ab_0 conda-forge appdirs 1.4.4 pyh9f0ad1d_0 conda-forge argon2-cffi 23.1.0 pyhd8ed1ab_0 conda-forge argon2-cffi-bindings 21.2.0 py39ha55989b_4 conda-forge arrow 1.3.0 pyhd8ed1ab_0 conda-forge asteval 0.9.31 pyhd8ed1ab_0 conda-forge asttokens 2.4.0 pyhd8ed1ab_0 conda-forge astunparse 1.6.3 pyhd8ed1ab_0 conda-forge async-lru 2.0.4 pyhd8ed1ab_0 conda-forge attrs 23.1.0 pyh71513ae_1 conda-forge babel 2.13.0 pyhd8ed1ab_0 conda-forge backcall 0.2.0 pyh9f0ad1d_0 conda-forge backports 1.0 pyhd8ed1ab_3 conda-forge backports.functools_lru_cache 1.6.5 pyhd8ed1ab_0 conda-forge beautifulsoup4 4.12.2 pyha770c72_0 conda-forge bleach 6.1.0 pyhd8ed1ab_0 conda-forge blinker 1.6.3 pyhd8ed1ab_0 conda-forge boto3 1.28.68 pypi_0 pypi botocore 1.31.68 pypi_0 pypi bottleneck 1.3.7 pypi_0 pypi brightway2 2.4.3 pyhd8ed1ab_0 conda-forge brotli 1.1.0 hcfcfb64_1 conda-forge brotli-bin 1.1.0 hcfcfb64_1 conda-forge brotli-python 1.1.0 py39h99910a6_1 conda-forge brotlicffi 1.1.0.0 py39h99910a6_1 conda-forge bw2analyzer 0.10 pyhd8ed1ab_0 conda-forge bw2calc 1.8.2 py39hcbf5309_0 conda-forge bw2data 3.6.6 pyhd8ed1ab_0 conda-forge bw2io 0.8.10 pypi_0 pypi bw2parameters 1.1.0 pyhd8ed1ab_0 conda-forge bw_migrations 0.2 pyhd8ed1ab_0 conda-forge bzip2 1.0.8 h8ffe710_4 conda-forge ca-certificates 2023.7.22 h56e8100_0 conda-forge cached-property 1.5.2 hd8ed1ab_1 conda-forge cached_property 1.5.2 pyha770c72_1 conda-forge certifi 2023.7.22 pypi_0 pypi cffi 1.16.0 py39ha55989b_0 conda-forge chardet 5.2.0 pypi_0 pypi charset-normalizer 3.3.1 pyhd8ed1ab_0 conda-forge click 8.1.7 win_pyh7428d3b_0 conda-forge colorama 0.4.6 pyhd8ed1ab_0 conda-forge comm 0.1.4 pyhd8ed1ab_0 conda-forge constructive-geometries 0.8.2 pypi_0 pypi contextlib2 21.6.0 pypi_0 pypi contourpy 1.1.1 py39h1f6ef14_1 conda-forge country-converter 1.0.0 pypi_0 pypi cryptography 41.0.4 pypi_0 pypi cycler 0.12.1 pyhd8ed1ab_0 conda-forge datapackage 1.15.2 pypi_0 pypi debugpy 1.8.0 py39h99910a6_1 conda-forge decorator 5.1.1 pyhd8ed1ab_0 conda-forge defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge dill 0.3.7 pyhd8ed1ab_0 conda-forge docopt 0.6.2 pypi_0 pypi eidl 2.0.1 pyhd8ed1ab_0 conda-forge eight 1.0.1 pyhd8ed1ab_4 conda-forge entrypoints 0.4 pyhd8ed1ab_0 conda-forge et_xmlfile 1.1.0 pyhd8ed1ab_0 conda-forge exceptiongroup 1.1.3 pyhd8ed1ab_0 conda-forge executing 1.2.0 pyhd8ed1ab_0 conda-forge fasteners 0.19 pypi_0 pypi flask 3.0.0 pyhd8ed1ab_0 conda-forge fonttools 4.43.1 py39ha55989b_0 conda-forge fqdn 1.5.1 pyhd8ed1ab_0 conda-forge freetype 2.12.1 hdaf720e_2 conda-forge future 0.18.3 pyhd8ed1ab_0 conda-forge gettext 0.21.1 h5728263_0 conda-forge glib 2.78.0 h12be248_0 conda-forge glib-tools 2.78.0 h12be248_0 conda-forge greenlet 3.0.0 pypi_0 pypi gst-plugins-base 1.22.6 h001b923_2 conda-forge gstreamer 1.22.6 hb4038d2_2 conda-forge icu 72.1 h63175ca_0 conda-forge idna 3.4 pyhd8ed1ab_0 conda-forge ijson 3.2.3 pypi_0 pypi importlib-metadata 6.8.0 pyha770c72_0 conda-forge importlib-resources 6.1.0 pyhd8ed1ab_0 conda-forge importlib_metadata 6.8.0 hd8ed1ab_0 conda-forge importlib_resources 6.1.0 pyhd8ed1ab_0 conda-forge inflate64 0.3.1 py39ha55989b_4 conda-forge intel-openmp 2023.2.0 h57928b3_50496 conda-forge ipykernel 6.25.2 pyh60829e3_0 conda-forge ipython 8.16.1 pyh5737063_0 conda-forge ipython-genutils 0.2.0 pypi_0 pypi ipywidgets 8.1.1 pypi_0 pypi isodate 0.6.1 pypi_0 pypi isoduration 20.11.0 pyhd8ed1ab_0 conda-forge itsdangerous 2.1.2 pyhd8ed1ab_0 conda-forge jedi 0.19.1 pyhd8ed1ab_0 conda-forge jinja2 3.1.2 pyhd8ed1ab_1 conda-forge jmespath 1.0.1 pypi_0 pypi json5 0.9.14 pyhd8ed1ab_0 conda-forge jsonlines 4.0.0 pypi_0 pypi jsonpointer 2.4 py39hcbf5309_3 conda-forge jsonschema 4.19.1 pyhd8ed1ab_0 conda-forge jsonschema-specifications 2023.7.1 pyhd8ed1ab_0 conda-forge jsonschema-with-format-nongpl 4.19.1 pyhd8ed1ab_0 conda-forge jupyter 1.0.0 pypi_0 pypi jupyter-console 6.6.3 pypi_0 pypi jupyter-lsp 2.2.0 pyhd8ed1ab_0 conda-forge jupyter_client 8.4.0 pyhd8ed1ab_0 conda-forge jupyter_core 5.4.0 py39hcbf5309_0 conda-forge jupyter_events 0.8.0 pyhd8ed1ab_0 conda-forge jupyter_server 2.8.0 pyhd8ed1ab_0 conda-forge jupyter_server_terminals 0.4.4 pyhd8ed1ab_1 conda-forge jupyterlab 4.0.7 pyhd8ed1ab_0 conda-forge jupyterlab-widgets 3.0.9 pypi_0 pypi jupyterlab_pygments 0.2.2 pyhd8ed1ab_0 conda-forge jupyterlab_server 2.25.0 pyhd8ed1ab_0 conda-forge jupyterthemes 0.20.0 pypi_0 pypi kiwisolver 1.4.5 py39h1f6ef14_1 conda-forge krb5 1.20.1 heb0366b_0 conda-forge lcms2 2.15 he9d350c_2 conda-forge lerc 4.0.0 h63175ca_0 conda-forge lesscpy 0.15.1 pypi_0 pypi libblas 3.9.0 19_win64_mkl conda-forge libbrotlicommon 1.1.0 hcfcfb64_1 conda-forge libbrotlidec 1.1.0 hcfcfb64_1 conda-forge libbrotlienc 1.1.0 hcfcfb64_1 conda-forge libcblas 3.9.0 19_win64_mkl conda-forge libclang 16.0.6 default_heb8d277_1 conda-forge libclang13 16.0.6 default_hc80b9e7_1 conda-forge libdeflate 1.19 hcfcfb64_0 conda-forge libffi 3.4.2 h8ffe710_5 conda-forge libglib 2.78.0 he8f3873_0 conda-forge libhwloc 2.9.3 default_haede6df_1009 conda-forge libiconv 1.17 h8ffe710_0 conda-forge libjpeg-turbo 2.1.5.1 hcfcfb64_1 conda-forge liblapack 3.9.0 19_win64_mkl conda-forge libogg 1.3.4 h8ffe710_1 conda-forge libpng 1.6.39 h19919ed_0 conda-forge libsodium 1.0.18 h8d14728_1 conda-forge libsqlite 3.43.2 hcfcfb64_0 conda-forge libtiff 4.6.0 h4554b19_1 conda-forge libvorbis 1.3.7 h0e60522_0 conda-forge libwebp 1.3.2 hcfcfb64_1 conda-forge libwebp-base 1.3.2 hcfcfb64_0 conda-forge libxcb 1.15 hcd874cb_0 conda-forge libxml2 2.11.5 hc3477c8_1 conda-forge libxslt 1.1.37 h6070c61_1 conda-forge libzlib 1.2.13 hcfcfb64_5 conda-forge linear-tsv 1.1.0 pypi_0 pypi llvmlite 0.41.1 pypi_0 pypi lxml 4.9.3 pypi_0 pypi m2w64-gcc-libgfortran 5.3.0 6 conda-forge m2w64-gcc-libs 5.3.0 7 conda-forge m2w64-gcc-libs-core 5.3.0 7 conda-forge m2w64-gmp 6.1.0 2 conda-forge m2w64-libwinpthread-git 5.0.0.4634.697f757 2 conda-forge markupsafe 2.1.3 py39ha55989b_1 conda-forge matplotlib-base 3.8.0 py39hf19769e_2 conda-forge matplotlib-inline 0.1.6 pyhd8ed1ab_0 conda-forge mistune 3.0.1 pyhd8ed1ab_0 conda-forge mkl 2023.2.0 h6a75c08_50496 conda-forge mrio_common_metadata 0.2.1 pyhd8ed1ab_0 conda-forge msys2-conda-epoch 20160418 1 conda-forge multiprocess 0.70.15 py39ha55989b_1 conda-forge multivolumefile 0.2.3 pyhd8ed1ab_0 conda-forge munkres 1.1.4 pyh9f0ad1d_0 conda-forge nbclient 0.8.0 pyhd8ed1ab_0 conda-forge nbconvert-core 7.9.2 pyhd8ed1ab_0 conda-forge nbformat 5.9.2 pyhd8ed1ab_0 conda-forge nest-asyncio 1.5.8 pyhd8ed1ab_0 conda-forge networkx 3.2 pyhd8ed1ab_1 conda-forge notebook 7.0.6 pypi_0 pypi notebook-shim 0.2.3 pyhd8ed1ab_0 conda-forge numba 0.58.1 pypi_0 pypi numpy 1.23.0 pypi_0 pypi openjpeg 2.5.0 h3d672ee_3 conda-forge openpyxl 3.1.2 py39ha55989b_1 conda-forge openssl 3.1.3 hcfcfb64_0 conda-forge overrides 7.4.0 pyhd8ed1ab_0 conda-forge packaging 23.2 pyhd8ed1ab_0 conda-forge pandas 2.1.1 py39h32e6231_1 conda-forge pandocfilters 1.5.0 pyhd8ed1ab_0 conda-forge parso 0.8.3 pyhd8ed1ab_0 conda-forge pathos 0.3.1 pyhd8ed1ab_0 conda-forge patsy 0.5.3 pyhd8ed1ab_0 conda-forge pcre2 10.40 h17e33f8_0 conda-forge peewee 3.17.0 py39habf9222_0 conda-forge pickleshare 0.7.5 py_1003 conda-forge pillow 10.0.1 py39h6ef006c_1 conda-forge pint 0.22 pyhd8ed1ab_1 conda-forge pip 23.3.1 pyhd8ed1ab_0 conda-forge pkgutil-resolve-name 1.3.10 pyhd8ed1ab_1 conda-forge platformdirs 3.11.0 pyhd8ed1ab_0 conda-forge ply 3.11 py_1 conda-forge pox 0.3.3 pyhd8ed1ab_0 conda-forge ppft 1.7.6.7 pyhd8ed1ab_0 conda-forge premise 1.8.0 pypi_0 pypi premise-gwp 0.9.6 pypi_0 pypi presamples 0.2.7 pyhd8ed1ab_1 conda-forge prettytable 3.9.0 pypi_0 pypi prometheus_client 0.17.1 pyhd8ed1ab_0 conda-forge prompt-toolkit 3.0.39 pyha770c72_0 conda-forge prompt_toolkit 3.0.39 hd8ed1ab_0 conda-forge psutil 5.9.6 pypi_0 pypi pthread-stubs 0.4 hcd874cb_1001 conda-forge pthreads-win32 2.9.1 hfa6e2cd_3 conda-forge pure_eval 0.2.2 pyhd8ed1ab_0 conda-forge py7zr 0.20.6 pyhd8ed1ab_0 conda-forge pyarrow 13.0.0 pypi_0 pypi pybcj 1.0.1 py39ha55989b_3 conda-forge pycountry 22.3.5 pypi_0 pypi pycparser 2.21 pyhd8ed1ab_0 conda-forge pycryptodomex 3.19.0 py39ha55989b_1 conda-forge pygments 2.16.1 pyhd8ed1ab_0 conda-forge pypardiso 0.4.3 pyhd8ed1ab_0 conda-forge pyparsing 3.1.1 pyhd8ed1ab_0 conda-forge pyperclip 1.8.2 pyhd8ed1ab_2 conda-forge pyppmd 1.0.0 py39h99910a6_2 conda-forge pyprind 2.11.3 pypi_0 pypi pyqt 5.15.9 py39hb77abff_5 conda-forge pyqt5-sip 12.12.2 py39h99910a6_5 conda-forge pyqtwebengine 5.15.9 py39h2f4a3f1_5 conda-forge pyside2 5.15.8 py39hdfd5cee_3 conda-forge pysocks 1.7.1 pyh0701188_6 conda-forge python 3.9.18 h4de0772_0_cpython conda-forge python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python-fastjsonschema 2.18.1 pyhd8ed1ab_0 conda-forge python-json-logger 2.0.7 pyhd8ed1ab_0 conda-forge python-tzdata 2023.3 pyhd8ed1ab_0 conda-forge python_abi 3.9 4_cp39 conda-forge pytz 2023.3.post1 pyhd8ed1ab_0 conda-forge pywin32 306 py39h99910a6_2 conda-forge pywinpty 2.0.12 py39h99910a6_0 conda-forge pyxlsb 1.0.10 pyhd8ed1ab_0 conda-forge pyyaml 6.0.1 py39ha55989b_1 conda-forge pyzmq 25.1.1 py39h7eaf5a6_2 conda-forge pyzstd 0.15.9 py39h95af829_1 conda-forge qt-main 5.15.8 h2c8576c_12 conda-forge qt-webengine 5.15.8 h5b1ea0b_0 conda-forge qtconsole 5.4.4 pypi_0 pypi qtpy 2.4.0 pypi_0 pypi referencing 0.30.2 pyhd8ed1ab_0 conda-forge requests 2.31.0 pyhd8ed1ab_0 conda-forge rfc3339-validator 0.1.4 pyhd8ed1ab_0 conda-forge rfc3986 2.0.0 pypi_0 pypi rfc3986-validator 0.1.1 pyh9f0ad1d_0 conda-forge rpds-py 0.10.6 py39hf21820d_0 conda-forge s3transfer 0.7.0 pypi_0 pypi salib 1.4.7 pyhd8ed1ab_0 conda-forge schema 0.7.5 pypi_0 pypi scipy 1.11.3 py39hddb5d58_1 conda-forge seaborn 0.13.0 hd8ed1ab_0 conda-forge seaborn-base 0.13.0 pyhd8ed1ab_0 conda-forge send2trash 1.8.2 pyh08f2357_0 conda-forge setuptools 68.2.2 pyhd8ed1ab_0 conda-forge setuptools-scm 8.0.4 pyhd8ed1ab_0 conda-forge sip 6.7.12 py39h99910a6_0 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge sniffio 1.3.0 pyhd8ed1ab_0 conda-forge soupsieve 2.5 pyhd8ed1ab_1 conda-forge sparse 0.14.0 pypi_0 pypi sqlalchemy 2.0.22 pypi_0 pypi stack_data 0.6.2 pyhd8ed1ab_0 conda-forge stats_arrays 0.6.6 pyhd8ed1ab_0 conda-forge statsmodels 0.14.0 py39hd88c2e4_2 conda-forge tableschema 1.20.2 pypi_0 pypi tabulate 0.9.0 pyhd8ed1ab_1 conda-forge tabulator 1.53.5 pypi_0 pypi tbb 2021.10.0 h91493d7_2 conda-forge terminado 0.15.0 py39hcbf5309_0 conda-forge texttable 1.7.0 pyhd8ed1ab_0 conda-forge tinycss2 1.2.1 pyhd8ed1ab_0 conda-forge tk 8.6.13 hcfcfb64_0 conda-forge toml 0.10.2 pyhd8ed1ab_0 conda-forge tomli 2.0.1 pyhd8ed1ab_0 conda-forge toolz 0.12.0 pypi_0 pypi tornado 6.3.3 py39ha55989b_1 conda-forge tqdm 4.66.1 pypi_0 pypi traitlets 5.11.2 pyhd8ed1ab_0 conda-forge types-python-dateutil 2.8.19.14 pyhd8ed1ab_0 conda-forge typing-extensions 4.8.0 hd8ed1ab_0 conda-forge typing_extensions 4.8.0 pyha770c72_0 conda-forge typing_utils 0.1.0 pyhd8ed1ab_0 conda-forge tzdata 2023c h71feb2d_0 conda-forge ucrt 10.0.22621.0 h57928b3_0 conda-forge unfold 1.1.5 pypi_0 pypi unicodecsv 0.14.1 pypi_0 pypi unicodedata2 15.1.0 py39ha55989b_0 conda-forge unidecode 1.3.7 pyhd8ed1ab_0 conda-forge uri-template 1.3.0 pyhd8ed1ab_0 conda-forge urllib3 1.26.18 pypi_0 pypi vc 14.3 h64f974e_17 conda-forge vc14_runtime 14.36.32532 hdcecf7f_17 conda-forge voluptuous 0.13.1 pyhd8ed1ab_0 conda-forge vs2015_runtime 14.36.32532 h05e6639_17 conda-forge wcwidth 0.2.8 pyhd8ed1ab_0 conda-forge webcolors 1.13 pyhd8ed1ab_0 conda-forge webencodings 0.5.1 pyhd8ed1ab_2 conda-forge websocket-client 1.6.4 pyhd8ed1ab_0 conda-forge werkzeug 3.0.0 pyhd8ed1ab_0 conda-forge wheel 0.41.2 pyhd8ed1ab_0 conda-forge whoosh 2.7.4 py39hcbf5309_8 conda-forge widgetsnbextension 4.0.9 pypi_0 pypi win_inet_pton 1.1.0 pyhd8ed1ab_6 conda-forge winpty 0.4.3 4 conda-forge wrapt 1.15.0 py39ha55989b_1 conda-forge wurst 0.3.4 pypi_0 pypi xarray 2023.10.1 pypi_0 pypi xlrd 2.0.1 pyhd8ed1ab_3 conda-forge xlsxwriter 3.1.9 pypi_0 pypi xorg-libxau 1.0.11 hcd874cb_0 conda-forge xorg-libxdmcp 1.1.3 hcd874cb_0 conda-forge xz 5.2.6 h8d14728_0 conda-forge yaml 0.2.5 h8ffe710_2 conda-forge zeromq 4.3.5 h63175ca_0 conda-forge zipfile-deflate64 0.2.0 py39ha55989b_4 conda-forge zipp 3.17.0 pyhd8ed1ab_0 conda-forge zstd 1.5.5 h12be248_0 conda-forge

[romainsacchi]

It's a tough one. Your libraries version are correct, and your script seems correct. At this point, I'd recommend to scrap your current conda environment and create a new one and try again. But before you do that, maybe try to do

from premise import clear_cache
clear_cache()

and try your script again.

[BastienPais]

update_all() seems to work when I do clear_cache(), but when I want to create a superstructure database the following problem occurs with ecoinvent 3.9: KeyError: ('Gas, natural', 'natural resource', 'in ground', 'Sm3') And something similar with ecoinvent 3.8 (difficulty in linking a biosphere flow to the generated database)

[romainsacchi]

OK, can you try:

import bw2io
bw2io.create_default_biosphere3(overwrite=True)

This will update your biosphere3, and hopefully fix this issue.

[BastienPais]

Now, the issue is: OperationalError: cannot VACUUM from within a transaction Seems similar to what you reported here: https://github.com/brightway-lca/brightway2-data/issues/127

[romainsacchi]

It seems you're going through all the possible struggles :-)

Before running NewDatabase(), try to delete the remnants of the database first:

if "ecoinvent_cutoff_3.9_remind_SSP2-PkBudg1150_2020" in bw.databases:
    del bw.databases["ecoinvent_cutoff_3.9_remind_SSP2-PkBudg1150_2020"]

That should delete the failed/old attempt at registering the database (just do bw.databases if you're not sure about the name).

Note that producing a prospective LCA database for 2020 is maybe not ideal -- I'd use the original ecoinvent database as a reference to 2020.

[BastienPais]

Sorry, that wasn't clear: it's when I runbw2io.create_default_biosphere3() that OperationalError: cannot VACUUM from within a transaction appears Thanks for the tip on the 2020 reference!

[romainsacchi]

OK, then try to restart the kernel of your notebook.

[BastienPais]

bw2io.create_default_biosphere3(overwrite=True) works fine but the issue KeyError: ('Gas, natural', 'natural resource', 'in ground', 'Sm3') is still present when i'm writing the superstructure database...

[romainsacchi]

A temporary fix can be to go to: ~\AppData\Local\miniconda3\envs\premise\lib\site-packages\premise\data\utils\export\flows_biosphere_39.csv

and to make sure you have these two flows:

Gas, mine, off-gas, process, coal mining;natural resource;in ground;standard cubic meter;3ed5f377-344f-423a-b5ec-9a9a1162b944 Gas, mine, off-gas, process, coal mining;natural resource;in ground;Sm3;3ed5f377-344f-423a-b5ec-9a9a1162b944

[BastienPais]

I don't have the second one (in Sm3 unit) Should I add it manually on the CSV file?

[romainsacchi]

Yes, you can duplicate the row containing "standard cubic meter" and replace the unit with "Sm3".

[romainsacchi]

Did this fix your issue?

[BastienPais]

It seems to work. Thanks a lot @romainsacchi !!