metals/activities_mapping file-type problem in update.all()

[Paul-Robineau]

Hello there,

I am using premise 2023.12.29 py_0 romainsacchi (from conda list) through a jupyter notebook. Everything seems to be working fine when I do (cells fusionned for example):

from premise import *
import bw2data
bw2data.projects.set_current("premise")
bw2data.databases # <-- I verify I have my biosphere and cutoff391 database created through bw2 in the same env, though I then create the new database directly from the ecospold files
ndb = NewDatabase(
    scenarios=[
        {"model":"image", "pathway":"SSP2-RCP19", "year":2040},
    ],
    source_type="ecospold",
    source_file_path=r"C:\Users\mypath\ecoinvent 3.9.1_cutoff_ecoSpold02\datasets",
    source_version="3.9",
    key='XXX', # <-- I have a decryption key
    use_multiprocessing=False, # <-- multiprocessing was indeed causing troubles
    keep_uncertainty_data=False,
    use_absolute_efficiency=False,
)

But then ndb.update_all() never works.

First it prints:

Extracted 1 worksheets in 5.51 seconds Migrating to 3.8 first Applying strategy: migrate_datasets Applying strategy: migrate_exchanges Applying strategy: migrate_datasets Applying strategy: migrate_exchanges Applying strategy: migrate_datasets Applying strategy: migrate_exchanges Anomalies found: check the change report. Done!

Then I have an error message regarding the update_metals() sub-function, linked to the absence of the premise/data/metals/activities_mapping.xlsx file. I looked for the premise/data/metals folder (which is not present in the github repo), and I indeed do not have this file, but activities_mapping.yml instead.

Trying to use ndb.generate_scenario_report() or ndb.generate_change_report() after that immediatly leads to kernel crashing.

Full error message below:

---

FileNotFoundError Traceback (most recent call last) Cell In[4], line 1 ----> 1 ndb.update_all()

File ~\AppData\Local\miniconda3\envs\bw2\Lib\site-packages\premise\ecoinvent_modification.py:1409, in NewDatabase.update_all(self) 1407 else: 1408 for s, scenario in enumerate(self.scenarios): -> 1409 self.scenarios[s] = _update_all( 1410 scenario=scenario, 1411 version=self.version, 1412 system_model=self.system_model, 1413 use_absolute_efficiency=self.use_absolute_efficiency, 1414 vehicle_type="truck", 1415 gains_scenario=self.gains_scenario, 1416 ) 1418 self.update_external_scenario()

File ~\AppData\Local\miniconda3\envs\bw2\Lib\site-packages\premise\ecoinvent_modification.py:487, in _update_all(scenario, version, system_model, use_absolute_efficiency, vehicle_type, gains_scenario) 475 scenario, cache = _update_cement( 476 scenario=scenario, 477 version=version, 478 system_model=system_model, 479 cache=cache, 480 ) 481 scenario, cache = _update_steel( 482 scenario=scenario, 483 version=version, 484 system_model=system_model, 485 cache=cache, 486 ) --> 487 scenario, cache = _update_metals( 488 scenario=scenario, 489 version=version, 490 system_model=system_model, 491 cache=cache, 492 ) 493 scenario, cache = _update_fuels( 494 scenario=scenario, 495 version=version, 496 system_model=system_model, 497 cache=cache, 498 ) 500 scenario, cache = _update_heat( 501 scenario=scenario, 502 version=version, 503 system_model=system_model, 504 cache=cache, 505 )

File ~\AppData\Local\miniconda3\envs\bw2\Lib\site-packages\premise\metals.py:34, in _update_metals(scenario, version, system_model, cache) 33 def _update_metals(scenario, version, system_model, cache=None): ---> 34 metals = Metals( 35 database=scenario["database"], 36 model=scenario["model"], 37 pathway=scenario["pathway"], 38 iam_data=scenario["iam data"], 39 year=scenario["year"], 40 version=version, 41 system_model=system_model, 42 cache=cache, 43 ) 45 metals.create_metal_markets() 46 metals.update_metals_use_in_database()

File ~\AppData\Local\miniconda3\envs\bw2\Lib\site-packages\premise\metals.py:301, in Metals.init(self, database, iam_data, model, pathway, year, version, system_model, cache) 296 # Precompute the median values for each metal and origin_var for the year 2020 297 self.precomputed_medians = self.metals.sel(variable="median").interp( 298 year=self.year, method="nearest", kwargs={"fill_value": "extrapolate"} 299 ) --> 301 self.activities_mapping = load_activities_mapping() # 4 303 self.conversion_factors = load_conversion_factors() # 3 304 # Precompute conversion factors as a dictionary for faster lookups

File ~\AppData\Local\miniconda3\envs\bw2\Lib\site-packages\premise\metals.py:117, in load_activities_mapping() 111 """ 112 Load mapping for the ecoinvent exchanges to be 113 updated by the new metal intensities 114 """ 116 filepath = DATA_DIR / "metals" / "activities_mapping.xlsx" --> 117 df = pd.read_excel(filepath, sheet_name="activities_mapping") 118 return df

File ~\AppData\Local\miniconda3\envs\bw2\Lib\site-packages\pandas\io\excel_base.py:495, in read_excel(io, sheet_name, header, names, index_col, usecols, dtype, engine, converters, true_values, false_values, skiprows, nrows, na_values, keep_default_na, na_filter, verbose, parse_dates, date_parser, date_format, thousands, decimal, comment, skipfooter, storage_options, dtype_backend, engine_kwargs) 493 if not isinstance(io, ExcelFile): 494 should_close = True --> 495 io = ExcelFile( 496 io, 497 storage_options=storage_options, 498 engine=engine, 499 engine_kwargs=engine_kwargs, 500 ) 501 elif engine and engine != io.engine: 502 raise ValueError( 503 "Engine should not be specified when passing " 504 "an ExcelFile - ExcelFile already has the engine set" 505 )

File ~\AppData\Local\miniconda3\envs\bw2\Lib\site-packages\pandas\io\excel_base.py:1550, in ExcelFile.init(self, path_or_buffer, engine, storage_options, engine_kwargs) 1548 ext = "xls" 1549 else: -> 1550 ext = inspect_excel_format( 1551 content_or_path=path_or_buffer, storage_options=storage_options 1552 ) 1553 if ext is None: 1554 raise ValueError( 1555 "Excel file format cannot be determined, you must specify " 1556 "an engine manually." 1557 )

File ~\AppData\Local\miniconda3\envs\bw2\Lib\site-packages\pandas\io\excel_base.py:1402, in inspect_excel_format(content_or_path, storage_options) 1399 if isinstance(content_or_path, bytes): 1400 content_or_path = BytesIO(content_or_path) -> 1402 with get_handle( 1403 content_or_path, "rb", storage_options=storage_options, is_text=False 1404 ) as handle: 1405 stream = handle.handle 1406 stream.seek(0)

File ~\AppData\Local\miniconda3\envs\bw2\Lib\site-packages\pandas\io\common.py:882, in get_handle(path_or_buf, mode, encoding, compression, memory_map, is_text, errors, storage_options) 873 handle = open( 874 handle, 875 ioargs.mode, (...) 878 newline="", 879 ) 880 else: 881 # Binary mode --> 882 handle = open(handle, ioargs.mode) 883 handles.append(handle) 885 # Convert BytesIO or file objects passed with an encoding

FileNotFoundError: [Errno 2] No such file or directory: 'C:\Users\probinea\AppData\Local\miniconda3\envs\bw2\Lib\site-packages\premise\data\metals\activities_mapping.xlsx'

[romainsacchi]

Hi @Paul-Robineau , conda installs are fetching versions from the last commit, meaning it's not guaranteed to fetch a code that works. To get a working version, you should install from pip (pip install premise==1.8.2dev7).

[romainsacchi]

Or if you really only can work with conda, you can fetch it from confa-forge: conda install conda-forge::premise

[Paul-Robineau]

Hi @romainsacchi , I guessed that would be the explanation, but I did not see this issue being raised so I thought I should document the error (even though you might have already encountered it already through dev testing) :)

I created a VM with only miniconda (python 3.11.5), jupyterlab and premise 1.8.2dev7 (through pip) and I still encounter a problem, now regarding the IAM locations, a bit like described by @Michael-ljn (but I did not installed any brightway 2 or 2.5 component myself apart from what might be installed with premise). Should I create a dedicated issue for clarity?

Thank you for the work!

[Paul-Robineau]

Hi @romainsacchi , I guessed that would be the explanation, but I did not see this issue being raised so I thought I should document the error (even though you might have already encountered it already through dev testing) :)

I created a VM with only miniconda (python 3.11.5), jupyterlab and premise 1.8.2dev7 (through pip) and I still encounter a problem, now regarding the IAM locations, a bit like described by @Michael-ljn (but I did not installed any brightway 2 or 2.5 component myself apart from what might be installed with premise). Should I create a dedicated issue for clarity?

Thank you for the work!

Nevermind, I restarted the VM and that problem disappeared!