Closed simb-sdu closed 2 years ago
Could you post the conda list
output (type that in the terminal and copy paste the output)? The issue you mentioned is rather old, so this should not be happening with current versions of AB
It would also help if you could share the database/databases and scenario files and what calculation setup you are using so we can try to replicate the problem.
Thanks for your reply! I tried with most recent AB version (as well as the the latest dev release)
Could you post the
conda list
output (type that in the terminal and copy paste the output)? The issue you mentioned is rather old, so this should not be happening with current versions of AB
packages in environment at C:\Users\simb\Anaconda3\envs\ab:
Name Version Build Channel
activity-browser 2.6.8 py_0 bsteubing
appdirs 1.4.4 pyh9f0ad1d_0 conda-forge
arrow 1.2.2 pyhd8ed1ab_0 conda-forge
asteval 0.9.23 pyhd8ed1ab_0 conda-forge
astunparse 1.6.3 pyhd8ed1ab_0 conda-forge
beautifulsoup4 4.10.0 pyha770c72_0 conda-forge
brightway2 2.4.1 py_1 cmutel
brotli 1.0.9 h8ffe710_6 conda-forge
brotli-bin 1.0.9 h8ffe710_6 conda-forge
brotli-python 1.0.9 py39h415ef7b_6 conda-forge
brotlicffi 1.0.9.2 py39h415ef7b_1 conda-forge
brotlipy 0.7.0 py39hb82d6ee_1003 conda-forge
bw2analyzer 0.10 py_1 cmutel
bw2calc 1.8.0 py_0 cmutel
bw2data 3.6.4 py_0 cmutel
bw2io 0.8.6 py_1 cmutel
bw2parameters 0.7 py_0 cmutel
bw_migrations 0.1 py_0 cmutel
bzip2 1.0.8 h8ffe710_4 conda-forge
ca-certificates 2021.10.8 h5b45459_0 conda-forge
certifi 2021.10.8 py39hcbf5309_1 conda-forge
cffi 1.15.0 py39h0878f49_0 conda-forge
charset-normalizer 2.0.12 pyhd8ed1ab_0 conda-forge
cryptography 36.0.1 py39h7bc7c5c_0 conda-forge
cycler 0.11.0 pyhd8ed1ab_0 conda-forge
dill 0.3.4 pyhd8ed1ab_0 conda-forge
docopt 0.6.2 py_1 conda-forge
eidl 1.4.2 py_0 bsteubing
eight 1.0.0 py_0 conda-forge
et_xmlfile 1.0.1 py_1001 conda-forge
fasteners 0.17.3 pyhd8ed1ab_0 conda-forge
fonttools 4.30.0 py39hb82d6ee_0 conda-forge
freetype 2.10.4 h546665d_1 conda-forge
future 0.18.2 py39hcbf5309_4 conda-forge
fuzzywuzzy 0.18.0 pyhd8ed1ab_0 conda-forge
icu 68.2 h0e60522_0 conda-forge
idna 3.3 pyhd8ed1ab_0 conda-forge
importlib-metadata 4.11.3 py39hcbf5309_0 conda-forge
importlib_metadata 4.11.3 hd8ed1ab_0 conda-forge
intel-openmp 2022.0.0 h57928b3_3663 conda-forge
jbig 2.1 h8d14728_2003 conda-forge
jpeg 9e h8ffe710_0 conda-forge
kiwisolver 1.4.0 py39h2e07f2f_0 conda-forge
lcms2 2.12 h2a16943_0 conda-forge
lerc 3.0 h0e60522_0 conda-forge
libblas 3.9.0 13_win64_mkl conda-forge
libbrotlicommon 1.0.9 h8ffe710_6 conda-forge
libbrotlidec 1.0.9 h8ffe710_6 conda-forge
libbrotlienc 1.0.9 h8ffe710_6 conda-forge
libcblas 3.9.0 13_win64_mkl conda-forge
libclang 11.1.0 default_h5c34c98_1 conda-forge
libdeflate 1.10 h8ffe710_0 conda-forge
libffi 3.4.2 h8ffe710_5 conda-forge
libiconv 1.16 he774522_0 conda-forge
liblapack 3.9.0 13_win64_mkl conda-forge
libpng 1.6.37 h1d00b33_2 conda-forge
libtiff 4.3.0 hc4061b1_3 conda-forge
libwebp 1.2.2 h57928b3_0 conda-forge
libwebp-base 1.2.2 h8ffe710_1 conda-forge
libxcb 1.13 hcd874cb_1004 conda-forge
libxml2 2.9.12 hf5bbc77_1 conda-forge
libxslt 1.1.33 h65864e5_3 conda-forge
libzlib 1.2.11 h8ffe710_1013 conda-forge
lxml 4.8.0 py39h4fd7cdf_0 conda-forge
lz4-c 1.9.3 h8ffe710_1 conda-forge
m2w64-gcc-libgfortran 5.3.0 6 conda-forge
m2w64-gcc-libs 5.3.0 7 conda-forge
m2w64-gcc-libs-core 5.3.0 7 conda-forge
m2w64-gmp 6.1.0 2 conda-forge
m2w64-libwinpthread-git 5.0.0.4634.697f757 2 conda-forge
matplotlib-base 3.5.1 py39h581301d_0 conda-forge
mkl 2022.0.0 h0e2418a_796 conda-forge
mrio_common_metadata 0.1.1 py_0 cmutel
msys2-conda-epoch 20160418 1 conda-forge
multiprocess 0.70.12.2 py39hb82d6ee_1 conda-forge
multivolumefile 0.2.3 pyhd8ed1ab_0 conda-forge
munkres 1.1.4 pyh9f0ad1d_0 conda-forge
networkx 2.7.1 pyhd8ed1ab_0 conda-forge
numpy 1.22.3 py39h6331f09_0 conda-forge
openjpeg 2.4.0 hb211442_1 conda-forge
openpyxl 3.0.9 pyhd8ed1ab_0 conda-forge
openssl 1.1.1l h8ffe710_0 conda-forge
packaging 21.3 pyhd8ed1ab_0 conda-forge
pandas 1.4.1 py39h2e25243_0 conda-forge
pathos 0.2.8 pyhd8ed1ab_0 conda-forge
patsy 0.5.2 pyhd8ed1ab_0 conda-forge
peewee 3.14.10 py39h539719c_0 conda-forge
pillow 9.0.1 py39ha53f419_2 conda-forge
pip 22.0.4 pyhd8ed1ab_0 conda-forge
pox 0.3.0 pyhd8ed1ab_0 conda-forge
ppft 1.6.6.4 pyhd8ed1ab_0 conda-forge
presamples 0.2.3 h39e3cac_0 bsteubing
psutil 5.9.0 py39hb82d6ee_0 conda-forge
pthread-stubs 0.4 hcd874cb_1001 conda-forge
py7zr 0.18.1 pyhd8ed1ab_0 conda-forge
pybcj 0.5.0 py39hb82d6ee_2 conda-forge
pybcpy 0.0.17 pyhd8ed1ab_0 conda-forge
pycparser 2.21 pyhd8ed1ab_0 conda-forge
pycryptodomex 3.14.1 py39hb82d6ee_0 conda-forge
pyopenssl 22.0.0 pyhd8ed1ab_0 conda-forge
pypardiso 0.4.0 pyhd8ed1ab_0 conda-forge
pyparsing 3.0.7 pyhd8ed1ab_0 conda-forge
pyppmd 0.18.0 py39h415ef7b_0 conda-forge
pyprind 2.11.2 py39hcbf5309_1002 conda-forge
pyside2 5.13.2 py39hc084194_4 conda-forge
pysocks 1.7.1 py39hcbf5309_4 conda-forge
python 3.9.10 h9a09f29_2_cpython conda-forge
python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge
python-levenshtein 0.12.2 py39hb82d6ee_1 conda-forge
python_abi 3.9 2_cp39 conda-forge
pytz 2021.3 pyhd8ed1ab_0 conda-forge
pyzstd 0.15.1 py39h46500c5_0 conda-forge
qt 5.12.9 h5909a2a_4 conda-forge
requests 2.27.1 pyhd8ed1ab_0 conda-forge
salib 1.4.5 pyh6c4a22f_0 conda-forge
scipy 1.7.3 py39hc0c34ad_0 conda-forge
seaborn 0.11.2 hd8ed1ab_0 conda-forge
seaborn-base 0.11.2 pyhd8ed1ab_0 conda-forge
setuptools 60.10.0 py39hcbf5309_0 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
soupsieve 2.3.1 pyhd8ed1ab_0 conda-forge
sqlite 3.37.1 h8ffe710_0 conda-forge
stats_arrays 0.6.5 py_2 cmutel
statsmodels 0.13.2 py39h5d4886f_0 conda-forge
tabulate 0.8.9 pyhd8ed1ab_0 conda-forge
tbb 2021.5.0 h2d74725_0 conda-forge
texttable 1.6.4 pyhd8ed1ab_0 conda-forge
tk 8.6.12 h8ffe710_0 conda-forge
typing_extensions 4.1.1 pyha770c72_0 conda-forge
tzdata 2022a h191b570_0 conda-forge
ucrt 10.0.20348.0 h57928b3_0 conda-forge
unicodecsv 0.14.1 py_1 conda-forge
unicodedata2 14.0.0 py39hb82d6ee_0 conda-forge
unidecode 1.3.4 pyhd8ed1ab_0 conda-forge
urllib3 1.26.9 pyhd8ed1ab_0 conda-forge
vc 14.2 hb210afc_6 conda-forge
voluptuous 0.12.2 pyhd8ed1ab_1 conda-forge
vs2015_runtime 14.29.30037 h902a5da_6 conda-forge
wheel 0.37.1 pyhd8ed1ab_0 conda-forge
whoosh 2.7.4 py39hcbf5309_5 conda-forge
win_inet_pton 1.1.0 py39hcbf5309_3 conda-forge
wrapt 1.14.0 py39hb82d6ee_0 conda-forge
xlrd 1.2.0 pyh9f0ad1d_1 conda-forge
xlsxwriter 3.0.3 pyhd8ed1ab_0 conda-forge
xorg-libxau 1.0.9 hcd874cb_0 conda-forge
xorg-libxdmcp 1.1.3 hcd874cb_0 conda-forge
xz 5.2.5 h62dcd97_1 conda-forge
zipfile-deflate64 0.2.0 py39hb82d6ee_1 conda-forge
zipp 3.7.0 pyhd8ed1ab_1 conda-forge
zlib 1.2.11 h8ffe710_1013 conda-forge
zstd 1.5.2 h6255e5f_0 conda-forge
It would also help if you could share the database/databases and scenario files and what calculation setup you are using so we can try to replicate the problem.
superstructure database is created based on the IMAGE SSP2-Base default scenarios supplied with PREMISE. I am not sure if I can share these files publicly, as I had to request en encryption key form the developers to unlock the default scenarios (but I guess its okay if I email you the superstructure database excel file in case this helps the troubleshooting?)
a very simple calculation setup, which works if I export the transformed databases directly into AB instead of using the superstructure database
Currently in the proces of trying everything with ecoinvent 3.7.1 and 3.6 to see if that makes any difference.
Thanks for the info!
superstructure database is created based on the IMAGE SSP2-Base default scenarios supplied with PREMISE. I am not sure if I can share these files publicly, as I had to request en encryption key form the developers to unlock the default scenarios (but I guess its okay if I email you the superstructure database excel file in case this helps the troubleshooting?)
No worries, I should have that DB somewhere as well.
I see in the screenshot that you've used washing machince
from the ecoinvent 3.8 cutoff
database. Are you sure that is the right database in combination with those scenarios? In your database list on the left there are also some Remind
databases, it could be you're required to use those instead.
If that is indeed the problem, you may also want to include all of the premise scenarios in one super structure database to assess using different years in the same scenario (e.g. SSP2 base). For that, I'd recommend having a look at the premsie docs, as I don't know from the top of my head how to do that.
Hope that helps!
@simb-sdu If your problem is resolved, please let us know so we can close the issue.
@simb-sdu If your problem is resolved, please let us know so we can close the issue.
I have not been able to resolve the issue
1) was unable to produce superstructure databases for ecoinvent 3.6 and 3.7.1 to test if they work
2) created a new superstructure database for ecoinvent 3.8 cutoff. I am able to load the superstructure scenario file, but when calculating, i get the "AssertionError: Cannot find all keys"
3) i tried taking the EDIT: oops, my bad. I can now confirm that the superstructure database must be used for the reference flow as well
4) fresh install of activity browser environment and projects, as well as freshly imported databases made no differencewashing machine
reference flow from the Remind
databses - it made no difference, and in theory it should not make any difference. tried other reference flows as well, no difference
- created a new superstructure database for ecoinvent 3.8 cutoff. I am able to load the superstructure scenario file, but when calculating, i get the "AssertionError: Cannot find all keys"
Did you also update the reference flow to come from that new database?
- i tried taking the
washing machine
reference flow from theRemind
databses - it made no difference, and in theory it should not make any difference. tried other reference flows as well, no difference
Could you perhaps edit the scenario file to only contain a few (~10ish) edited flows and see if it is still broken? Sharing such a file should not be a licensing problem as the information contained is very limited. But sharing that would help me troubleshoot.
Superstructure database from update_steel()
ie. only steel sector from Remind
is transformed. With this database, there is no errors, and it works perfectly. See output below
(base) C:\Users\simb>conda activate ab_dev
(ab_dev) C:\Users\simb>activity-browser
-hiding tab: History
DevTools listening on ws://127.0.0.1:3999/devtools/browser/1e158180-a3a6-454e-b188-4f658de0ebd5
-hiding tab: Activity Details
-hiding tab: Characterization Factors
-hiding tab: Graph Explorer
-hiding tab: LCA results
Reset project settings directory to: C:\Users\simb\AppData\Local\pylca\Brightway3\default.c21f969b5f03d33d43e04f8f136e7682
Loading user settings:
dirpath already loaded
Reset project settings directory to: C:\Users\simb\AppData\Local\pylca\Brightway3\premise5.229fb8035d480137d0111334f1b0618c
Loaded project: premise5
Brightway2 data directory: C:\Users\simb\AppData\Local\pylca\Brightway3
Brightway2 active project: premise5
Qt Version: 5.12.9
Remote debugging server started successfully. Try pointing a Chromium-based browser to http://127.0.0.1:3999
2022-03-25T10:49:08.020ZE [7404:ShellIpcClient] shell_ipc_client.cc:129:Connect Can't connect to socket at: \\.\Pipe\GoogleDriveFSPipe_simb_shell
2022-03-25T10:49:08.023ZE [7404:ShellIpcClient] shell_ipc_client.cc:602:operator() Failed to connect to the server: Status::UNAVAILABLE_RESOURCE
2022-03-25T10:49:08.024ZE [26284:ShellIpcClient] shell_ipc_client.cc:129:Connect Can't connect to socket at: \\.\Pipe\GoogleDriveFSPipe_simb_shell
2022-03-25T10:49:08.024ZE [26284:ShellIpcClient] shell_ipc_client.cc:602:operator() Failed to connect to the server: Status::UNAVAILABLE_RESOURCE
2022-03-25T10:49:08.024ZE [25708:ShellIpcClient] shell_ipc_client.cc:129:Connect Can't connect to socket at: \\.\Pipe\GoogleDriveFSPipe_simb_shell
2022-03-25T10:49:08.025ZE [25708:ShellIpcClient] shell_ipc_client.cc:602:operator() Failed to connect to the server: Status::UNAVAILABLE_RESOURCE
Loading Scenario file. This may take a while for large files
Current shape and databases in the MetaDataStore: (0, 0) set()
Adding: biosphere3
Adding: super_db_2022-03-25
+showing tab: LCA results
Traceback (most recent call last):
File "C:\Users\simb\Anaconda3\envs\ab_dev\lib\site-packages\matplotlib\backends\backend_qt.py", line 347, in resizeEvent
self.figure.set_size_inches(winch, hinch, forward=False)
File "C:\Users\simb\Anaconda3\envs\ab_dev\lib\site-packages\matplotlib\figure.py", line 2662, in set_size_inches
raise ValueError(f'figure size must be positive finite not {size}')
ValueError: figure size must be positive finite not [ 0.4 -0.09]
Traceback (most recent call last):
File "C:\Users\simb\Anaconda3\envs\ab_dev\lib\site-packages\matplotlib\backends\backend_qt.py", line 347, in resizeEvent
self.figure.set_size_inches(winch, hinch, forward=False)
File "C:\Users\simb\Anaconda3\envs\ab_dev\lib\site-packages\matplotlib\figure.py", line 2662, in set_size_inches
raise ValueError(f'figure size must be positive finite not {size}')
ValueError: figure size must be positive finite not [ 0.4 -0.09]
Traceback (most recent call last):
File "C:\Users\simb\Anaconda3\envs\ab_dev\lib\site-packages\matplotlib\backends\backend_qt.py", line 347, in resizeEvent
self.figure.set_size_inches(winch, hinch, forward=False)
File "C:\Users\simb\Anaconda3\envs\ab_dev\lib\site-packages\matplotlib\figure.py", line 2662, in set_size_inches
raise ValueError(f'figure size must be positive finite not {size}')
ValueError: figure size must be positive finite not [ 0.34 -0.09]
Traceback (most recent call last):
File "C:\Users\simb\Anaconda3\envs\ab_dev\lib\site-packages\matplotlib\backends\backend_qt.py", line 347, in resizeEvent
self.figure.set_size_inches(winch, hinch, forward=False)
File "C:\Users\simb\Anaconda3\envs\ab_dev\lib\site-packages\matplotlib\figure.py", line 2662, in set_size_inches
raise ValueError(f'figure size must be positive finite not {size}')
ValueError: figure size must be positive finite not [ 0.34 -0.09]
Current shape and databases in the MetaDataStore: (25280, 30) {'biosphere3', 'super_db_2022-03-25'}
But the superstructure database with update_all()
ie. where all sectors from Remind
are transformed, gives the assertion error: cannot find all keys
see output below.
Adding: super_db_2022-03-24
Pandas Dataframe passed to sync. (0, 5)
Pandas Dataframe passed to sync. (149, 5)
Pandas Dataframe passed to sync. (0, 5)
Pandas Dataframe passed to sync. (0, 5)
Pandas Dataframe passed to sync. (0, 5)
Pandas Dataframe passed to sync. (3, 5)
Dropevent from: <activity_browser.ui.tables.inventory.ActivitiesBiosphereTable(0x2133bbbeae0) at 0x000002133708F640>
C:\Users\simb\Anaconda3\envs\ab_dev\lib\site-packages\activity_browser\ui\tables\models\lca_setup.py:151: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
self._dataframe = self._dataframe.append(data, ignore_index=True)
2022-03-25T10:51:26.266ZE [7404:ShellIpcClient] shell_ipc_client.cc:129:Connect Can't connect to socket at: \\.\Pipe\GoogleDriveFSPipe_simb_shell
2022-03-25T10:51:26.266ZE [7404:ShellIpcClient] shell_ipc_client.cc:602:operator() Failed to connect to the server: Status::UNAVAILABLE_RESOURCE
2022-03-25T10:51:26.269ZE [26284:ShellIpcClient] shell_ipc_client.cc:129:Connect Can't connect to socket at: \\.\Pipe\GoogleDriveFSPipe_simb_shell
2022-03-25T10:51:26.271ZE [26284:ShellIpcClient] shell_ipc_client.cc:602:operator() Failed to connect to the server: Status::UNAVAILABLE_RESOURCE
2022-03-25T10:51:26.273ZE [25708:ShellIpcClient] shell_ipc_client.cc:129:Connect Can't connect to socket at: \\.\Pipe\GoogleDriveFSPipe_simb_shell
2022-03-25T10:51:26.274ZE [25708:ShellIpcClient] shell_ipc_client.cc:602:operator() Failed to connect to the server: Status::UNAVAILABLE_RESOURCE
Loading Scenario file. This may take a while for large files
Traceback (most recent call last):
File "C:\Users\simb\Anaconda3\envs\ab_dev\lib\site-packages\activity_browser\layouts\tabs\LCA_setup.py", line 257, in scenario_calculation
'data': self.scenario_panel.combined_dataframe(),
File "C:\Users\simb\Anaconda3\envs\ab_dev\lib\site-packages\activity_browser\layouts\tabs\LCA_setup.py", line 424, in combined_dataframe
manager = SuperstructureManager(*data)
File "C:\Users\simb\Anaconda3\envs\ab_dev\lib\site-packages\activity_browser\bwutils\superstructure\manager.py", line 21, in __init__
SuperstructureManager.remove_duplicates(df)
File "C:\Users\simb\Anaconda3\envs\ab_dev\lib\site-packages\activity_browser\bwutils\superstructure\manager.py", line 109, in remove_duplicates
df.index = SuperstructureManager.build_index(df)
File "C:\Users\simb\Anaconda3\envs\ab_dev\lib\site-packages\activity_browser\bwutils\superstructure\manager.py", line 126, in build_index
assert df.loc[:, EXCHANGE_KEYS].notna().all().all(), "Cannot find all keys."
AssertionError: Cannot find all keys.
I will try to make a superstructure database for all sectors individually (like i did for the steel) to test them individually to try and lcoate the error.
- created a new superstructure database for ecoinvent 3.8 cutoff. I am able to load the superstructure scenario file, but when calculating, i get the "AssertionError: Cannot find all keys"
Did you also update the reference flow to come from that new database?
Yes. Although this is necessary in order to use the sueprstructure, it does not resolve the cannot find all keys
issue
Could you perhaps edit the scenario file to only contain a few (~10ish) edited flows and see if it is still broken? Sharing such a file should not be a licensing problem as the information contained is very limited. But sharing that would help me troubleshoot.
Thanks. I am not sure this will elicit the issue, if we consider my comment above. I will experiment more next week and get back!
Sorry, I really can't do something about the issue if I don't have all of the following:
With those datapoints I can actually see what is happening and try to replicate the problem, this terminal output is not helping me further at the moment. Unfortunately I can't really prove I an ecoinvent license, so that it would be 'safe' to share this, but I'm guessing you'll need to trust me 👍. If you could share your files with me here, it at least doesn't end up on the public internet.
Otherwise, I'm not sure how I could help you further.
problem was related to the genration of the sueprstructure database in PREMISE. #https://github.com/polca/premise/issues/65
I still get the key error when calculating scenario lca
I changed the max_rows value in line 472 of Anaconda3\envs\PREMISE\Lib\site-packages\pandas\io\formats\excel.py. don't know if this causes the error? I had increase the max allowed number of rows in order to make the superstructure database with PREMISE and ecoinvent 3.8 cutoff
Originally posted by @simb-sdu in https://github.com/LCA-ActivityBrowser/activity-browser/issues/432#issuecomment-1074218363