LCA-ActivityBrowser / activity-browser

GUI for brightway2
GNU Lesser General Public License v3.0
152 stars 58 forks source link

Ecoinvent 3.10 import #1318

Closed dvivas-eth closed 3 months ago

dvivas-eth commented 5 months ago

Updating AB

What happened?

Even after the last update, I am not able to download ecoinvent 3.10 (the latest I can download is ecoinvent 3.9.1), from the Database Import Wizard, when importing from ecoinvent online directly. I believe the issue is that it puts 3.10 as an older version than the rest (since it starts with 1 and not 9), and hence it does not appear in the scroll-out options.

I am quite bad with coding unfortunately so I do not understand what the latest update of Activity-Browser, 3 weeks ago, did in order to solve this issue.

I tried updating everything several times but it did not help unfortunately.

Relevant errors

No response

Operating system

Windows 10

Conda environment

(base) C:\Users\dvivas>conda activate ab

(ab) C:\Users\dvivas>conda list
# packages in environment at C:\Users\dvivas\AppData\Local\anaconda3\envs\ab:
#
# Name                    Version                   Build  Channel
activity-browser          2.9.7              pyhd8ed1ab_0    conda-forge
appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
arrow                     1.3.0              pyhd8ed1ab_0    conda-forge
asteval                   0.9.31             pyhd8ed1ab_0    conda-forge
astunparse                1.6.3              pyhd8ed1ab_0    conda-forge
blinker                   1.7.0              pyhd8ed1ab_0    conda-forge
blis                      0.9.0                hcfcfb64_1    conda-forge
brightway2                2.4.4              pyhd8ed1ab_0    conda-forge
brotli                    1.1.0                hcfcfb64_1    conda-forge
brotli-bin                1.1.0                hcfcfb64_1    conda-forge
brotli-python             1.1.0            py39h99910a6_1    conda-forge
brotlicffi                1.1.0.0          py39h99910a6_1    conda-forge
bw2analyzer               0.10               pyhd8ed1ab_0    conda-forge
bw2calc                   1.8.2            py39hcbf5309_0    conda-forge
bw2data                   3.6.6              pyhd8ed1ab_0    conda-forge
bw2io                     0.8.10             pyhd8ed1ab_0    conda-forge
bw2parameters             1.1.0              pyhd8ed1ab_0    conda-forge
bw_migrations             0.2                pyhd8ed1ab_0    conda-forge
bzip2                     1.0.8                hcfcfb64_5    conda-forge
ca-certificates           2024.6.2             h56e8100_0    conda-forge
certifi                   2024.6.2           pyhd8ed1ab_0    conda-forge
cffi                      1.16.0           py39ha55989b_0    conda-forge
charset-normalizer        3.3.2              pyhd8ed1ab_0    conda-forge
click                     8.1.7           win_pyh7428d3b_0    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
contourpy                 1.2.0            py39h1f6ef14_0    conda-forge
cycler                    0.12.1             pyhd8ed1ab_0    conda-forge
dill                      0.3.8              pyhd8ed1ab_0    conda-forge
docopt                    0.6.2                      py_1    conda-forge
eidl                      2.0.1              pyhd8ed1ab_0    conda-forge
eight                     1.0.1              pyhd8ed1ab_4    conda-forge
et_xmlfile                1.1.0              pyhd8ed1ab_0    conda-forge
fasteners                 0.17.3             pyhd8ed1ab_0    conda-forge
flask                     3.0.2              pyhd8ed1ab_0    conda-forge
fonttools                 4.49.0           py39ha55989b_0    conda-forge
freetype                  2.12.1               hdaf720e_2    conda-forge
future                    1.0.0              pyhd8ed1ab_0    conda-forge
gettext                   0.21.1               h5728263_0    conda-forge
glib                      2.78.4               h12be248_0    conda-forge
glib-tools                2.78.4               h12be248_0    conda-forge
gst-plugins-base          1.22.9               h001b923_0    conda-forge
gstreamer                 1.22.9               hb4038d2_0    conda-forge
icu                       73.2                 h63175ca_0    conda-forge
idna                      3.6                pyhd8ed1ab_0    conda-forge
importlib-metadata        7.0.1              pyha770c72_0    conda-forge
importlib-resources       6.1.2              pyhd8ed1ab_0    conda-forge
importlib_metadata        7.0.1                hd8ed1ab_0    conda-forge
importlib_resources       6.1.2              pyhd8ed1ab_0    conda-forge
inflate64                 1.0.0            py39ha55989b_1    conda-forge
intel-openmp              2024.0.0         h57928b3_49841    conda-forge
itsdangerous              2.1.2              pyhd8ed1ab_0    conda-forge
jinja2                    3.1.3              pyhd8ed1ab_0    conda-forge
kiwisolver                1.4.5            py39h1f6ef14_1    conda-forge
krb5                      1.21.2               heb0366b_0    conda-forge
lcms2                     2.16                 h67d730c_0    conda-forge
lerc                      4.0.0                h63175ca_0    conda-forge
libblas                   3.9.0             21_win64_blis    conda-forge
libbrotlicommon           1.1.0                hcfcfb64_1    conda-forge
libbrotlidec              1.1.0                hcfcfb64_1    conda-forge
libbrotlienc              1.1.0                hcfcfb64_1    conda-forge
libcblas                  3.9.0             21_win64_blis    conda-forge
libclang                  15.0.7          default_h77d9078_3    conda-forge
libclang13                15.0.7          default_h77d9078_3    conda-forge
libdeflate                1.19                 hcfcfb64_0    conda-forge
libffi                    3.4.2                h8ffe710_5    conda-forge
libglib                   2.78.4               h16e383f_0    conda-forge
libhwloc                  2.9.1                h51c2c0f_0    conda-forge
libiconv                  1.17                 hcfcfb64_2    conda-forge
libjpeg-turbo             3.0.0                hcfcfb64_1    conda-forge
liblapack                 3.9.0           5_hd5c7e75_netlib    conda-forge
libogg                    1.3.4                h8ffe710_1    conda-forge
libpng                    1.6.43               h19919ed_0    conda-forge
libsqlite                 3.45.1               hcfcfb64_0    conda-forge
libtiff                   4.6.0                h6e2ebb7_2    conda-forge
libvorbis                 1.3.7                h0e60522_0    conda-forge
libwebp                   1.3.2                hcfcfb64_1    conda-forge
libwebp-base              1.3.2                hcfcfb64_0    conda-forge
libxcb                    1.15                 hcd874cb_0    conda-forge
libxml2                   2.10.4               hc3477c8_0    conda-forge
libxslt                   1.1.37               h0192164_0    conda-forge
libzlib                   1.2.13               hcfcfb64_5    conda-forge
lxml                      4.9.2            py39h0942119_0    conda-forge
m2w64-gcc-libgfortran     5.3.0                         6    conda-forge
m2w64-gcc-libs            5.3.0                         7    conda-forge
m2w64-gcc-libs-core       5.3.0                         7    conda-forge
m2w64-gmp                 6.1.0                         2    conda-forge
m2w64-libwinpthread-git   5.0.0.4634.697f757               2    conda-forge
markupsafe                2.1.5            py39ha55989b_0    conda-forge
matplotlib-base           3.8.3            py39hf19769e_0    conda-forge
mkl                       2024.0.0         h66d3029_49658    conda-forge
mrio_common_metadata      0.2.1              pyhd8ed1ab_0    conda-forge
msys2-conda-epoch         20160418                      1    conda-forge
multiprocess              0.70.16          py39ha55989b_0    conda-forge
multivolumefile           0.2.3              pyhd8ed1ab_0    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
networkx                  3.2.1              pyhd8ed1ab_0    conda-forge
numpy                     1.23.5           py39hbccbffa_0    conda-forge
openjpeg                  2.5.2                h3d672ee_0    conda-forge
openpyxl                  3.1.2            py39ha55989b_1    conda-forge
openssl                   3.3.1                h2466b09_1    conda-forge
packaging                 23.2               pyhd8ed1ab_0    conda-forge
pandas                    2.1.4            py39h32e6231_0    conda-forge
pathos                    0.3.2              pyhd8ed1ab_1    conda-forge
patsy                     0.5.6              pyhd8ed1ab_0    conda-forge
pcre2                     10.42                h17e33f8_0    conda-forge
peewee                    3.17.1           py39habf9222_0    conda-forge
pillow                    10.2.0           py39h368b509_0    conda-forge
pint                      0.21               pyhd8ed1ab_0    conda-forge
pip                       24.0               pyhd8ed1ab_0    conda-forge
ply                       3.11                       py_1    conda-forge
pox                       0.3.4              pyhd8ed1ab_0    conda-forge
ppft                      1.7.6.8            pyhd8ed1ab_0    conda-forge
presamples                0.2.7              pyhd8ed1ab_1    conda-forge
psutil                    5.9.8            py39ha55989b_0    conda-forge
pthread-stubs             0.4               hcd874cb_1001    conda-forge
pthreads-win32            2.9.1                hfa6e2cd_3    conda-forge
py7zr                     0.20.8             pyhd8ed1ab_1    conda-forge
pybaselines               1.1.0                    pypi_0    pypi
pybcj                     1.0.2            py39ha55989b_0    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pycryptodomex             3.19.0           py39ha55989b_1    conda-forge
pypardiso                 0.4.3              pyhd8ed1ab_0    conda-forge
pyparsing                 3.1.1              pyhd8ed1ab_0    conda-forge
pyperclip                 1.8.2              pyhd8ed1ab_2    conda-forge
pyppmd                    1.1.0            py39h99910a6_0    conda-forge
pyprind                   2.11.2          py39hcbf5309_1005    conda-forge
pyqt                      5.15.9           py39hb77abff_5    conda-forge
pyqt5-sip                 12.12.2          py39h99910a6_5    conda-forge
pyqtwebengine             5.15.9           py39h2f4a3f1_5    conda-forge
pyside2                   5.15.8           py39h56cbfbc_2    conda-forge
pysocks                   1.7.1              pyh0701188_6    conda-forge
python                    3.9.18          h4de0772_1_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-tzdata             2024.1             pyhd8ed1ab_0    conda-forge
python_abi                3.9                      4_cp39    conda-forge
pytz                      2024.1             pyhd8ed1ab_0    conda-forge
pyxlsb                    1.0.10             pyhd8ed1ab_0    conda-forge
pyzstd                    0.15.9           py39h95af829_1    conda-forge
qt-main                   5.15.8              h9e85ed6_19    conda-forge
qt-webengine              5.15.8               h4bf5c4e_4    conda-forge
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
salib                     1.4.8              pyhd8ed1ab_0    conda-forge
scipy                     1.12.0           py39hddb5d58_2    conda-forge
seaborn                   0.13.2               hd8ed1ab_0    conda-forge
seaborn-base              0.13.2             pyhd8ed1ab_0    conda-forge
setuptools                69.1.1             pyhd8ed1ab_0    conda-forge
setuptools-scm            8.0.4              pyhd8ed1ab_0    conda-forge
sip                       6.7.12           py39h99910a6_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
stats_arrays              0.6.6              pyhd8ed1ab_0    conda-forge
statsmodels               0.14.1           py39hd88c2e4_0    conda-forge
tabulate                  0.9.0              pyhd8ed1ab_1    conda-forge
tbb                       2021.9.0             h91493d7_0    conda-forge
texttable                 1.7.0              pyhd8ed1ab_0    conda-forge
tk                        8.6.13               h5226925_1    conda-forge
toml                      0.10.2             pyhd8ed1ab_0    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
tqdm                      4.66.2             pyhd8ed1ab_0    conda-forge
types-python-dateutil     2.8.19.20240106    pyhd8ed1ab_0    conda-forge
typing-extensions         4.10.0               hd8ed1ab_0    conda-forge
typing_extensions         4.10.0             pyha770c72_0    conda-forge
tzdata                    2024a                h0c530f3_0    conda-forge
ucrt                      10.0.22621.0         h57928b3_0    conda-forge
unicodecsv                0.14.1             pyhd8ed1ab_2    conda-forge
unicodedata2              15.1.0           py39ha55989b_0    conda-forge
unidecode                 1.3.8              pyhd8ed1ab_0    conda-forge
urllib3                   2.2.1              pyhd8ed1ab_0    conda-forge
vc                        14.3                hcf57466_18    conda-forge
vc14_runtime              14.38.33130         h82b7239_18    conda-forge
voluptuous                0.14.2             pyhd8ed1ab_0    conda-forge
vs2015_runtime            14.38.33130         hcb4865c_18    conda-forge
werkzeug                  3.0.1              pyhd8ed1ab_0    conda-forge
wheel                     0.42.0             pyhd8ed1ab_0    conda-forge
whoosh                    2.7.4            py39hcbf5309_8    conda-forge
win_inet_pton             1.1.0              pyhd8ed1ab_6    conda-forge
wrapt                     1.16.0           py39ha55989b_0    conda-forge
xlrd                      2.0.1              pyhd8ed1ab_3    conda-forge
xlsxwriter                3.1.9              pyhd8ed1ab_0    conda-forge
xorg-libxau               1.0.11               hcd874cb_0    conda-forge
xorg-libxdmcp             1.1.3                hcd874cb_0    conda-forge
xz                        5.2.6                h8d14728_0    conda-forge
zipfile-deflate64         0.2.0            py39ha55989b_4    conda-forge
zipp                      3.17.0             pyhd8ed1ab_0    conda-forge
zstd                      1.5.5                h12be248_0    conda-forge
mrvisscher commented 5 months ago

Hi @dvivas-eth,

Right now it's only possible to install ecoinvent 3.10 into a new project through the setup project wizard. You cannot download ecoinvent 3.10 into an existing project because the biosphere version will not match and there is no way to patch right now.

Kind regards,

Marin

mrvisscher commented 5 months ago

Also, on further inspection I see that you're still running 2.9.7. You can't update like usual because your environment is using Python 3.9, which isn't supported bij Activity Browser 2.10. Please update using the following commands instead:

conda create -n ab -c conda-forge --solver libmamba activity-browser

And confirm when the updater asks you if you want to overwrite your environment.

This will overwrite the old environment, but your projects and other data will be untouched.

Kind regards,

Marin

LetinoisU commented 5 months ago

Dear Marin, that is also for me very helpful, I managed to upgrade to Activity-browser 2.10 and import Eco-Invent 3.10. Now the question I have is: How can I now import into my new project (which uses EcoInvent 3.10) my former own database which I created in Activity-browser 2.9.7? I have database back-ups as brightway2 file AB 2.10 import of own database trouble shoot.docx and as xlsx-file. but when I try to import one of these files, AB runs idle. [Uploading AB 2.10 import of own database trouble shoot.docx…]()

mrvisscher commented 5 months ago

Hi @LetinoisU

I don't see a direct error related to this in your file. Are you sure you are giving it enough time? Importing a database may take some time.

Kind regards,

Marin

LetinoisU commented 5 months ago

confidential

Dear Marin,

It looks like some re-linking does not work. I manually solved one re-linking error, it was related to an “oxygen, liquid entry” by deleting in my old project the “oxygen-model” in one activity and putting again oxygen, liquid into the model. @.*** But after doing so, creating again a new back-up and trying to re-import it into my EI 3.10 project, I got another exchange-error:

@.***

Is there a general method to work with the re-linking issues?

Kind regards,

Ulla

From: Marin Visscher @.> Sent: Monday, July 1, 2024 15:47 To: LCA-ActivityBrowser/activity-browser @.> Cc: Ulla Letinois @.>; Mention @.> Subject: Re: [LCA-ActivityBrowser/activity-browser] Ecoinvent 3.10 import (Issue #1318)

CAUTION: This email originated from outside dsm-firmenich. DO NOT CLICK on links and DO NOT OPEN attachments unless you recognize the sender and know the content is safe. If you think it is suspicious, report this email using Report Suspicious Email button available (once the email is opened) in the Microsoft Outlook ribbon above.

Hi @LetinoisUhttps://github.com/LetinoisU

I don't see a direct error related to this in your file. Are you sure you are giving it enough time? Importing a database may take some time.

Kind regards,

Marin

— Reply to this email directly, view it on GitHubhttps://github.com/LCA-ActivityBrowser/activity-browser/issues/1318#issuecomment-2200205401, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BFDKUDJFVF3RC5KK7R2TLMLZKFMT7AVCNFSM6AAAAABKB7LWDKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBQGIYDKNBQGE. You are receiving this because you were mentioned.Message ID: @.**@.>>


DISCLAIMER: This email and any files transmitted with it, including replies and forwarded copies subsequently transmitted from dsm-firmenich, are confidential and for the intended recipient only. If you have received this email by mistake then please notify the sender immediately and then delete it from your system; access, disclosure, copying, distribution or reliance on any of it by anyone else is prohibited.


DISCLAIMER: This email and any files transmitted with it, including replies and forwarded copies subsequently transmitted from dsm-firmenich, are confidential and for the intended recipient only. If you have received this email by mistake then please notify the sender immediately and then delete it from your system; access, disclosure, copying, distribution or reliance on any of it by anyone else is prohibited.

mrvisscher commented 5 months ago

Hi @LetinoisU,

Quick message to let you know that replying to a GitHub thread is public, even if you reply via e-mail: https://github.com/LCA-ActivityBrowser/activity-browser/issues/1318#issuecomment-2202124535

You can mail me at: m.r.visscher@cml.leidenuniv.nl if you want to share something non-publicly

Kind regards,

Marin

KellyC-IBP commented 3 months ago

Hello, I am a beginner in Brightway and activity browser, I am trying to import the ecoinvent database into Activity Browser but have yet to succeed. I have made sure that my Activity Browseer is the newest version 2.10. From the posts I have gathered the problem of compatibility of Activity Browser with EcoInvent 3.10 is already solved?

I have the Ecoinvent 3.10 as .ecospold 2 and .xlsx file format stored in the network (so not local computer). I have tried different ways in importing the database, like downloading the files to my local computer in .7z or .zip format.

Can anyone please tell me how to proceed? Thank you in advance for your support.

mrvisscher commented 3 months ago

Hi @KellyC-IBP ,

Do you get any errors in the terminal (the black screen with the text)? That would be very helpful for us to diagnose what's going wrong.

Kind regards,

Marin

mrvisscher commented 3 months ago

This may well be related to #1321 by the way. Right now importing the .z7 directly is broken, but unpacking the file first and then importing should work as usual.

KellyC-IBP commented 3 months ago

Hi Marin,

Thank you for your reply, this is very helpful. I’ll try and see if it works.

Best, Kelly

From: Marin Visscher @.> Sent: Tuesday, August 6, 2024 3:38 PM To: LCA-ActivityBrowser/activity-browser @.> Cc: Chang, Hsin-Yu @.>; Mention @.> Subject: Re: [LCA-ActivityBrowser/activity-browser] Ecoinvent 3.10 import (Issue #1318)

This may well be related to #1321https://github.com/LCA-ActivityBrowser/activity-browser/issues/1321 by the way. Right now importing the .z7 directly is broken, but unpacking the file first and then importing should work as usual.

— Reply to this email directly, view it on GitHubhttps://github.com/LCA-ActivityBrowser/activity-browser/issues/1318#issuecomment-2271318327, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BKKNC72ETS6L7FHHMFD4EY3ZQDGTXAVCNFSM6AAAAABKB7LWDKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENZRGMYTQMZSG4. You are receiving this because you were mentioned.Message ID: @.***>

KellyC-IBP commented 3 months ago

Hi, Turns out my initial problem was that I chose the wrong file for ecoinvent (the larger '_lci_ecospold2' one)

I am encountering the same problem as above mentioned when I try to import EcoInvent 3.10 into my project. image

When I create a new project, I always have to import biosphere 3 first ('set up your project with default data'), how can I get around this problem?

Thanks!

mrvisscher commented 3 months ago

Okay, right, I should've seen this coming.

At this moment, you cannot install only the biosphere3 for ecoinvent 3.10 through the Activity Browser. In technical terms: it is not available in bw2io as a direct install, and can only be installed through the ecoinvent_interface library, which will download it directly for you from the ecoinvent servers.

This means the following for you: the only way to install the right version of biosphere3, with the right methods, that is compatible with ecoinvent 3.10 is through the project setup wizard. In there you can choose to setup your project with ecoinvent 3.10 after you've logged in with your ecoinvent credentials.

This setup is still quite slow, so hang in there. But it should setup your project with everything you need.

Kind regards,

Marin

KellyC-IBP commented 3 months ago

Hi Marin,

thank you for your reply. Unfortunately I do not have ecoinvent login credentials, but I have .ecospold2 files in the server.

I also tried to import EcoInvent 3.10 to Brightway25, but there are many unlinked exchanges. The solutions to this also suggested using EcoInvent credentials. I am quite confused right now regarding the different options of Activity Browser, Brightway 2 and Brightway 25 and the individual issues they have with EcoInvent 3.10.

Can you help me on deciding which option to go for and how to proceed more smoothly? Just to be clear on my situation again:

Have:

Don't have:

Thank you so much for your support Best, Kelly

mrvisscher commented 3 months ago

Hi @KellyC-IBP ,

The situation is indeed confusing right now, this is something we'll need to work on. Having the credentials would make the process a lot easier. Not having them means you're going to have to write your own Python code to do this.

What you need

First things first, you'll need to at least have: the EcoInvent 3.10 ecospold2 .7z that contains both the datasets and MasterData folders, and the LCIA Implementation file (excel), that contains the EcoInvent 3.10 methods.

Importing databases

Importing the biosphere and ecoinvent databases is doable, you can use the Ecospold2BiosphereImporter and SingleOutputEcospold2Importer that are part of bw2io. Point them to the right files/folders of your unpacked EcoInvent 3.10.

Importing methods

Now, the methods, that's another story. I checked out bw2io, and right now there doesn't seem to be a straightforward way to import the LCIA Implementation 3.10.xlsx file. If I were you, I'd write my own function, taking inspiration from the EcoinventLCIAImporter and the import_ecoinvent_release function in bw2io.ecoinvent.

Final remark

I know this is quite a lot, which shows that we need to do better in this regard.

Hopefully this helps in some way!

Kind regards,

Marin