search

LCA-ActivityBrowser / activity-browser

GUI for brightway2
GNU Lesser General Public License v3.0
139 stars 54 forks source link

We should freeze dependencies in stable version #953

Closed Pan6ora closed 1 year ago

Pan6ora commented 1 year ago

Idea

Having "x or newer" dependencies (ex: brightway2 >=2.4.2) doesn't seem like a good practice as it can break everything on dependencies updates.

There is the discussion about versioning ( #899 ) so what we consider a "stable" version may change in the future. But I think we should at least ensure the AB in conda-forge is always downloadable by freezing dependencies versions (at least in a working range).

Context

Few days ago one of the developers in my laboratory reported not being able to get AB from conda.

Indeed the install failed because of broken dependencies in pyside2 and qt-webengine (on a clean Debian VM). I solved it by downloading these two packages in a sane version before installing AB:

conda install -c conda-forge pyside2=5.15.5 qt-webengine=5.15.4
conda install -c conda-forge activity-browser

I've created a pull request to freeze these two depedencies version (see #952 ), but I think we should avoid systematically this kind of issues.

Zoophobus commented 1 year ago

Hi @Pan6ora, sorry I'm probably being a bit stupid, I don't have much experience with VMs, but I was wondering why conda couldn't resolve this using the glib versions on conda-forge?

Pan6ora commented 1 year ago

I mentioned using a VM because it means the problem is reproducible (it is not specific to our config, the OS was a fresh Debian install).

why conda couldn't resolve this using the glib versions on conda-forge?

I don't really now what the problem is but this is a possible scenario : Suppose that we have the following dependencies :

activity-browser  
  | pyside2 >= 5.15.4
  | some_other_package

pyside 5.15.4
  | glibc 2.17

pyside 5.15.8
  | glibc 2.31

some_other_package
  | glibc 2.17

When pyside 5.15.4 was the last version, no problem : both pyside2 and some_other_package want glibc 2.17.

But now that pyside 5.15.8 exists, and because AB wants pyside2 >= 5.15.4 we have a problem: conda gets the last version of pyside2 but can't find a glibc version matching both pyside2 and some_other_package dependencies.

My understanding is that this is the kind of problem happening.

Zoophobus commented 1 year ago

OK, I hope you can bear with my stupidity for a bit longer. What's really confusing me is that while I get the same behaviour with either conda version 22.11 and 23.1 in the latest version of Ubuntu, it works perfectly in the older OSs.

Is that normal for an issue like this?

Actually this would have had to happen within the last week, because I've been checking the installation whenever there's a related user issue

Zoophobus commented 1 year ago

BTW, there's no question I'll accept your PR and push this to conda-forge pronto

Pan6ora commented 1 year ago

it works perfectly in the older OSs. Is that normal for an issue like this?

I don't really know. I don't have a full understanding of the problem and no time to dig it at the moment. The main point for me was to fix the install quickly.

Libraries are a bit of a mess sometime, because programs may assume they are already in our system path. Maybe this is the case here, and Ubuntu updated glibc in latest versions ?

marc-vdm commented 1 year ago

I'll just leave this here relevant comic

Zoophobus commented 1 year ago

Well it's definitely not working at the moment, this is not just isolated to Ubuntu or Linux and is a much wider problem. @m-rossi do you have any input on this?

m-rossi commented 1 year ago

@Pan6ora Weird problem. Could you provide the output of conda info and conda list for a working and a not-working activity-browser installation?

Zoophobus commented 1 year ago

@m-rossi and @Pan6ora

So there's been getting a bit of difficulty the last two days for a windows user trying to install the activity browser. In the end the dependency checking process took around two hours to generate a working system, with the current conda-forge setup (on a modern system, although with 8GB of RAM).

This is not, I would consider, a satisfactory situation for new users. To try and help I've been going through different configurations of the dependency file and checking the efficiency on both windows (11) and Linux (Ubuntu 22.04) systems. All I've really managed to find is that loosening the restrictions on either qt-webengine, or pyside2 have had the best effect.

The configuration file I've been using is (with changes to the version comparisons): python >=3.8,<3.10 arrow brightway2 >=2.4.2 pyperclip eidl >=1.4.2 networkx pyside2 >=5.15.5 qt-webengine =5.15.4 numpy =1.23.5 salib >=1.4 seaborn The "working" environment then, unless I'm absolutely stupid would contain the following plus, of course, the latest version of the AB (this is in Ubuntu 22.04) `# packages in environment at /home/zoo/anaconda3/envs/sansAB: #

Name Version Build Channel

_libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 2_kmp_llvm conda-forge alsa-lib 1.2.8 h166bdaf_0 conda-forge appdirs 1.4.4 pyh9f0ad1d_0 conda-forge arrow 1.2.3 pyhd8ed1ab_0 conda-forge asteval 0.9.29 pyhd8ed1ab_0 conda-forge astunparse 1.6.3 pyhd8ed1ab_0 conda-forge attr 2.5.1 h166bdaf_1 conda-forge beautifulsoup4 4.12.0 pyha770c72_0 conda-forge brightway2 2.4.3 pyhd8ed1ab_0 conda-forge brotli 1.0.9 h166bdaf_8 conda-forge brotli-bin 1.0.9 h166bdaf_8 conda-forge brotli-python 1.0.9 py39h5a03fae_8 conda-forge brotlicffi 1.0.9.2 py39h5a03fae_3 conda-forge brotlipy 0.7.0 py39hb9d737c_1005 conda-forge bw2analyzer 0.10 pyhd8ed1ab_0 conda-forge bw2calc 1.8.2 py39hf3d152e_0 conda-forge bw2data 3.6.6 pyhd8ed1ab_0 conda-forge bw2io 0.8.8 pyhd8ed1ab_1 conda-forge bw2parameters 0.7 pyhd8ed1ab_0 conda-forge bw_migrations 0.2 pyhd8ed1ab_0 conda-forge bzip2 1.0.8 h7f98852_4 conda-forge ca-certificates 2022.12.7 ha878542_0 conda-forge cairo 1.16.0 ha61ee94_1014 conda-forge certifi 2022.12.7 pyhd8ed1ab_0 conda-forge cffi 1.15.1 py39he91dace_3 conda-forge charset-normalizer 2.1.1 pyhd8ed1ab_0 conda-forge click 8.1.3 unix_pyhd8ed1ab_2 conda-forge contourpy 1.0.7 py39h4b4f3f3_0 conda-forge cryptography 40.0.1 py39h079d5ae_0 conda-forge cycler 0.11.0 pyhd8ed1ab_0 conda-forge dbus 1.13.6 h5008d03_3 conda-forge dill 0.3.6 pyhd8ed1ab_1 conda-forge docopt 0.6.2 py_1 conda-forge eidl 1.5.1 pyhd8ed1ab_1 conda-forge eight 1.0.1 pyhd8ed1ab_4 conda-forge et_xmlfile 1.1.0 pyhd8ed1ab_0 conda-forge expat 2.5.0 h27087fc_0 conda-forge fasteners 0.17.3 pyhd8ed1ab_0 conda-forge fftw 3.3.10 nompi_hf0379b8_106 conda-forge flask 2.2.3 pyhd8ed1ab_0 conda-forge font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge font-ttf-inconsolata 3.000 h77eed37_0 conda-forge font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge font-ttf-ubuntu 0.83 hab24e00_0 conda-forge fontconfig 2.14.2 h14ed4e7_0 conda-forge fonts-conda-ecosystem 1 0 conda-forge fonts-conda-forge 1 0 conda-forge fonttools 4.39.3 py39h72bdee0_0 conda-forge freetype 2.12.1 hca18f0e_1 conda-forge future 0.18.3 pyhd8ed1ab_0 conda-forge gettext 0.21.1 h27087fc_0 conda-forge giflib 5.2.1 h0b41bf4_3 conda-forge glib 2.74.1 h6239696_1 conda-forge glib-tools 2.74.1 h6239696_1 conda-forge gmp 6.2.1 h58526e2_0 conda-forge graphite2 1.3.13 h58526e2_1001 conda-forge gst-plugins-base 1.21.3 h4243ec0_1 conda-forge gstreamer 1.21.3 h25f0c4b_1 conda-forge gstreamer-orc 0.4.33 h166bdaf_0 conda-forge harfbuzz 6.0.0 h8e241bc_0 conda-forge icu 70.1 h27087fc_0 conda-forge idna 3.4 pyhd8ed1ab_0 conda-forge importlib-metadata 6.1.0 pyha770c72_0 conda-forge importlib-resources 5.12.0 pyhd8ed1ab_0 conda-forge importlib_metadata 6.1.0 hd8ed1ab_0 conda-forge importlib_resources 5.12.0 pyhd8ed1ab_0 conda-forge inflate64 0.3.1 py39hb9d737c_3 conda-forge itsdangerous 2.1.2 pyhd8ed1ab_0 conda-forge jack 1.9.22 h11f4161_0 conda-forge jinja2 3.1.2 pyhd8ed1ab_1 conda-forge jpeg 9e h0b41bf4_3 conda-forge keyutils 1.6.1 h166bdaf_0 conda-forge kiwisolver 1.4.4 py39hf939315_1 conda-forge krb5 1.20.1 h81ceb04_0 conda-forge lame 3.100 h166bdaf_1003 conda-forge lcms2 2.15 hfd0df8a_0 conda-forge ld_impl_linux-64 2.40 h41732ed_0 conda-forge lerc 4.0.0 h27087fc_0 conda-forge libblas 3.9.0 16_linux64_openblas conda-forge libbrotlicommon 1.0.9 h166bdaf_8 conda-forge libbrotlidec 1.0.9 h166bdaf_8 conda-forge libbrotlienc 1.0.9 h166bdaf_8 conda-forge libcap 2.67 he9d0100_0 conda-forge libcblas 3.9.0 16_linux64_openblas conda-forge libclang 15.0.7 default_had23c3d_1 conda-forge libclang13 15.0.7 default_h3e3d535_1 conda-forge libcups 2.3.3 h36d4200_3 conda-forge libdb 6.2.32 h9c3ff4c_0 conda-forge libdeflate 1.17 h0b41bf4_0 conda-forge libedit 3.1.20191231 he28a2e2_2 conda-forge libevent 2.1.10 h28343ad_4 conda-forge libffi 3.4.2 h7f98852_5 conda-forge libflac 1.4.2 h27087fc_0 conda-forge libgcc-ng 12.2.0 h65d4601_19 conda-forge libgcrypt 1.10.1 h166bdaf_0 conda-forge libgfortran-ng 12.2.0 h69a702a_19 conda-forge libgfortran5 12.2.0 h337968e_19 conda-forge libglib 2.74.1 h606061b_1 conda-forge libgpg-error 1.46 h620e276_0 conda-forge libhwloc 2.9.0 hd6dc26d_0 conda-forge libiconv 1.17 h166bdaf_0 conda-forge liblapack 3.9.0 16_linux64_openblas conda-forge libllvm15 15.0.7 hadd5161_1 conda-forge libnsl 2.0.0 h7f98852_0 conda-forge libogg 1.3.4 h7f98852_1 conda-forge libopenblas 0.3.21 pthreads_h78a6416_3 conda-forge libopus 1.3.1 h7f98852_1 conda-forge libpng 1.6.39 h753d276_0 conda-forge libpq 15.2 hb675445_0 conda-forge libsndfile 1.2.0 hb75c966_0 conda-forge libsqlite 3.40.0 h753d276_0 conda-forge libstdcxx-ng 12.2.0 h46fd767_19 conda-forge libsystemd0 253 h8c4010b_1 conda-forge libtiff 4.5.0 h6adf6a1_2 conda-forge libtool 2.4.7 h27087fc_0 conda-forge libudev1 253 h0b41bf4_1 conda-forge libuuid 2.32.1 h7f98852_1000 conda-forge libvorbis 1.3.7 h9c3ff4c_0 conda-forge libwebp 1.2.4 h1daa5a0_1 conda-forge libwebp-base 1.2.4 h166bdaf_0 conda-forge libxcb 1.13 h7f98852_1004 conda-forge libxkbcommon 1.5.0 h79f4944_1 conda-forge libxml2 2.10.3 hca2bb57_4 conda-forge libxslt 1.1.37 h873f0b0_0 conda-forge libzlib 1.2.13 h166bdaf_4 conda-forge llvm-openmp 16.0.0 h417c0b6_0 conda-forge lxml 4.9.2 py39h14694de_0 conda-forge lz4-c 1.9.4 hcb278e6_0 conda-forge markupsafe 2.1.2 py39h72bdee0_0 conda-forge matplotlib-base 3.7.1 py39he190548_0 conda-forge mkl 2023.0.0 h84fe81f_26648 conda-forge mpg123 1.31.3 hcb278e6_0 conda-forge mrio_common_metadata 0.2.1 pyhd8ed1ab_0 conda-forge multiprocess 0.70.14 py39hb9d737c_3 conda-forge multivolumefile 0.2.3 pyhd8ed1ab_0 conda-forge munkres 1.1.4 pyh9f0ad1d_0 conda-forge mysql-common 8.0.32 ha901b37_1 conda-forge mysql-libs 8.0.32 hd7da12d_1 conda-forge ncurses 6.3 h27087fc_1 conda-forge networkx 3.0 pyhd8ed1ab_0 conda-forge nspr 4.35 h27087fc_0 conda-forge nss 3.89 he45b914_0 conda-forge numpy 1.23.5 py39h3d75532_0 conda-forge openjpeg 2.5.0 hfec8fc6_2 conda-forge openpyxl 3.1.1 py39h72bdee0_0 conda-forge openssl 3.1.0 h0b41bf4_0 conda-forge packaging 23.0 pyhd8ed1ab_0 conda-forge pandas 1.5.3 py39h2ad29b5_0 conda-forge pathos 0.3.0 pyhd8ed1ab_0 conda-forge patsy 0.5.3 pyhd8ed1ab_0 conda-forge pcre2 10.40 hc3806b6_0 conda-forge peewee 3.16.0 py39h02e676a_0 conda-forge pillow 9.4.0 py39h2320bf1_1 conda-forge pip 23.0.1 pyhd8ed1ab_0 conda-forge pixman 0.40.0 h36c2ea0_0 conda-forge platformdirs 3.2.0 pyhd8ed1ab_0 conda-forge pooch 1.7.0 pyha770c72_3 conda-forge pox 0.3.2 pyhd8ed1ab_0 conda-forge ppft 1.7.6.6 pyhd8ed1ab_0 conda-forge psutil 5.9.4 py39hb9d737c_0 conda-forge pthread-stubs 0.4 h36c2ea0_1001 conda-forge pulseaudio 16.1 hcb278e6_3 conda-forge pulseaudio-client 16.1 h5195f5e_3 conda-forge pulseaudio-daemon 16.1 ha8d29e2_3 conda-forge py7zr 0.20.4 pyhd8ed1ab_0 conda-forge pybcj 1.0.1 py39hb9d737c_2 conda-forge pybcpy 0.0.17 pyhd8ed1ab_0 conda-forge pycparser 2.21 pyhd8ed1ab_0 conda-forge pycryptodomex 3.16.0 py39hb9d737c_0 conda-forge pyopenssl 23.1.1 pyhd8ed1ab_0 conda-forge pypardiso 0.4.2 pyhd8ed1ab_1 conda-forge pyparsing 3.0.9 pyhd8ed1ab_0 conda-forge pyperclip 1.8.2 pyhd8ed1ab_2 conda-forge pyppmd 1.0.0 py39h5a03fae_1 conda-forge pyprind 2.11.2 py39hf3d152e_1004 conda-forge pyside2 5.15.8 py39hc68acd0_2 conda-forge pysocks 1.7.1 pyha2e5f31_6 conda-forge python 3.9.16 h2782a2a_0_cpython conda-forge python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python_abi 3.9 3_cp39 conda-forge pytz 2023.3 pyhd8ed1ab_0 conda-forge pyxlsb 1.0.10 pyhd8ed1ab_0 conda-forge pyzstd 0.15.4 py39h52f73a1_0 conda-forge qt-main 5.15.6 h602db52_6 conda-forge qt-webengine 5.15.4 h325cec9_3 conda-forge readline 8.2 h8228510_1 conda-forge requests 2.28.2 pyhd8ed1ab_0 conda-forge salib 1.4.7 pyhd8ed1ab_0 conda-forge scipy 1.10.1 py39h7360e5f_0 conda-forge seaborn 0.12.2 hd8ed1ab_0 conda-forge seaborn-base 0.12.2 pyhd8ed1ab_0 conda-forge setuptools 67.6.1 pyhd8ed1ab_0 conda-forge setuptools-scm 7.1.0 pyhd8ed1ab_0 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge soupsieve 2.3.2.post1 pyhd8ed1ab_0 conda-forge stats_arrays 0.6.5 pyhd8ed1ab_0 conda-forge statsmodels 0.13.5 py39h2ae25f5_2 conda-forge tabulate 0.9.0 pyhd8ed1ab_1 conda-forge tbb 2021.8.0 hf52228f_0 conda-forge texttable 1.6.7 pyhd8ed1ab_0 conda-forge tk 8.6.12 h27826a3_0 conda-forge tomli 2.0.1 pyhd8ed1ab_0 conda-forge typing-extensions 4.5.0 hd8ed1ab_0 conda-forge typing_extensions 4.5.0 pyha770c72_0 conda-forge tzdata 2023c h71feb2d_0 conda-forge unicodecsv 0.14.1 py_1 conda-forge unicodedata2 15.0.0 py39hb9d737c_0 conda-forge unidecode 1.3.6 pyhd8ed1ab_0 conda-forge urllib3 1.26.15 pyhd8ed1ab_0 conda-forge voluptuous 0.13.1 pyhd8ed1ab_0 conda-forge werkzeug 2.2.3 pyhd8ed1ab_0 conda-forge wheel 0.40.0 pyhd8ed1ab_0 conda-forge whoosh 2.7.4 py39hf3d152e_7 conda-forge wrapt 1.15.0 py39h72bdee0_0 conda-forge xcb-util 0.4.0 h166bdaf_0 conda-forge xcb-util-image 0.4.0 h166bdaf_0 conda-forge xcb-util-keysyms 0.4.0 h166bdaf_0 conda-forge xcb-util-renderutil 0.3.9 h166bdaf_0 conda-forge xcb-util-wm 0.4.1 h166bdaf_0 conda-forge xkeyboard-config 2.38 h0b41bf4_0 conda-forge xlrd 2.0.1 pyhd8ed1ab_3 conda-forge xlsxwriter 3.0.9 pyhd8ed1ab_0 conda-forge xorg-kbproto 1.0.7 h7f98852_1002 conda-forge xorg-libice 1.0.10 h7f98852_0 conda-forge xorg-libsm 1.2.3 hd9c2040_1000 conda-forge xorg-libx11 1.8.4 h0b41bf4_0 conda-forge xorg-libxau 1.0.9 h7f98852_0 conda-forge xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge xorg-libxext 1.3.4 h0b41bf4_2 conda-forge xorg-libxrender 0.9.10 h7f98852_1003 conda-forge xorg-renderproto 0.11.1 h7f98852_1002 conda-forge xorg-xextproto 7.3.0 h0b41bf4_1003 conda-forge xorg-xproto 7.0.31 h7f98852_1007 conda-forge xz 5.2.6 h166bdaf_0 conda-forge zipfile-deflate64 0.2.0 py39hb9d737c_3 conda-forge zipp 3.15.0 pyhd8ed1ab_0 conda-forge zlib 1.2.13 h166bdaf_4 conda-forge zstd 1.5.2 h3eb15da_6 conda-forge The output ofconda infofor this environment is: active environment : sansAB active env location : /home/zoo/anaconda3/envs/sansAB shell level : 2 user config file : /home/zoo/.condarc populated config files : /home/zoo/.condarc conda version : 23.3.0 conda-build version : 3.24.0 python version : 3.9.16.final.0 virtual packages : __archspec=1=x86_64 glibc=2.35=0 linux=5.19.0=0 __unix=0=0 base environment : /home/zoo/anaconda3 (writable) conda av data dir : /home/zoo/anaconda3/etc/conda conda av metadata url : None channel URLs : https://conda.anaconda.org/conda-forge/linux-64 https://conda.anaconda.org/conda-forge/noarch https://repo.anaconda.com/pkgs/main/linux-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/r/linux-64 https://repo.anaconda.com/pkgs/r/noarch https://conda.anaconda.org/cmutel/linux-64 https://conda.anaconda.org/cmutel/noarch https://conda.anaconda.org/bsteubing/linux-64 https://conda.anaconda.org/bsteubing/noarch package cache : /home/zoo/anaconda3/pkgs /home/zoo/.conda/pkgs envs directories : /home/zoo/anaconda3/envs /home/zoo/.conda/envs platform : linux-64 user-agent : conda/23.3.0 requests/2.28.2 CPython/3.9.16 Linux/5.19.0-38-generic ubuntu/22.04.2 glibc/2.35 UID:GID : 1000:1000 netrc file : None offline mode : False `

Pan6ora commented 1 year ago

@Pan6ora Weird problem. Could you provide the output of conda info and conda list for a working and a not-working activity-browser installation?

Not really because it's the same installation. The only difference was that I had to install qt-webengine 5.15.4 and pyside2 5.15.5 before installing activity-browser. Without that conda would complain about a dependency problem.

In the end the dependency checking process took around two hours to generate a working system

Two hours and you can't be sure if it's going to work ! In my first message it was the same but at the end conda finally complained about not being able to solve dependencies.

I have personally moved to micromamba which is way better at solving dependencies. It never took more than 20s, and the Solving environment: failed with initial frozen solve. Retrying with flexible solve. thing never happens. Not a solution for users but if you want to test if there is a real dependency problem or not without waiting 2 hours for conda to find it this is a great tool.

m-rossi commented 1 year ago

Not really because it's the same installation. The only difference was that I had to install qt-webengine 5.15.4 and pyside2 5.15.5 before installing activity-browser. Without that conda would complain about a dependency problem.

Important information, I thought the issue only appears when try to run activity-browser then. Nevertheless, could you provide conda info and conda list for your environment?

I have personally moved to micromamba which is way better at solving dependencies. It never took more than 20s, and the Solving environment: failed with initial frozen solve. Retrying with flexible solve. thing never happens. Not a solution for users but if you want to test if there is a real dependency problem or not without waiting 2 hours for conda to find it this is a great tool.

Indeed. All our company stuff is running on mamba instead of conda. Although I will try to find out what's causing the dependency issues there. Normally creation of an environment with conda create -n ab activity-browser should be fast too. The problems arise when you try to install it in your main environment.

m-rossi commented 1 year ago

Quick update from my side: I do not see any issues or exceptional long solving times when using conda instead of mamba on my Windows machines or a Linux server (glibc=2.27) I have access to.

I will do a few more experiments on more recent ubuntu/debian and therefore glibc versions.

bsteubing commented 1 year ago

@m-rossi do you know how to roll back version on conda-forge? I think we should actually go back to a working version of the dependencies (if this is possible) and to give the users a way to install the AB (even if not the latest version right now)

And thanks a lot @m-rossi and @Pan6ora for supporting the search for a solution here!!

m-rossi commented 1 year ago

@m-rossi do you know how to roll back version on conda-forge? I think we should actually go back to a working version of the dependencies (if this is possible) and to give the users a way to install the AB (even if not the latest version right now)

There are ways to remove broken packages. But as I said above, currently I am unable to reproduce these issues so I do not want to remove the current version at this point.

bsteubing commented 1 year ago

@m-rossi: I have the same issues on my computer (solving taking too long, I just aborted after 15 min)

So I would suggest testing if the recipe of the last version on conda-forge (or the last working version that we know of) still works and if so, reverting to that in the meantime... I think we should always have a version on conda-forge that we know works...

haasad commented 1 year ago

Hi all, dependency resolution for AB 2.7.5 isn't completely broken, it just takes forever (between 10 minutes on Ubuntu to 20 minutes on Windows, see here). I quickly built a github action pipeline which installs the AB according to the instructions every 24h. This should help to identify broken dependency resolution early #957

haasad commented 1 year ago

I assume the strict pinning here is to blame for the ultra-slow solving of the environment: https://github.com/LCA-ActivityBrowser/activity-browser/blob/aa1864da39676649ed2a8e01ba9aa785cf0571d0/ci/recipe/stable/meta.yaml#L30-L32

In my experience pinning to an exact version always leads to longer dependency resolution and it's usually just a matter of time until it breaks completely. If you want to provide users with a known working environment you have the "freeze" every single dependency and not just a few. I.e. we could provide a full conda environment file like the artifacts we already produce in the normal pipeline (e.g. here)

bsteubing commented 1 year ago

Hi @haasad thanks for this! So the current problem seems to be that installation is working, but takes so long that it is not really practical and users might abort early (at least it is not nice at all). The solution you present, as I understand it, would let us know a) if the dependency resolution is broken and b) how long it took (but I guess only by looking into the actions summary "manually")

This seems to be a useful step, but perhaps not yet the solution to our current problem...

Does anyone know if the installation would work (again/better) IF we switched back to one of the previous versions on conda-forge? I ask since I would give priority to seamless installation over featuring the latest version.

bsteubing commented 1 year ago

Oh and since I see mamba mentioned here several times. Would you think it could make sense to offer a version of the AB via mamba? Or even in the mid-term move to mamba? I have no experience so far with this.

haasad commented 1 year ago

b) how long it took (but I guess only by looking into the actions summary "manually")

We can define an upper limit of what is acceptable (e.g. 10 min) and add a timeout which would make the pipeline step fail, which in turn would send a notification about the failing pipeline

This seems to be a useful step, but perhaps not yet the solution to our current problem...

Yes, this doesn't solve the problem. Just trying to bring some clarity to the situation

marc-vdm commented 1 year ago

Some comments cleaning the reqs:

- python >=3.8,<3.10 --> 3.8 was relevant when 3.7 was still used often, I think we can drop that part @haasad will know abt 3.10 I think?
- arrow --> I don't know what this is or why we have it
- brightway2 >=2.4.2 --> I think we could bump this to 2.4.3, that will fix the numpy req below
- pyperclip --> required for copy-paste on debian based linux distros
- eidl >=1.4.2 --> @haasad Do you know is this can be updated?
- networkx --> required for Sankey
- pyside2 =5.15.5 --> @Zoophobus  knows about this
- qt-webengine =5.15.4 --> @Zoophobus  knows about this
- numpy =1.23.5 --> Can be removed once we move to bw 2.4.3
- salib >=1.4 --> Required for Global Sensitivity Analysis, can possibly be relaxed --> @Zoophobus check please
- seaborn --> I don't think we use this right now?
Zoophobus commented 1 year ago

@haasad Your canary is a fantastic idea and can really help us to fix these issues before they become too problematic.

@marc-vdm seaborn is actually still used for the uncertainties so we can't remove it pyside2 =5.15.5 is due to changes in the latest versions of debian based distros qt-webengine =5.15.4 is also due to changes in the latest versions of debian based distros

(Changing pyside2 and qt-webengine to looser conditions is what I've been looking at ineffectively)

bsteubing commented 1 year ago

If the restrictions on pyside and qt-webengine are only affecting debian, could we not relax this and add a note for debian users on how they currently need to fix their installation? I think most users are on Windows, so what worries me is not being able to provide an installable (not in reasonable time) version of the AB, so hopefully we can fix this soon

m-rossi commented 1 year ago

In my experience pinning to an exact version always leads to longer dependency resolution and it's usually just a matter of time until it breaks completely.

Indeed.

If you want to provide users with a known working environment you have the "freeze" every single dependency and not just a few. I.e. we could provide a full conda environment file like the artifacts we already produce in the normal pipeline

This would also mean to use a separate conda environment for activity-browser. I think those issues we see here are mostly related to the fact that users install activity-browser in there base environment where you are battling with other dependencies.

Does anyone know if the installation would work (again/better) IF we switched back to one of the previous versions on conda-forge? I ask since I would give priority to seamless installation over featuring the latest version.

The issue we discuss here are due to changes on the os side. I guess older versions would also break there.

Oh and since I see mamba mentioned here several times. Would you think it could make sense to offer a version of the AB via mamba? Or even in the mid-term move to mamba? I have no experience so far with this.

mamba is a drop-in replacement for conda. You can install it via conda:

conda install mamba

and use mamba then everytime you would normally use conda.

haasad commented 1 year ago

I suspect that if we can drop the pin for numpy this would give the biggest improvement, because so many of the AB's dependencies directly or indirectly depend on numpy.

For pyside2 and qt-webengine you should probably look for a combination that works with the same version number as they seem to use the versioning of qt-main. Having strict pins for different versions sounds very broken to me.

For python I wouldn't rush. Tests for 3.10 still segfault

m-rossi commented 1 year ago

In my experience lower bounds are not a big problem.

haasad commented 1 year ago

This would also mean to use a separate conda environment for activity-browser. I think those issues we see here are mostly related to the fact that users install activity-browser in there base environment where you are battling with other dependencies.

The installation instructions already suggest to install the AB in a separate environment. And the dependency resolution takes forever in a new environment, not only in an existing one.

@bsteubing Until 3 days ago the README also encouraged users to install the AB in its own conda env.

In my experience lower bounds are not a big problem.

Agreed