Closed maximlt closed 1 year ago
skupr-anaconda
commented
1 year ago holoviews-1.14.9 pyhd8ed1ab_0 is installing with bokeh-3.0.2 pyhd8ed1ab_0
holoviews-1.15.2 pyhd8ed1ab_0 is installing with bokeh-2.4.3 pyhd8ed1ab_3
Both results because the recipe was updated for v1.15.2 https://github.com/conda-forge/holoviews-feedstock/commit/0d10f2159bdfd618b984b26bb69e87963ea43eb0
But the correct one should be holoviews-1.15.2 pyhd8ed1ab_0 installing with bokeh-2.4.3 pyhd8ed1ab_3
maximlt
commented
1 year ago But the correct one should be holoviews-1.15.2 pyhd8ed1ab_0 installs with bokeh-2.4.3 pyhd8ed1ab_3
I'm not sure that there is a "correct" one, both are valid solutions. What I'd expect is conda to always return the same solution :) (But yes I prefer the solution you suggest as the other one needs to install a pretty old version of Panel (a dependency of HoloViews) that didn't have any upper pin for Bokeh.)
jezdez
commented
1 year ago @maximlt I'm just acknowledging that @jaimergp and me have seen and discussed this briefly before he left for PTO and that it was not enough time to work on before the currently being prepared conda release happened. Appreciate the filing, though!
jaimergp
commented
1 year ago I cannot reproduce this on linux-64 or osx-64 (using CONDA_SUBDIR appropiately). Maybe because of https://github.com/conda-forge/conda-forge-repodata-patches-feedstock/pull/357 we don't have the necessary packaging landscape?
This is what I get:
$ python determinism.py
[('noarch', '2.4.3')]
[('noarch', '2.4.3')]
[('noarch', '2.4.3')]
[('noarch', '2.4.3')]
[('noarch', '2.4.3')]
[('noarch', '2.4.3')]
[('noarch', '2.4.3')]
[('noarch', '2.4.3')]
[('noarch', '2.4.3')]
[('noarch', '2.4.3')]Yes certainly. Do you know if there's a way to download a repodata.json file that dates from before that patch?
jaimergp
commented
1 year ago Technically we can:
<channel>/<subdir>/repodata_from_packages.json.However I tried the unpatched repodata directly (with --repodata-fn repodata_from_packages.json) and it gave the same results. So unless it's the patching that introduce the multi-solve problem, then I don't know if we can reproduce.
hoxbro
commented
1 year ago Try to set holoviews==1.15.2. Holoviews 1.15.3 have put a lower pin on panel.
hoxbro
commented
1 year ago I can still get a non-deterministic solution with the following code, but only for conda+libmamba. When running mamba, I get a deterministic solution.
import re
import subprocess
cmd = 'conda create --experimental-solver libmamba -n testtesttest -c conda-forge --override-channels python=3.8 bokeh hvplot --dry-run --offline'
pattern = r'\/(.+)::bokeh-(\d+.\d+.\d+)'
for _ in range(10):
result = subprocess.run(cmd.split(' '), capture_output=True, text=True)
print(re.findall(pattern, result.stdout))
cmd = 'mamba create -n testtesttest -c conda-forge --override-channels python=3.8 bokeh hvplot --dry-run --offline'
pattern = r"bokeh\s+(\d+.\d+.\d+).+?conda-forge\/(.+?)\s"
for _ in range(10):
result = subprocess.run(cmd.split(' '), capture_output=True, text=True)
print(re.findall(pattern, result.stdout))[('noarch', '3.0.3')]
[('noarch', '3.0.3')]
[('noarch', '2.4.3')]
[('noarch', '3.0.3')]
[('noarch', '2.4.3')]
[('noarch', '3.0.3')]
[('noarch', '3.0.3')]
[('noarch', '3.0.3')]
[('noarch', '3.0.3')]
[('noarch', '3.0.3')][('3.0.3', 'noarch')]
[('3.0.3', 'noarch')]
[('3.0.3', 'noarch')]
[('3.0.3', 'noarch')]
[('3.0.3', 'noarch')]
[('3.0.3', 'noarch')]
[('3.0.3', 'noarch')]
[('3.0.3', 'noarch')]
[('3.0.3', 'noarch')]
[('3.0.3', 'noarch')]I noticed that if I swap the order of hvplot and bokeh and use this command: mamba create -n testtesttest -c conda-forge --override-channels python=3.8 hvplot bokeh --dry-run --offline.
The output for mamba is still non-deterministic, but the solve gives bokeh=2.4.3. So maybe some sorting of the packages is done by mamba but not with conda+libmamba.
jaimergp
commented
1 year ago This is excellent @Hoxbro, now I can look into this and see where the non-determinism is coming from. Thank you so much!
jaimergp
commented
1 year ago I still get deterministic results:
python -I determinism.py
[('noarch', '2.4.3')]
[('noarch', '2.4.3')]
[('noarch', '2.4.3')]
[('noarch', '2.4.3')]
[('noarch', '2.4.3')]
[('noarch', '2.4.3')]
[('noarch', '2.4.3')]
[('noarch', '2.4.3')]
[('noarch', '2.4.3')]
[('noarch', '2.4.3')]Maybe it's a version-dependent thing? My base environment contains:
conda 22.11.1 <dev> # from current main
conda-libmamba-solver 22.12.0 <dev> # from current main
libmamba 1.0.0 ha06983f_0 defaults
libmambapy 1.0.0 py39ha06983f_0 defaults
libsolv 0.7.22 he621ea3_0 defaultsWe haven't done much work on the solver input side lately so I don't think the dev versions have much to do. Can you report your version and channels?
hoxbro
commented
1 year ago My base environment:
conda 22.11.1 py310hff52083_1 conda-forge
conda-libmamba-solver 22.12.0 pyhd8ed1ab_0 conda-forge
libmamba 1.1.0 hde2b089_3 conda-forge
libmambapy 1.1.0 py310h1428755_3 conda-forge
libsolv 0.7.23 h3eb15da_0 conda-forge conda info conda list cat ~/.condarc 
costrouc
commented
1 year ago Hey @maximlt I'm going to work on investigating this issue. Thanks for the detailed information and @Hoxbro appreciate the example to reproduce this issue :smile:
costrouc
commented
1 year ago To reproduce I have run the steps in https://github.com/conda/conda-libmamba-solver/blob/main/docs/dev/setup.md using the docker container.
active environment : base
active env location : /opt/conda
shell level : 1
user config file : /home/test_user/.condarc
populated config files :
conda version : 23.1.0
conda-build version : 3.23.3
python version : 3.9.15.final.0
virtual packages : __archspec=1=x86_64
__glibc=2.31=0
__linux=6.0.7=0
__unix=0=0
base environment : /opt/conda (read only)
conda av data dir : /opt/conda/etc/conda
conda av metadata url : None
channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
package cache : /opt/conda/pkgs
/home/test_user/.conda/pkgs
envs directories : /home/test_user/.conda/envs
/opt/conda/envs
platform : linux-64
user-agent : conda/23.1.0 requests/2.28.1 CPython/3.9.15 Linux/6.0.7 debian/11 glibc/2.31
UID:GID : 1001:1001
netrc file : None
offline mode : False
Using the python script above in :
conda@main(1d53d2953) and conda-libmamba-solver@main(568fbd8) I get [('noarch', '2.4.3')] 10 timesconda@23.1.0 and conda-libmamba-solver@22.12.0 I get [('noarch', '2.4.3')] 10 timesconda@22.11.1 and conda-libmamba-solver@22.12.0 I get [('noarch', '2.4.3')] 10 timesconda@22.9.0 and conda-libmamba-solver@22.9.0 I get [('noarch', '3.0.3')] 10 timesconda@22.9.0 and conda-libmamba-solver@22.6.0 I get [('noarch', '3.0.3')] 10 times (interestingly the solve was MUCH slower when I tested this tag).When I ran the docker container script again for each combination I would occasionally get 2.4.3 10 times and other times 3.0.3 10 times but it was always consistent within a singledocker run ....
So in all I haven't been able to reproduce yet.
jaimergp
commented
1 year ago So conda-libmamba-solver <= 22.8.1 (I guess this is the version you mean by 22.9.0) gives 3.0.3, but 22.12 gives 2.4.3 🤔 There are no big changes in the solver logic itself between the two releases (but they might be pinning different libmambas and libsolvs, we should also report these in the table). I'd take a look at #52 (shouldn't have an effect, these are fresh envs!) and #78 (nothing major catches my attention) for code changes but, again, I don't see how those have an effect on this. I think it's more about which libmamba versions are running with each.
What happens if we sample more than 10 attempts? Let's say 50?
PS: 22.6 is expected to be slower due to the "is there a new conda update?" checks.
costrouc
commented
1 year ago So conda-libmamba-solver <= 22.8.1 (I guess this is the version you mean by 22.9.0) gives 3.0.3, but 22.12 gives 2.4.3
@jaimergp I wouldn't read into it like that. I was getting 2.4.3 on some runs and 3.0.3 on some runs. I should have been more clear :smile:
costrouc
commented
1 year ago So I was able to reproduce this.
[('noarch', '2.4.3')]
[('noarch', '3.0.3')]
[('noarch', '3.0.3')]
[('noarch', '2.4.3')]
[('noarch', '2.4.3')]
[('noarch', '3.0.3')]
[('noarch', '3.0.3')]
[('noarch', '3.0.3')]
[('noarch', '3.0.3')]
[('noarch', '2.4.3')]I went another route and just used github actions to see if I could reproduce the behavior to test linux/macos/windows as well and I could. Here is a reproducing run https://github.com/costrouc/conda-libmamba-solver-reproducibility/actions/runs/3964342931/jobs/6793117312#step:6:1
The actual workflow is here https://github.com/costrouc/conda-libmamba-solver-reproducibility/blob/master/.github/workflows/reproducible.yaml. I'll spend some time later trying the same this with checking out the git source for both projects.
costrouc
commented
1 year ago A full action running. I can reproduce the error:
Github actions doing the test https://github.com/costrouc/conda-libmamba-solver-reproducibility/actions/runs/3964442719/jobs/6793306009.
costrouc
commented
1 year ago Added git repo testing checking out the actual releases https://github.com/costrouc/conda-libmamba-solver-reproducibility/actions/runs/3964684703/jobs/6793761850#step:8:51 and can't reproduce it that way. This matches what you saw @jaimergp as well.
jaimergp
commented
1 year ago This is great news, @costrouc! So to summarize:
Differences I can think of between these contexts (OD=Outside-Docker, ID=Inside-Docker):
-c conda-forge?jaimergp
commented
1 year ago Ugh, yes, the initialization scripts in the Docker image are setting PYTHONHASHSEED T_T
I think it comes from conda.core.initialize.initialize_dev (see this line). So I am sure it will reproduce in Docker if we do not set a seed. Clear the env var before running the tests and it should check out.
Details:
(base) test_user@8bb8f7890a4f:/opt/conda-libmamba-solver-src$ env | sort
CONDA_DEFAULT_ENV=base
CONDA_EXE=/opt/conda/bin/conda
CONDA_PREFIX=/opt/conda
CONDA_PROMPT_MODIFIER=(base)
CONDA_PYTHON_EXE=/opt/conda/bin/python
CONDA_SHLVL=1
HOME=/home/test_user
HOSTNAME=8bb8f7890a4f
LANG=C.UTF-8
LC_ALL=C.UTF-8
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
OLDPWD=/opt/conda-src
PATH=/opt/conda/bin:/opt/conda/condabin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/opt/conda-libmamba-solver-src
PYTHONHASHSEED=4243125599
PYTHON_MAJOR_VERSION=3
RUNNING_ON_DEVCONTAINER=
SHLVL=0
TERM=xterm
TEST_PLATFORM=unix
_=/usr/bin/env
_CE_CONDA=
_CE_M=Edit - eureka:
[('noarch', '3.0.3')]
[('noarch', '3.0.3')]
[('noarch', '3.0.3')]
[('noarch', '3.0.3')]
[('noarch', '2.4.3')]
[('noarch', '3.0.3')]
[('noarch', '3.0.3')]
[('noarch', '3.0.3')]
[('noarch', '3.0.3')]
[('noarch', '2.4.3')]Overall I think this is good news, because it means that the problem is on the Python side, so it should be very easy to fix. It must be a container without order guarantees, like a set or something.
jaimergp
commented
1 year ago The intriguing part is that libsolv seems to be sensitive to the input order o.o I wonder if we need to make sure and sort everything before passing things to libmamba, because python=3.8 bokeh>=1.0 holoviews>=1.1.0 might give different results than python=3.8 holoviews>=1.1.0 bokeh>=1.0.
maximlt
commented
1 year ago So nice to this being fixed, thanks so much guys for all the hard work reproducing the issue and fixing it! 👍 That was holding us, we're free to move to the libmamba world now :)
The intriguing part is that libsolv seems to be sensitive to the input order o.o I wonder if we need to make sure and sort everything before passing things to libmamba, because python=3.8 bokeh>=1.0 holoviews>=1.1.0 might give different results than python=3.8 holoviews>=1.1.0 bokeh>=1.0.
I would say that users with enough knowledge to understand that the SAT solver is a complicated thing to deal with wouldn't be overly surprised by this. All the others yes :)
jaimergp
commented
1 year ago We added tests to control for that too so we should be ok. The input specs were being sorted as they came, but then that order was lost before reaching libmamba because they were de-duped with a set(). Thank you so much for the report and patience!
LtDan33
commented
1 year ago @maximlt Thanks for reporting this and glad it helped you out!
Checklist
What happened?
The same command - installing
python=3.8 bokeh>=1.0 holoviews>=1.1.0in a new environment - ends up installing different package versions. I'd expect the solve the provide the same output, given the same input?Here's a script to reproduce that:
Output:
The output shows that conda may install Bokeh 2.4.3 or 3.0.2. This is not the only version that changes but I chose to highlight just one to reveal what I think is the issue, i.e. that the same command could lead to different environments.
(The latest conda-forge HoloViews version - 1.15.2 - pins Bokeh to
<3)Conda Info
Conda Config
No response
Conda list
Additional Context
No response