search

conda-forge / miniforge

A conda-forge distribution.
https://conda-forge.org/miniforge
Other
6.43k stars 326 forks source link

issue with condaforge/mambaforge latest 4.14.0-0 #350

Closed mieczyslaw closed 2 years ago

mieczyslaw commented 2 years ago

Solution to issue cannot be found in the documentation.

Issue

I am sorry if this is not the correct place for docker image but it is difficult to find a way to report it via docker hub and it is not obvious how docker image is related to this repo. Otherwise, please advise where to send this bug report.

The latest version of condaforge/mambaforge Docker image (4.14.0-0) is broken. Shows:

OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /opt/conda/lib/python3.10/site-packages/certifi/cacert.pem

when installing python 3.8 or 3.9 to the base env (so uninstalling base 3.10) and the whole build crashes. It seems like an old path to python 3.10 is left within requests package when downgrading to 3.8 or 3.9. There is no issue with 4.13.0-1 (the version before latest).

My Dockerfile:

FROM condaforge/mambaforge
COPY . /package
RUN mamba env update -n base -f /package/environment.yml && mamba clean --all -y

My environment.yml in the package:

channels:
  - conda-forge
dependencies:
  - python==3.9

Installed packages

This is dockerbuild, so I cannot easily provide it.

Environment info

This is dockerbuild, so I cannot easily provide it.
hmaarrfk commented 2 years ago

you shouldn't be trying to install python 3.9 in base. I've updated base to use 3.10. You should likely create a new environment for yourself.

mieczyslaw commented 2 years ago

Yes, separate env would be great but there are issues with activating it. When you call

mamba activate envname

It asks for mamba init. When you call it before activating, it tells you to close and open shell, but you can't do that during dockerbuild.

I tried instead to source /root/.bashrc, with . /root/.bashrc (as source command doesn't exist in this shell), but it didn't help:

 => CACHED [4/8] COPY . /package                                                                                         0.0s
 => CACHED [5/8] RUN mamba env update -n compute -f /package/environment.yml && mamba clean --all -y                     0.0s
 => CACHED [6/8] RUN mamba init                                                                                          0.0s
 => CACHED [7/8] RUN . /root/.bashrc                                                                                     0.0s
 => ERROR [8/8] RUN mamba activate compute                                                                               0.6s
------
 > [8/8] RUN mamba activate compute:
#12 0.500 Run 'mamba init' to be able to run mamba activate/deactivate
#12 0.500 and start a new shell session. Or use conda to activate/deactivate.
#12 0.500

(the same issue is with conda activate).

I also tried another trick/hack to change shell, init mamba, exit it and enter again, but there is the same issue with activation:

 => [ 5/10] RUN bash                                                                                                     0.2s
 => [ 6/10] RUN mamba init                                                                                               0.7s
 => [ 7/10] RUN exit                                                                                                     0.2s
 => [ 8/10] RUN bash                                                                                                     0.2s
 => [ 9/10] RUN mamba env update -n compute -f /package/environment.yml && mamba clean --all -y                        146.0s
 => ERROR [10/10] RUN mamba activate compute                                                                             0.8s
------
 > [10/10] RUN mamba activate compute:
#14 0.708 Run 'mamba init' to be able to run mamba activate/deactivate
#14 0.708 and start a new shell session. Or use conda to activate/deactivate.
#14 0.708

Another way was to source what is added by mamba init, but it didn't help neither:

=> CACHED [4/7] COPY . /package                                                                                                                                                                                                                         0.0s
 => CACHED [5/7] RUN mamba env update -n compute -f /package/environment.yml && mamba clean --all -y                                                                                                                                                     0.0s
 => [6/7] RUN . "/opt/conda/etc/profile.d/mamba.sh"                                                                                                                                                                                                      0.2s
 => ERROR [7/7] RUN mamba activate compute                                                                                                                                                                                                               0.6s
------
 > [7/7] RUN mamba activate compute:
#10 0.580 Run 'mamba init' to be able to run mamba activate/deactivate
#10 0.580 and start a new shell session. Or use conda to activate/deactivate.
#10 0.580

There are some discussions online on how to activate conda in dockerbuild process, the recommendation was to use base env.

Do you have then any recipe how to make the working docker image of the tool using Python 3.8 or 3.9 when starting with your latest mambaforge image? I mean, how to activate a custom environment during build process?

Thanks for your help.

hmaarrfk commented 2 years ago

Does this docker image help? https://github.com/conda-forge/miniforge-images/blob/master/ubuntu/Dockerfile

raybellwaves commented 2 years ago

Local

Testing local behavior (conda installed locally using miniconda; later installed mamba. conda: 22.9.0; mamba 0.24.0. note I believe mamba 0.25.0 is python 3.10 only)

$ conda create -n test_env python=3.10 --y && conda activate test_env
$ echo "name: main
channels:
  - nodefaults
  - conda-forge
dependencies:
  - python=3.9" > env.yml
$ conda env update --name test_env --file env.yml

Creating a python 3.10 environment then updating it to 3.9 works

Same works above using mamba:

$ mamba create -n test_env python=3.10 --y && conda activate test_env
$ mamba env update --name test_env --file env.yml

condaforge/mambaforge

Doing this in the condaforge/mambaforge image fails:

$ docker run -it condaforge/mambaforge /bin/bash
$ echo "name: main
channels:
  - nodefaults
  - conda-forge
dependencies:
  - python=3.9" > env.yml
$ conda env update --name test_env --file env.yml
Collecting package metadata (repodata.json): done
Solving environment: done

==> WARNING: A newer version of conda exists. <==
  current version: 4.14.0
  latest version: 22.9.0

Please update conda by running

    $ conda update -n base -c conda-forge conda

Downloading and Extracting Packages
pycparser-2.21       | 100 KB    | ####################################################### | 100%
sqlite-3.39.2        | 788 KB    | ####################################################### | 100%
conda-package-handli | 1.0 MB    | ####################################################### | 100%
ca-certificates-2022 | 150 KB    | ####################################################### | 100%
pysocks-1.7.1        | 19 KB     | ####################################################### | 100%
mamba-0.25.0         | 46 KB     | ####################################################### | 100%
conda-4.14.0         | 1011 KB   | ####################################################### | 100%
pyopenssl-22.0.0     | 49 KB     | ####################################################### | 100%
colorama-0.4.5       | 18 KB     | ####################################################### | 100%
libmambapy-0.25.0    | 296 KB    | ####################################################### | 100%
certifi-2022.9.24    | 155 KB    | ####################################################### | 100%
requests-2.28.1      | 53 KB     | ####################################################### | 100%
python-3.9.13        | 26.7 MB   | ####################################################### | 100%
wheel-0.37.1         | 31 KB     | ####################################################### | 100%
tqdm-4.64.0          | 81 KB     | ####################################################### | 100%
charset-normalizer-2 | 36 KB     | ####################################################### | 100%
python_abi-3.9       | 4 KB      | ####################################################### | 100%
pycosat-0.6.3        | 108 KB    | ####################################################### | 100%
brotlipy-0.7.0       | 342 KB    | ####################################################### | 100%
idna-3.3             | 55 KB     | ####################################################### | 100%
urllib3-1.26.11      | 102 KB    | ####################################################### | 100%
cryptography-37.0.4  | 1.5 MB    | ####################################################### | 100%
toolz-0.12.0         | 48 KB     | ####################################################### | 100%
cffi-1.15.1          | 229 KB    | ####################################################### | 100%
pip-22.2.2           | 1.5 MB    | ####################################################### | 100%
setuptools-65.4.0    | 731 KB    | ####################################################### | 100%
ruamel_yaml-0.15.80  | 255 KB    | ####################################################### | 100%
six-1.16.0           | 14 KB     | ####################################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate base
#
# To deactivate an active environment, use
#
#     $ conda deactivate

Retrieving notices: ...working... failed
Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/conda/exceptions.py", line 1125, in __call__
  File "/opt/conda/lib/python3.10/site-packages/conda_env/cli/main.py", line 80, in do_call
  File "/opt/conda/lib/python3.10/site-packages/conda/notices/core.py", line 75, in wrapper
  File "/opt/conda/lib/python3.10/site-packages/conda/notices/core.py", line 39, in display_notices
  File "/opt/conda/lib/python3.10/site-packages/conda/notices/http.py", line 36, in get_notice_responses
  File "/opt/conda/lib/python3.10/site-packages/conda/notices/http.py", line 39, in <genexpr>
  File "/opt/conda/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator
  File "/opt/conda/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel
  File "/opt/conda/lib/python3.10/concurrent/futures/_base.py", line 458, in result
  File "/opt/conda/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
  File "/opt/conda/lib/python3.10/concurrent/futures/thread.py", line 58, in run
  File "/opt/conda/lib/python3.10/site-packages/conda/notices/http.py", line 42, in <lambda>
  File "/opt/conda/lib/python3.10/site-packages/conda/notices/cache.py", line 37, in wrapper
  File "/opt/conda/lib/python3.10/site-packages/conda/notices/http.py", line 58, in get_channel_notice_response
  File "/opt/conda/lib/python3.10/site-packages/requests/sessions.py", line 600, in get
  File "/opt/conda/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
  File "/opt/conda/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
  File "/opt/conda/lib/python3.10/site-packages/requests/adapters.py", line 460, in send
  File "/opt/conda/lib/python3.10/site-packages/requests/adapters.py", line 263, in cert_verify
OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /opt/conda/lib/python3.10/site-packages/certifi/cacert.pem

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/bin/conda-env", line 7, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.10/site-packages/conda_env/cli/main.py", line 91, in main
  File "/opt/conda/lib/python3.10/site-packages/conda/exceptions.py", line 1413, in conda_exception_handler
  File "/opt/conda/lib/python3.10/site-packages/conda/exceptions.py", line 1128, in __call__
  File "/opt/conda/lib/python3.10/site-packages/conda/exceptions.py", line 1170, in handle_exception
  File "/opt/conda/lib/python3.10/site-packages/conda/exceptions.py", line 1181, in handle_unexpected_exception
  File "/opt/conda/lib/python3.10/site-packages/conda/exceptions.py", line 1251, in print_unexpected_error_report
ModuleNotFoundError: No module named 'conda.cli.main_info'

Using mamba instead of conda gives

Looking for: ['python=3.9']

Transaction

  Prefix: /opt/conda

  Updating specs:

   - python=3.9

  Package                     Version  Build               Channel                    Size
────────────────────────────────────────────────────────────────────────────────────────────
  Install:
────────────────────────────────────────────────────────────────────────────────────────────

  + sqlite                     3.39.2  h4ff8645_1          conda-forge/linux-64      807kB

  Change:
────────────────────────────────────────────────────────────────────────────────────────────

  - brotlipy                    0.7.0  py310h5764c6d_1004  conda-forge
  + brotlipy                    0.7.0  py39hb9d737c_1004   conda-forge/linux-64      350kB
  - cffi                       1.15.1  py310h255011f_0     conda-forge
  + cffi                       1.15.1  py39he91dace_0      conda-forge/linux-64      235kB
  - cryptography               37.0.4  py310h597c629_0     conda-forge
  + cryptography               37.0.4  py39hd97740a_0      conda-forge/linux-64        2MB
  - pycosat                     0.6.3  py310h5764c6d_1010  conda-forge
  + pycosat                     0.6.3  py39hb9d737c_1010   conda-forge/linux-64      110kB
  - pysocks                     1.7.1  py310hff52083_5     conda-forge
  + pysocks                     1.7.1  pyha2e5f31_6        conda-forge/noarch         19kB
  - ruamel_yaml               0.15.80  py310h5764c6d_1007  conda-forge
  + ruamel_yaml               0.15.80  py39hb9d737c_1007   conda-forge/linux-64      261kB

  Reinstall:
────────────────────────────────────────────────────────────────────────────────────────────

  o charset-normalizer          2.1.1  pyhd8ed1ab_0        conda-forge
  o colorama                    0.4.5  pyhd8ed1ab_0        conda-forge
  o idna                          3.3  pyhd8ed1ab_0        conda-forge
  o pip                        22.2.2  pyhd8ed1ab_0        conda-forge
  o pycparser                    2.21  pyhd8ed1ab_0        conda-forge
  o pyopenssl                  22.0.0  pyhd8ed1ab_0        conda-forge
  o requests                   2.28.1  pyhd8ed1ab_0        conda-forge
  o six                        1.16.0  pyh6c4a22f_0        conda-forge
  o toolz                      0.12.0  pyhd8ed1ab_0        conda-forge
  o tqdm                       4.64.0  pyhd8ed1ab_0        conda-forge
  o urllib3                   1.26.11  pyhd8ed1ab_0        conda-forge
  o wheel                      0.37.1  pyhd8ed1ab_0        conda-forge

  Upgrade:
────────────────────────────────────────────────────────────────────────────────────────────

  - certifi                 2022.6.15  py310hff52083_0     conda-forge
  + certifi                 2022.9.24  pyhd8ed1ab_0        conda-forge/noarch        159kB
  - conda                      4.14.0  py310hff52083_0     conda-forge
  + conda                      22.9.0  py39hf3d152e_1      conda-forge/linux-64     1000kB
  - conda-package-handling      1.8.1  py310h5764c6d_1     conda-forge
  + conda-package-handling      1.9.0  py39hb9d737c_0      conda-forge/linux-64        1MB
  - setuptools                 65.2.0  py310hff52083_0     conda-forge
  + setuptools                 65.4.0  pyhd8ed1ab_0        conda-forge/noarch        749kB

  Downgrade:
────────────────────────────────────────────────────────────────────────────────────────────

  - libmamba                   0.25.0  hd8a31e3_2          conda-forge
  + libmamba                   0.24.0  hd8a31e3_1          conda-forge/linux-64        1MB
  - libmambapy                 0.25.0  py310hab0e683_2     conda-forge
  + libmambapy                 0.24.0  py39hd55135b_1      conda-forge/linux-64      294kB
  - mamba                      0.25.0  py310hf87f941_2     conda-forge
  + mamba                      0.24.0  py39hfa8f2c8_1      conda-forge/linux-64       46kB
  - python                     3.10.6  h582c2e5_0_cpython  conda-forge
  + python                     3.9.13  h9a8a25e_0_cpython  conda-forge/linux-64       28MB
  - python_abi                   3.10  2_cp310             conda-forge
  + python_abi                    3.9  2_cp39              conda-forge/linux-64        4kB

  Summary:

  Install: 1 packages
  Change: 6 packages
  Reinstall: 6 packages
  Upgrade: 4 packages
  Downgrade: 5 packages

  Total download: 36MB

────────────────────────────────────────────────────────────────────────────────────────────

[+] 0.2s
Downloading  (5) ━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━ 396.5kB /  38.3MB libmamba                   0.1s
Extracting       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━       0 /      28                            0.0sterminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
Aborted

miniforge-images/ubuntu/Dockerfile

Doing this with miniforge-images/ubuntu/Dockerfile fails:

$ git clone https://github.com/conda-forge/miniforge-images.git
$ cd miniforge-images/ubuntu
$ docker run --rm -it $(docker build -q .)
$ echo "name: main
channels:
  - nodefaults
  - conda-forge
dependencies:
  - python=3.9" > env.yml
$ conda env update --name base --file env.yml

Gives same error as above

Collecting package metadata (repodata.json): done
Solving environment: done

==> WARNING: A newer version of conda exists. <==
  current version: 4.14.0
  latest version: 22.9.0

Please update conda by running

    $ conda update -n base -c conda-forge conda

Downloading and Extracting Packages
brotlipy-0.7.0       | 342 KB    | ############################################################################################################################################################################################################## | 100%
toolz-0.12.0         | 48 KB     | ############################################################################################################################################################################################################## | 100%
urllib3-1.26.11      | 102 KB    | ############################################################################################################################################################################################################## | 100%
pysocks-1.7.1        | 19 KB     | ############################################################################################################################################################################################################## | 100%
certifi-2022.9.24    | 155 KB    | ############################################################################################################################################################################################################## | 100%
python_abi-3.9       | 4 KB      | ############################################################################################################################################################################################################## | 100%
pyopenssl-22.0.0     | 49 KB     | ############################################################################################################################################################################################################## | 100%
pip-22.2.2           | 1.5 MB    | ############################################################################################################################################################################################################## | 100%
cryptography-37.0.4  | 1.5 MB    | ############################################################################################################################################################################################################## | 100%
python-3.9.13        | 26.7 MB   | ############################################################################################################################################################################################################## | 100%
wheel-0.37.1         | 31 KB     | ############################################################################################################################################################################################################## | 100%
idna-3.3             | 55 KB     | ############################################################################################################################################################################################################## | 100%
sqlite-3.39.2        | 788 KB    | ############################################################################################################################################################################################################## | 100%
tqdm-4.64.0          | 81 KB     | ############################################################################################################################################################################################################## | 100%
pycparser-2.21       | 100 KB    | ############################################################################################################################################################################################################## | 100%
six-1.16.0           | 14 KB     | ############################################################################################################################################################################################################## | 100%
pycosat-0.6.3        | 108 KB    | ############################################################################################################################################################################################################## | 100%
setuptools-65.4.0    | 731 KB    | ############################################################################################################################################################################################################## | 100%
requests-2.28.1      | 53 KB     | ############################################################################################################################################################################################################## | 100%
conda-package-handli | 1.0 MB    | ############################################################################################################################################################################################################## | 100%
conda-4.14.0         | 1011 KB   | ############################################################################################################################################################################################################## | 100%
colorama-0.4.5       | 18 KB     | ############################################################################################################################################################################################################## | 100%
cffi-1.15.1          | 229 KB    | ############################################################################################################################################################################################################## | 100%
charset-normalizer-2 | 36 KB     | ############################################################################################################################################################################################################## | 100%
ca-certificates-2022 | 150 KB    | ############################################################################################################################################################################################################## | 100%
ruamel_yaml-0.15.80  | 255 KB    | ############################################################################################################################################################################################################## | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate base
#
# To deactivate an active environment, use
#
#     $ conda deactivate

Retrieving notices: ...working... failed
Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/conda/exceptions.py", line 1125, in __call__
  File "/opt/conda/lib/python3.10/site-packages/conda_env/cli/main.py", line 80, in do_call
  File "/opt/conda/lib/python3.10/site-packages/conda/notices/core.py", line 75, in wrapper
  File "/opt/conda/lib/python3.10/site-packages/conda/notices/core.py", line 39, in display_notices
  File "/opt/conda/lib/python3.10/site-packages/conda/notices/http.py", line 36, in get_notice_responses
  File "/opt/conda/lib/python3.10/site-packages/conda/notices/http.py", line 39, in <genexpr>
  File "/opt/conda/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator
  File "/opt/conda/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel
  File "/opt/conda/lib/python3.10/concurrent/futures/_base.py", line 458, in result
  File "/opt/conda/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
  File "/opt/conda/lib/python3.10/concurrent/futures/thread.py", line 58, in run
  File "/opt/conda/lib/python3.10/site-packages/conda/notices/http.py", line 42, in <lambda>
  File "/opt/conda/lib/python3.10/site-packages/conda/notices/cache.py", line 37, in wrapper
  File "/opt/conda/lib/python3.10/site-packages/conda/notices/http.py", line 58, in get_channel_notice_response
  File "/opt/conda/lib/python3.10/site-packages/requests/sessions.py", line 600, in get
  File "/opt/conda/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
  File "/opt/conda/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
  File "/opt/conda/lib/python3.10/site-packages/requests/adapters.py", line 460, in send
  File "/opt/conda/lib/python3.10/site-packages/requests/adapters.py", line 263, in cert_verify
OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /opt/conda/lib/python3.10/site-packages/certifi/cacert.pem

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/bin/conda-env", line 7, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.10/site-packages/conda_env/cli/main.py", line 91, in main
  File "/opt/conda/lib/python3.10/site-packages/conda/exceptions.py", line 1413, in conda_exception_handler
  File "/opt/conda/lib/python3.10/site-packages/conda/exceptions.py", line 1128, in __call__
  File "/opt/conda/lib/python3.10/site-packages/conda/exceptions.py", line 1170, in handle_exception
  File "/opt/conda/lib/python3.10/site-packages/conda/exceptions.py", line 1181, in handle_unexpected_exception
  File "/opt/conda/lib/python3.10/site-packages/conda/exceptions.py", line 1251, in print_unexpected_error_report
ModuleNotFoundError: No module named 'conda.cli.main_info'

Previous versions of condaforge/mambaforge

hmaarrfk commented 2 years ago

you are forcing the change from python 3.10 to 3.9 in the base environment. This is not really supported. If you want to do this:

  1. You may use constructor, to create your own installer with a specific version of python and conda and whatever else you need.
  2. You can create a new environment with a similar command. I changed mamba env update to mamba env create:
mamba env create --name test_env --file env.yml
conda-forge/linux-64                                        Using cache
conda-forge/noarch                                          Using cache

Looking for: ['python=3.9']

Transaction

  Prefix: /opt/conda/envs/test_env

  Updating specs:

   - python=3.9

  Package               Version  Build               Channel                   Size
─────────────────────────────────────────────────────────────────────────────────────
  Install:
─────────────────────────────────────────────────────────────────────────────────────

  + _libgcc_mutex           0.1  conda_forge         conda-forge/linux-64       3kB
  + _openmp_mutex           4.5  2_gnu               conda-forge/linux-64      24kB
  + bzip2                 1.0.8  h7f98852_4          conda-forge/linux-64     496kB
  + ca-certificates   2022.9.24  ha878542_0          conda-forge/linux-64     154kB
  + ld_impl_linux-64     2.36.1  hea4e1c9_2          conda-forge/linux-64     683kB
  + libffi                3.4.2  h7f98852_5          conda-forge/linux-64      58kB
  + libgcc-ng            12.1.0  h8d9b700_16         conda-forge/linux-64     963kB
  + libgomp              12.1.0  h8d9b700_16         conda-forge/linux-64     470kB
  + libnsl                2.0.0  h7f98852_0          conda-forge/linux-64      31kB
  + libsqlite            3.39.3  h753d276_0          conda-forge/linux-64     808kB
  + libuuid              2.32.1  h7f98852_1000       conda-forge/linux-64      28kB
  + libzlib              1.2.12  h166bdaf_3          conda-forge/linux-64      66kB
  + ncurses                 6.3  h27087fc_1          conda-forge/linux-64       1MB
  + openssl               3.0.5  h166bdaf_2          conda-forge/linux-64       3MB
  + pip                  22.2.2  pyhd8ed1ab_0        conda-forge/noarch         2MB
  + python               3.9.13  h2660328_0_cpython  conda-forge/linux-64      28MB
  + readline              8.1.2  h0f457ee_0          conda-forge/linux-64     298kB
  + setuptools           65.4.0  pyhd8ed1ab_0        conda-forge/noarch       749kB
  + sqlite               3.39.3  h4ff8645_0          conda-forge/linux-64     808kB
  + tk                   8.6.12  h27826a3_0          conda-forge/linux-64       3MB
  + tzdata                2022d  h191b570_0          conda-forge/noarch       121kB
  + wheel                0.37.1  pyhd8ed1ab_0        conda-forge/noarch        32kB
  + xz                    5.2.6  h166bdaf_0          conda-forge/linux-64     418kB

  Summary:

  Install: 23 packages

  Total download: 43MB

─────────────────────────────────────────────────────────────────────────────────────

_libgcc_mutex                                        2.6kB @   5.7kB/s  0.5s
_openmp_mutex                                       23.6kB @  52.1kB/s  0.5s
ca-certificates                                    153.9kB @ 311.6kB/s  0.5s
libffi                                              58.3kB @ 116.6kB/s  0.0s
libnsl                                              31.2kB @  61.9kB/s  0.1s
libgomp                                            470.3kB @ 918.8kB/s  0.5s
ld_impl_linux-64                                   683.5kB @   1.2MB/s  0.6s
xz                                                 418.4kB @ 741.5kB/s  0.1s
tk                                                   3.5MB @   5.2MB/s  0.2s
pip                                                  1.6MB @   2.3MB/s  0.2s
bzip2                                              495.7kB @ 730.9kB/s  0.1s
libsqlite                                          808.1kB @   1.2MB/s  0.1s
wheel                                               32.0kB @  45.0kB/s  0.1s
libuuid                                             28.3kB @  39.7kB/s  0.0s
libzlib                                             66.2kB @  91.0kB/s  0.1s
tzdata                                             121.1kB @ 161.5kB/s  0.1s
sqlite                                             807.5kB @   1.0MB/s  0.1s
libgcc-ng                                          962.7kB @   1.1MB/s  0.1s
openssl                                              3.0MB @   3.3MB/s  0.2s
readline                                           298.1kB @ 329.4kB/s  0.1s
setuptools                                         748.8kB @ 789.7kB/s  0.1s
ncurses                                              1.0MB @   1.1MB/s  0.2s
python                                              28.0MB @  21.7MB/s  0.8s
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate test_env
#
# To deactivate an active environment, use
#
#     $ conda deactivat
raybellwaves commented 2 years ago

Thanks! Motivates me to bump my stuff from 3.9 to 3.10. I should also switch my local testing to miniforge instead of miniconda

To summarise condaforge/mambaforge:latest has python 3.10 in the base environment and updating to an older version of python e.g. 3.9 isn't supported. Therefore you if you using an older version of python find an older tag in condaforge/mambaforge which using that version of python in the base environment (created https://github.com/conda-forge/miniforge/issues/354)

ngam commented 2 years ago

~Since this seems to be a resolver issue, and if it is a big deal for you to move to python 3.10, have you tried using conda-lock to install python3.9? It seems to work for me...~ Never mind problem persists, something deletes conda/mamba in the process...

ngam commented 2 years ago

Actually, could you try conda env update --name base --file env.yml twice? It will fail the first time, then it will be okay. I don't know why it is doing that...

mieczyslaw commented 2 years ago

Thanks guys for trying to reproduce! As I see now, the only way is to create a new env as base (already active) cannot be reused anymore.

Could someone please answer the question how to activate new env when building a docker image for the command line tool? See my message above with various tricks I tried, basically conda/mamba activate doesn't work for the new env as it asks you to call mamba init. It is easy to play with docker image if you log in into container, so you can exit after mamba init, but how to activate a new env in the single dockerbuild process?

hmaarrfk commented 2 years ago

base (already active) cannot be reused anymore.

This has always been the case since conda started. "base" while convinient will always break if you keep using it since it belongs to conda, not to you.

Could someone please answer the question how to activate new env when building a docker image for the command line tool? See my message above with various tricks I tried, basically conda/mamba activate doesn't work for the new env as it asks you to call mamba init. It is easy to play with docker image if you log in into container, so you can exit after mamba init, but how to activate a new env in the single dockerbuild process?

In my previous example:

(base) root@b76d77417766:/# mamba env create --name test_env --file env.yml
conda-forge/linux-64                                        Using cache
conda-forge/noarch                                          Using cache

Looking for: ['python=3.9']

Transaction

  Prefix: /opt/conda/envs/test_env

  Updating specs:

   - python=3.9

  Package               Version  Build               Channel                   Size
─────────────────────────────────────────────────────────────────────────────────────
  Install:
─────────────────────────────────────────────────────────────────────────────────────

  + _libgcc_mutex           0.1  conda_forge         conda-forge/linux-64       3kB
  + _openmp_mutex           4.5  2_gnu               conda-forge/linux-64      24kB
  + bzip2                 1.0.8  h7f98852_4          conda-forge/linux-64     496kB
  + ca-certificates   2022.9.24  ha878542_0          conda-forge/linux-64     154kB
  + ld_impl_linux-64     2.36.1  hea4e1c9_2          conda-forge/linux-64     683kB
  + libffi                3.4.2  h7f98852_5          conda-forge/linux-64      58kB
  + libgcc-ng            12.1.0  h8d9b700_16         conda-forge/linux-64     963kB
  + libgomp              12.1.0  h8d9b700_16         conda-forge/linux-64     470kB
  + libnsl                2.0.0  h7f98852_0          conda-forge/linux-64      31kB
  + libsqlite            3.39.3  h753d276_0          conda-forge/linux-64     808kB
  + libuuid              2.32.1  h7f98852_1000       conda-forge/linux-64      28kB
  + libzlib              1.2.12  h166bdaf_3          conda-forge/linux-64      66kB
  + ncurses                 6.3  h27087fc_1          conda-forge/linux-64       1MB
  + openssl               3.0.5  h166bdaf_2          conda-forge/linux-64       3MB
  + pip                  22.2.2  pyhd8ed1ab_0        conda-forge/noarch         2MB
  + python               3.9.13  h2660328_0_cpython  conda-forge/linux-64      28MB
  + readline              8.1.2  h0f457ee_0          conda-forge/linux-64     298kB
  + setuptools           65.4.0  pyhd8ed1ab_0        conda-forge/noarch       749kB
  + sqlite               3.39.3  h4ff8645_0          conda-forge/linux-64     808kB
  + tk                   8.6.12  h27826a3_0          conda-forge/linux-64       3MB
  + tzdata                2022d  h191b570_0          conda-forge/noarch       121kB
  + wheel                0.37.1  pyhd8ed1ab_0        conda-forge/noarch        32kB
  + xz                    5.2.6  h166bdaf_0          conda-forge/linux-64     418kB

  Summary:

  Install: 23 packages

  Total download: 43MB

─────────────────────────────────────────────────────────────────────────────────────

_libgcc_mutex                                        2.6kB @   5.7kB/s  0.5s
_openmp_mutex                                       23.6kB @  52.1kB/s  0.5s
ca-certificates                                    153.9kB @ 311.6kB/s  0.5s
libffi                                              58.3kB @ 116.6kB/s  0.0s
libnsl                                              31.2kB @  61.9kB/s  0.1s
libgomp                                            470.3kB @ 918.8kB/s  0.5s
ld_impl_linux-64                                   683.5kB @   1.2MB/s  0.6s
xz                                                 418.4kB @ 741.5kB/s  0.1s
tk                                                   3.5MB @   5.2MB/s  0.2s
pip                                                  1.6MB @   2.3MB/s  0.2s
bzip2                                              495.7kB @ 730.9kB/s  0.1s
libsqlite                                          808.1kB @   1.2MB/s  0.1s
wheel                                               32.0kB @  45.0kB/s  0.1s
libuuid                                             28.3kB @  39.7kB/s  0.0s
libzlib                                             66.2kB @  91.0kB/s  0.1s
tzdata                                             121.1kB @ 161.5kB/s  0.1s
sqlite                                             807.5kB @   1.0MB/s  0.1s
libgcc-ng                                          962.7kB @   1.1MB/s  0.1s
openssl                                              3.0MB @   3.3MB/s  0.2s
readline                                           298.1kB @ 329.4kB/s  0.1s
setuptools                                         748.8kB @ 789.7kB/s  0.1s
ncurses                                              1.0MB @   1.1MB/s  0.2s
python                                              28.0MB @  21.7MB/s  0.8s
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate test_env
#
# To deactivate an active environment, use
#
#     $ conda deactivate

(base) root@b76d77417766:/# ^C
(base) root@b76d77417766:/# mamba env create --name test_env --file env.yml^C
(base) root@b76d77417766:/# conda activate test_env
(test_env) root@b76d77417766:/#

seems to work. Please see how we setup the miniforge docker images quay.io/condaforge/mambaforge https://github.com/conda-forge/miniforge-images/

mieczyslaw commented 2 years ago

It seems that you run it from inside the container. Have you tried to use dockerbuild with FROM condaforge/mambaforge ?

I wonder why you don't see all those error messages which I posted above asking for mamba init ?

hmaarrfk commented 2 years ago

In the original post, you were trying to downgrade the python version. Which, is not supported.

Are you still trying to do this?

mieczyslaw commented 2 years ago

No, after you explained to me stuff around base env, I switched to:

FROM condaforge/mambaforge
COPY . /package
RUN mamba env create -n compute -f /package/environment.yml && mamba clean --all -y
RUN mamba activate compute

This crashes with:

 => ERROR [6/6] RUN mamba activate compute                                                                                                                                                                                                                                 0.8s
------
 > [6/6] RUN mamba activate compute:
#10 0.702 Run 'mamba init' to be able to run mamba activate/deactivate
#10 0.702 and start a new shell session. Or use conda to activate/deactivate.
#10 0.702

Basically when building a container for the tool, I must make sure that when someone calls docker exec, the tool runs in the environment containing the dependencies, here compute.

ngam commented 2 years ago

I highly recommend you follow these instructions if possible: https://uwekorn.com/2021/03/01/deploying-conda-environments-in-docker-how-to-do-it-right.html

ngam commented 2 years ago

What is your exact goal here? Having a container with a single env? Or having a container with a few envs? Remember conda envs are relocatable, so you can simply copy the file structure. I personally usually use something like this:

FROM condaforge/mambaforge:4.14.0-0 as mamba

ADD conda-linux-64.lock /locks/conda-linux-64.lock
RUN mamba create -p /opt/env --copy --file /locks/conda-linux-64.lock && mamba clean -aqy

FROM ubuntu:22.04

COPY --from=mamba /opt/env /opt/env

ENV PATH=/opt/env/bin:$PATH
hmaarrfk commented 2 years ago

I'm not a docker expert (seems like the link ngam might help explain things), but ultimately, there seems to be a trick that is used in order to activate the base environment. The docker commands are to build, but will not get "run upon" first.

https://github.com/conda-forge/miniforge-images/blob/master/ubuntu/Dockerfile#L39

    echo ". ${CONDA_DIR}/etc/profile.d/conda.sh && conda activate base" >> /etc/skel/.bashrc && \
    echo ". ${CONDA_DIR}/etc/profile.d/conda.sh && conda activate base" >> ~/.bashrc

You likely need to add similar lines.

$ cat Dockerfile
FROM condaforge/mambaforge
RUN mamba create --name compute "python=3.9" && mamba clean --all -y
RUN echo "conda activate compute" >> ~/.bashrc
$ docker run -it c96bee051dda
(compute) root@80e0a494370c:/# python --version
Python 3.9.1
ngam commented 2 years ago

The trick to activate the envs is the tini thing: https://github.com/conda-forge/miniforge-images/blob/1d061e7d165191096f4656cc21135c936d737210/ubuntu/Dockerfile#L5

ngam commented 2 years ago

Another good resources is https://github.com/mamba-org/micromamba-docker

mieczyslaw commented 2 years ago

Another good resources is https://github.com/mamba-org/micromamba-docker

This is another thing which confuses me (out of scope for this issue though), what's the difference betweem mambaforge and micromamba containers and what's the reason to develop both?

Thanks all for your contribution and links - I will follow the suggestions.

hmaarrfk commented 2 years ago
hmaarrfk commented 2 years ago

Let us know if you have any other questions, but it seems that things are mostly resolved. Will close in a few days if we don't hear back.

mieczyslaw commented 2 years ago

Yes, we can close this ticket. Thanks again all for your help!