base conda environment not automatically activated by `singularity shell ...` or `singularity exec ...`

[BEFH]

I want to download and extract a zip file in addition to creating an environment, but I cannot install with apt-get or with mamba.

With apt, I get the following error:

0.268 E: List directory /var/lib/apt/lists/partial is missing. - Acquire (13: Permission denied)

This is from the following Dockerfile:

FROM mambaorg/micromamba:0.23.0
COPY --chown=micromamba:micromamba gnomix.yaml /tmp/env.yaml
RUN apt-get update && apt-get install -y wget unzip && \
    micromamba install -y -n base -f /tmp/env.yaml && \
    micromamba clean --all --yes && \
    wget https://github.com/AI-sandbox/gnomix/archive/refs/heads/main.zip && \
    unzip main.zip && \
    mv gnomix-main /gnomix && \
    rm -rf main.zip /gnomix/demo
ENTRYPOINT ["python3 /gnomix/gnomix.py"]

If I try to install wget and unzip using the gnomix.yaml, wget is not found. Attempting to add conda activate base or mamba activate base is unsuccessful, with both conda and mamba not found.

[wholtz]

Hi @BEFH thanks for your interest in micromamba.

The micromamba docker image defaults to running as a non-root user. If you want to use apt-get you'll need to add a USER root command to your Dockerfile before the RUN command. Alternatively, you could add wget and unzip to your conda environment by adding them to the gomix.yaml file and not use apt-get.

You should also read our documentation on modifying the ENTRYPOINT

[wholtz]

Sorry, I just noticed that I missed the last two sentences of your issue. If you want to add wget and unzip via the conda environment, then please read our documentation on Activating a conda environment for RUN commands.

[BEFH]

Thank you for the pointers. So I had two issues, and now I have a third:

I had not done a second run directive after ARG MAMBA_DOCKERFILE_ACTIVATE=1.

I also couldn't use apt because I was mambauser instead of root. That also meant I could not write to the root directory.

Solving that and your suggestion about the ENTRYPOINT, I ended up with the following Dockerfile, which now works with docker run:

FROM mambaorg/micromamba:0.23.0
COPY --chown=micromamba:micromamba gnomix.yaml /tmp/env.yaml
RUN micromamba install -y -n base -f /tmp/env.yaml && \
    micromamba clean --all --yes
USER root
RUN mkdir /gnomix && \
    chown mambauser:mambauser /gnomix
USER mambauser
ARG MAMBA_DOCKERFILE_ACTIVATE=1
RUN wget https://github.com/AI-sandbox/gnomix/archive/refs/heads/main.zip && \
    unzip main.zip && \
    mv gnomix-main/* /gnomix/ && \
    rm -rf main.zip /gnomix/demo
ENTRYPOINT ["/usr/local/bin/_entrypoint.sh", "python3", "/gnomix/gnomix.py"]

The new issue is related to your suggestion about entrypoint: I use docker extensively through singularity, and singularity shell used to activate the conda environment as of version 0.17.0. It no longer does that, which is problematic for many use cases. I don't know what was changed in that time, but it's a huge problem for me.

It still didn't work when the Dockerfile looked like this:

FROM mambaorg/micromamba:0.23.0
COPY --chown=micromamba:micromamba gnomix.yaml /tmp/env.yaml
RUN micromamba install -y -n base -f /tmp/env.yaml && \
    micromamba clean --all --yes

Is there any reason why not to bring back whatever was previously activating the environment, or why not to add a script to /etc/profile.d/ that does that?

[wholtz]

There are several ways to execute a program within a container and we have worked hard to make sure we have a way to activate the conda environment on each of these paths. Back in 0.17.0 there were some paths we didn't have good methods of activating the conda environment and so we just threw the base conda environment into the PATH. We have since improved our methods of activating the environments and we stopped having the base environment always be in the path. This was required to make multi-environment images work as expected.

You'll likely get back the behavior you desire if you add

ENV PATH "$MAMBA_ROOT_PREFIX/bin:$PATH"

near the end of your Dockerfile.

I would like to better support singularity. I'll look into getting it running on my laptop. But in the mean time, it would be helpful to know exactly how you ran the container using singularity, the expected outputs and the actual outputs.

[BEFH]

@wholtz, that is good to know and explains why there is no conda env showing even in the old version.

I think the putting the following bash script in /etc/profile.d would work:

#!/usr/bin/env bash

if [[ -n $SINGULARITY_NAME ]]; then
  source _activate_current_env.sh
fi

This should work to activate the environment just for Sincularity, since SINGULARITY_NAME is only set within a singularity container.

The following commands can be used to drop into a singularity shell:

singularity pull --name gnomix.img docker://befh/gnomix
singularity shell gnomix.img

You can substitute any docker image on docker hub.

[BEFH]

Update:

Apparently, there is no functional init system in Docker or Singularity. I tried using phusion, but that didn't work either.

[wholtz]

I'm finally starting to dig into using our image with singularity. According to my tests:

1. singularity run docker://mambaorg/micromamba:latest micromamba info shows base environment is activated, appears to have desired behaviors.
2. singularity exec docker://mambaorg/micromamba:latest micromamba info shows no environment is activated
3. echo 'micromamba info' | singularity shell docker://mambaorg/micromamba:latest shows no environment is activated

Is there any reason our image needs to automatically activate a conda environment for singularity exec ... given that singularity run ... works?

I do see utility in better supporting singularity shell ... and I will look into this further. I quickly tried your /etc/profile.d suggestion and it did not work on the first pass. I'll need to dive into how singularity shell ... starts the shell command.

[BEFH]

I even tried using different base containers with different init systems and this appears to be a fundamental issue with docker and singularity. I have done the workaround that you suggested for my container, but don't know what the next step is

[wholtz]

@BEFH - are you still having an issue beyond singularity shell ... not activating the environment?

I know it is an extra step, but you can do the following:

$ singularity pull --name gnomix.img docker://befh/gnomix
##### MUCH OUTPUT DELETED #####
$ singularity shell gnomix.img
Singularity> eval "$(micromamba shell hook --shell=bash)" && micromamba activate
(base) Singularity> micromamba info

                                           __
          __  ______ ___  ____ _____ ___  / /_  ____ _
         / / / / __ `__ \/ __ `/ __ `__ \/ __ \/ __ `/
        / /_/ / / / / / / /_/ / / / / / / /_/ / /_/ /
       / .___/_/ /_/ /_/\__,_/_/ /_/ /_/_.___/\__,_/
      /_/

            environment : base (active)
           env location : /opt/conda
      user config files : /home/vagrant/.mambarc
 populated config files : 
       libmamba version : 0.23.0
     micromamba version : 0.23.0
           curl version : libcurl/7.82.0 OpenSSL/1.1.1n zlib/1.2.11 libssh2/1.10.0 nghttp2/1.47.0
     libarchive version : libarchive 3.3.3 zlib/1.2.11 liblzma/5.2.5 bz2lib/1.0.8 liblz4/1.9.2 libzstd/1.4.5
       virtual packages : __unix=0=0
                          __linux=4.15.0=0
                          __glibc=2.31=0
                          __archspec=1=x86_64
               channels : 
       base environment : /opt/conda
               platform : linux-64
(base) Singularity> which python
/opt/conda/bin/python
(base) Singularity>  exit
$

And if you are willing to use a singularity run ... command instead of shell, then everything appears to be working fine for me:

$ singularity run gnomix.img
...
--------------------------------------------------------------------------------
-----------------------------------  Gnomix  -----------------------------------
--------------------------------------------------------------------------------
When using this software, please cite: 
Helgi Hilmarsson, Arvind S Kumar, Richa Rastogi, Carlos D Bustamante, 
Daniel Mas Montserrat, Alexander G Ioannidis: 
"High Resolution Ancestry Deconvolution for Next Generation Genomic Data" 
https://www.biorxiv.org/content/10.1101/2021.09.19.460980v1
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
Usage when training a model from scratch:
   $ python3 gnomix.py <query_file> <output_basename> <chr_nr> <phase> <genetic_map_file> <reference_file> <sample_map_file>
Usage when using a pre-trained model:
   $ python3 gnomix.py <query_file> <output_basename> <chr_nr> <phase> <path_to_model>
$ 

[PhML]

I'm finally starting to dig into using our image with singularity. According to my tests:

1. `singularity run docker://mambaorg/micromamba:latest micromamba info` shows `base` environment is activated, appears to have desired behaviors.

2. `singularity exec docker://mambaorg/micromamba:latest micromamba info` shows no environment is activated

3. `echo 'micromamba info' | singularity shell docker://mambaorg/micromamba:latest` shows no environment is activated

Is there any reason our image needs to automatically activate a conda environment for singularity exec ... given that singularity run ... works?

I do see utility in better supporting singularity shell ... and I will look into this further. I quickly tried your /etc/profile.d suggestion and it did not work on the first pass. I'll need to dive into how singularity shell ... starts the shell command.

I think the difference is that singularity run use the entrypoint while singularity shell directly spawn a new shell. singularity exec execute directly the command.

The shell command allows you to spawn a new shell within your container and interact with it as though it were a virtual machine.

The exec command allows you to execute a custom command within a container by specifying the image file.

Apptainer containers contain runscripts. These are user-defined scripts that define the actions a container should perform when someone runs it. The runscript can be triggered with the run command

cf: https://apptainer.org/docs/user/main/quick_start.html#interacting-with-images

[BEFH]

The automatic activation for shell is useful for pipelining software like Snakemake that uses singularity for execution environments.

On Tue, Apr 18, 2023, 9:26 AM PhML @.***> wrote:

I'm finally starting to dig into using our image with singularity. According to my tests:

1. singularity run docker://mambaorg/micromamba:latest micromamba info shows base environment is activated, appears to have desired behaviors.

2. singularity exec docker://mambaorg/micromamba:latest micromamba info shows no environment is activated

3. echo 'micromamba info' | singularity shell docker://mambaorg/micromamba:latest shows no environment is activated

Is there any reason our image needs to automatically activate a conda environment for singularity exec ... given that singularity run ... works?

I do see utility in better supporting singularity shell ... and I will look into this further. I quickly tried your /etc/profile.d suggestion and it did not work on the first pass. I'll need to dive into how singularity shell ... starts the shell command.

I think the difference is that singularity run use the entrypoint while singularity shell directly spawn a new shell. singularity exec execute directly the command.

The shell https://apptainer.org/docs/user/main/cli/apptainer_shell.html command allows you to spawn a new shell within your container and interact with it as though it were a virtual machine.

The exec https://apptainer.org/docs/user/main/cli/apptainer_exec.html command allows you to execute a custom command within a container by specifying the image file.

Apptainer containers contain runscripts https://apptainer.org/docs/user/main/definition_files.html#runscript. These are user-defined scripts that define the actions a container should perform when someone run https://apptainer.org/docs/user/main/cli/apptainer_run.htmls it. The runscript can be triggered with the run command

cf: https://apptainer.org/docs/user/main/quick_start.html#interacting-with-images

— Reply to this email directly, view it on GitHub https://github.com/mamba-org/micromamba-docker/issues/155#issuecomment-1513154541, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZ2Z2E2IV2KYEWPK5YFGXDXB2I7JANCNFSM5U5ADH6Q . You are receiving this because you were mentioned.Message ID: @.***>

[pditommaso]

I'm facing the same problem with the base environment not being automatically activated when using singularity exec.

Adding the $MAMBA_ROOT_PREFIX/bin to the env $PATH looks like a viable workaround. I'm wondering if this approach can have some possible drawbacks; I guess that the environment activation may do something more than just add the root prefix bin to the $PATH.

Alternatively, is there no way to activate the base environment permanently at build time?

[BEFH]

I think the main issue is that if you activate an environment using the $PATH variable then activate another using conda or mamba, the first environment will remain activated. There may also be some issues with libraries, but it's unlikely to make a difference.

On Sat, Aug 19, 2023, 4:17 AM Paolo Di Tommaso @.***> wrote:

I'm facing the same problem with the base environment not being automatically activated when using singularity exec.

Adding the $MAMBA_ROOT_PREFIX/bin to the env $PATH looks like a viable workaround. I'm wondering if this approach can have some possible drawbacks; I guess that the environment activation may do something more than just add the root prefix bin to the $PATH.

Alternatively, is there no way to activate the base environment permanently at build time?

— Reply to this email directly, view it on GitHub https://github.com/mamba-org/micromamba-docker/issues/155#issuecomment-1684890431, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZ2Z2DGGEPFAJXAHPDS3XDXWBZCFANCNFSM5U5ADH6Q . You are receiving this because you were mentioned.Message ID: @.***>

[maresb]

When a Conda environment is activated, the main thing is adding the binaries to PATH and a few other less-important environment variables are set. In addition, packages are allowed to define an activation script which is automatically executed on activation. The majority of Conda packages don't define an activation script. With a fake PATH activation, these activation scripts won't automatically run. Unless one of the activation scripts is doing something important, the fake PATH activation is likely to work fine..

If you want to check for activation scripts, you can find them under /opt/conda/etc/conda/activate.d.

[pditommaso]

Thanks for the detailed explanation. The update of $PATH with the $MAMBA_ROOT_PREFIX/bin directory looks a reasonable workaround.

[wholtz]

If you are stuck using singularity exec ... and can't convert to singularity run ... because the call is made within some other tool, then you might want to try inserting /usr/local/bin/_entrypoint.sh before your command:

singularity exec docker://mambaorg/micromamba:latest  /usr/local/bin/_entrypoint.sh micromamba info 

[wholtz]

I have a way to activate an environment when using apptainer shell .....

However, there are a two caveats:

1. You have to modify the image
2. You need to pass a flag or configuration environmental variable to apptainer shell ...

FROM mambaorg/micromamba:1.5.0

USER root
RUN echo $'#!/bin/bash\n/bin/bash --rcfile /root/.bashrc -i' > /usr/local/bin/_apptainer_shell.sh \
    && chmod +x /usr/local/bin/_apptainer_shell.sh
USER $MAMBA_USER

apptainer shell --shell /usr/local/bin/_apptainer_shell.sh docker://my_modified_image:latest

OR

APPTAINER_SHELL=/usr/local/bin/_apptainer_shell.sh apptainer shell docker://my_modified_image:latest

When the next version of micromamba is released, I'll add that tiny _apptainer_shell.sh to our images.

[pditommaso]

Thanks for the detailed feedback. Updating the PATH with the root bin looks a fair workaround. After that, we are planning to switch to use singularity run that allow better handling this need. Thanks!

[wholtz]

I added the _apptainer_shell.sh script in v1.5.1 of the images 5a57120c7007d8225ea27f59a9bb874bfd416fb4.

While we cannot automatically activate environments for singularity exec and singularity shell, we now have workarounds in our FAQ (derived from this issue) for use with each of those sub-commands. This is probably the best that we can provide unless we start generating images specifically for use with singularity (PR anyone?). Therefore I am now closing this issue.