activating conda environment from bash script

[ajshedivy]

Current problem

Conda environment cannot be activated from a bash script. within pl-example.sh I have the following:

conda init bash
source ~/Anaconda3/etc/profile.d/conda.sh
conda activate pathlinker_env
conda info --envs

the output:

no change     C:\Users\ajshe\Anaconda3\Scripts\conda.exe
no change     C:\Users\ajshe\Anaconda3\Scripts\conda-env.exe
no change     C:\Users\ajshe\Anaconda3\Scripts\conda-script.py
no change     C:\Users\ajshe\Anaconda3\Scripts\conda-env-script.py
no change     C:\Users\ajshe\Anaconda3\condabin\conda.bat
no change     C:\Users\ajshe\Anaconda3\Library\bin\conda.bat
no change     C:\Users\ajshe\Anaconda3\condabin\_conda_activate.bat
no change     C:\Users\ajshe\Anaconda3\condabin\rename_tmp.bat
no change     C:\Users\ajshe\Anaconda3\condabin\conda_auto_activate.bat
no change     C:\Users\ajshe\Anaconda3\condabin\conda_hook.bat
no change     C:\Users\ajshe\Anaconda3\Scripts\activate.bat
no change     C:\Users\ajshe\Anaconda3\condabin\activate.bat
no change     C:\Users\ajshe\Anaconda3\condabin\deactivate.bat
no change     C:\Users\ajshe\Anaconda3\Scripts\activate
no change     C:\Users\ajshe\Anaconda3\Scripts\deactivate
no change     C:\Users\ajshe\Anaconda3\etc\profile.d\conda.sh
no change     C:\Users\ajshe\Anaconda3\etc\fish\conf.d\conda.fish
no change     C:\Users\ajshe\Anaconda3\shell\condabin\Conda.psm1
no change     C:\Users\ajshe\Anaconda3\shell\condabin\conda-hook.ps1
no change     C:\Users\ajshe\Anaconda3\Lib\site-packages\xonsh\conda.xsh
no change     C:\Users\ajshe\Anaconda3\etc\profile.d\conda.csh
no change     C:\Users\ajshe\.bash_profile
No action taken.
# conda environments:
#
base                     C:\Users\ajshe\Anaconda3
pathlinker_env        *  `C:\Users\ajshe\Anaconda3\envs\pathlinker_env

After running the script from the bash command line, conda info --envs produces the last section listing my environments. Does the '*' indicate an activate environment? To test this, I ran conda info from the command line in which the following was outputted:

active environment : base
    active env location : C:\Users\ajshe\Anaconda3
            shell level : 1
       user config file : C:\Users\ajshe\.condarc
 populated config files : C:\Users\ajshe\.condarc
          conda version : 4.6.14
    conda-build version : 3.17.8
         python version : 3.7.3.final.0
       base environment : C:\Users\ajshe\Anaconda3  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/win-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\Users\ajshe\Anaconda3\pkgs
                          C:\Users\ajshe\.conda\pkgs
                          C:\Users\ajshe\AppData\Local\conda\conda\pkgs
       envs directories : C:\Users\ajshe\Anaconda3\envs
                          C:\Users\ajshe\.conda\envs
                          C:\Users\ajshe\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/4.6.14 requests/2.21.0 CPython/3.7.3 Windows/10 Windows/10.0.17134
          administrator : False
             netrc file : None
           offline mode : False

Here, you can see that the pathlinker_env was not activated.

[ajshedivy]

issue resolved by using ml-bio-workshop script

[agitter]

That's good news you got it working now. Is conda init needed inside the script? I'm more familiar with the installation process for older versions of Anaconda, but conda init may be a one time command needed during the initial installation.

Does the '*' indicate an activate environment?

It should indicate the active environment. I'm not sure why conda info showed the base environment instead.

[ajshedivy]

I'm having issues again getting the pl-example.sh to activate my environment. A couple of days ago it seemed to be working fine with the script I had, yet today when I went to check it, it is giving me the same issue as above. There is a star by pathlinker_env, but when I run conda info it tells me my base is active. I am using the same script within the example subdirectory. This is the same script that worked for me a couple of days ago.

Like you noted, It doesn't seem to matter if I use conda init bash within the script. I'm not sure if this is what is causing me the error but my efforts to debug haven't been successful yet. Let me know if you have any thoughts on this issue.

[ajshedivy]

When debugging, I came across something that may be useful. Using the updated script, I added python --version at the end to see what was happening:

#!/usr/bin/env/bash

#attempts to activate pathlinker_env in order to run pathlinker

#if env not configured, then pathlinker_env will be created

#conda init bash
eval "$(conda shell.bash hook)"
conda activate pathlinker_env

if [ $? -ne 0 ]; then
    echo creating pathlinker_env environment
    conda env create -f environment.yml
    conda activate pathlinker_env
fi
conda info --envs
python --version

When I run this from the command line, I get the following output:

$ bash pl-example.sh
# conda environments:
#
base                     C:\Users\ajshe\Anaconda3
pathlinker_env        *  C:\Users\ajshe\Anaconda3\envs\pathlinker_env

Python 3.5.5 :: Anaconda custom (64-bit)

you can see that Python 3.5.5 has been "activated", but when running python --version again from the command line, I get:

$python --version
Python 3.7.3

additionally running conda info within the script produces this:

$ bash pl-example.sh
# conda environments:
#
base                     C:\Users\ajshe\Anaconda3
pathlinker_env        *  C:\Users\ajshe\Anaconda3\envs\pathlinker_env

Python 3.5.5 :: Anaconda custom (64-bit)

     active environment : pathlinker_env
    active env location : C:\Users\ajshe\Anaconda3\envs\pathlinker_env
            shell level : 2
       user config file : C:\Users\ajshe\.condarc
 populated config files : C:\Users\ajshe\.condarc
          conda version : 4.6.14
    conda-build version : 3.17.8
         python version : 3.7.3.final.0
       base environment : C:\Users\ajshe\Anaconda3  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/win-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\Users\ajshe\Anaconda3\pkgs
                          C:\Users\ajshe\.conda\pkgs
                          C:\Users\ajshe\AppData\Local\conda\conda\pkgs
       envs directories : C:\Users\ajshe\Anaconda3\envs
                          C:\Users\ajshe\.conda\envs
                          C:\Users\ajshe\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/4.6.14 requests/2.21.0 CPython/3.7.3 Windows/10 Windows/10.0.17134
          administrator : False
             netrc file : None
           offline mode : False

[agitter]

Are you running python --version again after the script terminates when you see 3.73 returned? That would make sense because bash scripts can have behavior that is not immediately intuitive. They run as a new child process that is created from the parent process (your command line terminal). The consequence is that some changes you make inside a script like setting environment variables, changing directories, or activating conda environments do not affect the parent process after the script finishes. There is a little discussion of that in http://mywiki.wooledge.org/BashFAQ/060

You can imagine the bash script having a local copy of your command line environment. You can change state, run command, etc. in that environment. Changes to files will persist, but changes to other state like environment variables will not persist after the script terminates. So if you want to run Python commands inside an environment, it is okay to activate the environment and run the commands inside it even if you are not still in the environment when the script terminates and your return to the parent process.

The source command is a way to make changes in the script propagate to the parent process, but I suggest working in the child process's environment for now.

[ajshedivy]

Thanks for the help! That makes sense. I was confused because after running the script I expected the environment to be activated within the parent process. What I was doing was running the script, seeing that the environment was activated (using conda info and python --version to check), and then after the script terminates, run the same commands to check if the environment was activated in the parent process. I think I have a better understanding of what the script is actually accomplishing.

Going forward, I should be able to write the PathLinker commands within the "local copy" that lives within the script without having any issues.