sims-lab / ccb-wiki

Wiki documenting the CBRG cluster setup
2 stars 0 forks source link

conda environment is not picked up automatically for P.run statements #4

Closed Charlie-George closed 3 years ago

Charlie-George commented 3 years ago

If running a job using P.run the conda environment you ran the script in is not automatically used for the P.run() job.

e.g. I run python pipeline_atacseq.py make full from my obds-py3 environment which contains fastqc

When i run

statement = 'fastqc --help' 
P.run(statement) 

I get an error - 'fastqc not found'

Charlie-George commented 3 years ago

Need to update .bashrc so that conda is only sourced and activated for interactive shells using if [[ $PS1 ]]; then

For example

# .bashrc

# Source global definitions from system bashrc file
if [ -f /etc/bashrc ]; then
 . /etc/bashrc
fi

# Non-interactive shells inherit the path and other variables
# from the calling shell, so this setup is not needed.
if [[ $PS1 ]]; then

    # Set umask for default file permissions
    umask 002

    ### Load environment modules
    # Load the latest version of Git (system version is old)
    module load git/2.31.1

    # remove usr/condabin from path
    PATH=$(echo "$PATH" | sed -e 's/\/usr\/condabin://')

    # unset python path
    unset PYTHONPATH

    ### source conda
    #source  /t1-data/user/cgeorge/2_conda/centos7/etc/profile.d/conda.sh
    source ~/conda/obds_conda/etc/profile.d/conda.sh
    conda activate base

    ### Set environment variables
    # Set DRMAA path for Ruffus / cgatcore pipelines to talk to slurm
    export DRMAA_LIBRARY_PATH=/usr/lib64/libdrmaa.so

    # Set temporary folders for Ruffus / cgatcore pipelines
    export TMPDIR=/tmp
    export SHARED_TMPDIR=/t1-data/user/${USER}/tmp

fi # if PS1

If conda is sourced in main body of script each time a job is submitted the .bashrc is sourced, conda is resourced and base environment (or what ever env you activate in the main body of the .bashrc is reactivated.

I've updated the example .bashrc