search

twang15 / K562-Analysis

1 stars 1 forks source link

BPNet tutorial #15

Closed twang15 closed 2 years ago

twang15 commented 2 years ago

Installation

  module load bedtools/2.27.1
  pyenv local 3.6.0
  python3.6 -m pip install --upgrade pip
  pip install -U pyyaml

Requirement already satisfied: pyyaml in /home/taowang9/.local/lib/python3.6/site-packages (5.3.1) Collecting pyyaml Downloading PyYAML-5.4.1-cp36-cp36m-manylinux1_x86_64.whl (640 kB) |████████████████████████████████| 640 kB 6.8 MB/s Installing collected packages: pyyaml Attempting uninstall: pyyaml Found existing installation: PyYAML 5.3.1 Uninstalling PyYAML-5.3.1: Successfully uninstalled PyYAML-5.3.1 ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. awscli 1.18.29 requires PyYAML<5.4,>=3.10; python_version != "3.4", but you have pyyaml 5.4.1 which is incompatible. Successfully installed pyyaml-5.4.1

Problem: pyyaml version is too high Solution: downgrade

pip install -U pyyaml==5.2

Collecting pyyaml==5.2 Downloading PyYAML-5.2.tar.gz (265 kB) |████████████████████████████████| 265 kB 6.7 MB/s Building wheels for collected packages: pyyaml Building wheel for pyyaml (setup.py) ... done Created wheel for pyyaml: filename=PyYAML-5.2-cp36-cp36m-linux_x86_64.whl size=44209 sha256=662e8c51ddf210c68024411ae41befd9f3c9061623e8270a804bf8380c613a5f Stored in directory: /home/taowang9/.cache/pip/wheels/af/4b/a6/d492db4435642aebd1df1cd3ed075815e75ad6665aa47f4d04 Successfully built pyyaml Installing collected packages: pyyaml Attempting uninstall: pyyaml Found existing installation: PyYAML 5.4.1 Uninstalling PyYAML-5.4.1: Successfully uninstalled PyYAML-5.4.1 Successfully installed pyyaml-5.2

  pip install wandb snakemake
  pip install -U jupyter_client>=6.1.2
  export HDF5_USE_FILE_LOCKING=FALSE
  pip install git+https://github.com/kundajelab/DeepExplain.git
  pip install bpnet
twang15 commented 2 years ago

pip install bpnet

Problem:

ERROR: Cannot install bpnet==0.0.1, bpnet==0.0.10, bpnet==0.0.11, bpnet==0.0.12, bpnet==0.0.13, bpnet==0.0.14, bpnet==0.0.15, bpnet==0.0.16, bpnet==0.0.17, bpnet==0.0.18, bpnet==0.0.19, bpnet==0.0.2, bpnet==0.0.20, bpnet==0.0.21, bpnet==0.0.22, bpnet==0.0.23, bpnet==0.0.3, bpnet==0.0.4, bpnet==0.0.5, bpnet==0.0.6, bpnet==0.0.7, bpnet==0.0.8 and bpnet==0.0.9 because these package versions have conflicting dependencies.

The conflict is caused by: bpnet 0.0.23 depends on pprint bpnet 0.0.22 depends on pprint bpnet 0.0.21 depends on pprint bpnet 0.0.20 depends on pprint bpnet 0.0.19 depends on pprint bpnet 0.0.18 depends on pprint bpnet 0.0.17 depends on pprint bpnet 0.0.16 depends on pprint bpnet 0.0.15 depends on pprint bpnet 0.0.14 depends on pprint bpnet 0.0.13 depends on pprint bpnet 0.0.12 depends on pprint bpnet 0.0.11 depends on pprint bpnet 0.0.10 depends on pprint bpnet 0.0.9 depends on pprint bpnet 0.0.8 depends on pprint bpnet 0.0.7 depends on pprint bpnet 0.0.6 depends on pprint bpnet 0.0.5 depends on pprint bpnet 0.0.4 depends on pprint bpnet 0.0.3 depends on pprint bpnet 0.0.2 depends on pprint bpnet 0.0.1 depends on pprint

To fix this you could try to:

  1. loosen the range of package versions you've specified
  2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies

install conda

curl -sL "https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh" > "Miniconda3.sh" bash Miniconda3.sh

twang15 commented 2 years ago

From Vivek:

We now use Zahoor’s basepairmodels repo. It is much easier to use. https://github.com/kundajelab/basepairmodels

twang15 commented 2 years ago

Install Conda

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh bash Miniconda3-latest-Linux-x86_64.sh

Welcome to Miniconda3 py39_4.10.3

In order to continue the installation process, please review the license agreement. Please, press ENTER to continue

Press Enter

Do you accept the license terms? [yes|no] [no] >>> yes

Miniconda3 will now be installed into this location: /home/taowang9/miniconda3

[/home/taowang9/miniconda3] >>> /oak/stanford/scg/prj_ENCODE/miniconda3 PREFIX=/oak/stanford/scg/prj_ENCODE/miniconda3 Unpacking payload ...

Bug: Conda always shows '[y/N]' then nothing happen

twang15 commented 2 years ago

[taowang9@smsh11dsu-srcf-d15-37 ChIP]$ conda init bash no change /oak/stanford/scg/prj_ENCODE/miniconda3/condabin/conda no change /oak/stanford/scg/prj_ENCODE/miniconda3/bin/conda no change /oak/stanford/scg/prj_ENCODE/miniconda3/bin/conda-env no change /oak/stanford/scg/prj_ENCODE/miniconda3/bin/activate no change /oak/stanford/scg/prj_ENCODE/miniconda3/bin/deactivate no change /oak/stanford/scg/prj_ENCODE/miniconda3/etc/profile.d/conda.sh no change /oak/stanford/scg/prj_ENCODE/miniconda3/etc/fish/conf.d/conda.fish no change /oak/stanford/scg/prj_ENCODE/miniconda3/shell/condabin/Conda.psm1 no change /oak/stanford/scg/prj_ENCODE/miniconda3/shell/condabin/conda-hook.ps1 no change /oak/stanford/scg/prj_ENCODE/miniconda3/lib/python3.6/site-packages/xontrib/conda.xsh no change /oak/stanford/scg/prj_ENCODE/miniconda3/etc/profile.d/conda.csh no change /home/taowang9/.bashrc No action taken.

[taowang9@smsh11dsu-srcf-d15-37 ChIP]$ vi /home/taowang9/.bashrc delete the content appended by previous installation

[taowang9@smsh11dsu-srcf-d15-37 ChIP]$ conda init bash no change /oak/stanford/scg/prj_ENCODE/miniconda3/condabin/conda no change /oak/stanford/scg/prj_ENCODE/miniconda3/bin/conda no change /oak/stanford/scg/prj_ENCODE/miniconda3/bin/conda-env no change /oak/stanford/scg/prj_ENCODE/miniconda3/bin/activate no change /oak/stanford/scg/prj_ENCODE/miniconda3/bin/deactivate no change /oak/stanford/scg/prj_ENCODE/miniconda3/etc/profile.d/conda.sh no change /oak/stanford/scg/prj_ENCODE/miniconda3/etc/fish/conf.d/conda.fish no change /oak/stanford/scg/prj_ENCODE/miniconda3/shell/condabin/Conda.psm1 no change /oak/stanford/scg/prj_ENCODE/miniconda3/shell/condabin/conda-hook.ps1 no change /oak/stanford/scg/prj_ENCODE/miniconda3/lib/python3.6/site-packages/xontrib/conda.xsh no change /oak/stanford/scg/prj_ENCODE/miniconda3/etc/profile.d/conda.csh modified /home/taowang9/.bashrc

==> For changes to take effect, close and re-open your current shell. <==

Exit current shell and re-enter a brand new shell

(base) [taowang9@smsh11dsu-srcf-d15-37 prj_ENCODE]$

twang15 commented 2 years ago

Model: basepairmodels

Download inputs for tutorial

wget https://www.encodeproject.org/files/ENCFF198CVB/@@download/ENCFF198CVB.bam -O rep1.bam wget https://www.encodeproject.org/files/ENCFF488CXC/@@download/ENCFF488CXC.bam -O rep2.bam wget https://www.encodeproject.org/files/ENCFF023NGN/@@download/ENCFF023NGN.bam -O control.bam

Prepare other tools

module load samtools module load bamtools/2.5.1 module load bedtools

samtools merge -f merged.bam rep1.bam rep2.bam samtools index merged.bam

https://raw.githubusercontent.com/igvteam/igv/master/genomes/sizes/hg38.chrom.sizes

bedtools genomecov (aka genomeCoverageBed)

twang15 commented 2 years ago

Model: BPNet

Installing pip dependencies: \ Ran pip subprocess with arguments: ['/oak/stanford/scg/prj_ENCODE/miniconda3/envs/bpnet/bin/python', '-m', 'pip', 'install', '-U', '-r', '/oak/stanford/scg/prj_ENCODE/ChIP/BPNet/bpnet/condaenv.b8r3tfih.requirements.txt'] Pip subprocess output: Collecting git+https://github.com/kundajelab/DeepExplain.git (from -r /oak/stanford/scg/prj_ENCODE/ChIP/BPNet/bpnet/condaenv.b8r3tfih.requirements.txt (line 1)) Cloning https://github.com/kundajelab/DeepExplain.git to /tmp/pip-req-build-dvwz3067 Resolved https://github.com/kundajelab/DeepExplain.git to commit 738c7145e915a7a48f3a4248d088bcc2e1a94614 Processing /oak/stanford/scg/prj_ENCODE/ChIP/BPNet/bpnet

Pip subprocess error: Running command git clone -q https://github.com/kundajelab/DeepExplain.git /tmp/pip-req-build-dvwz3067 DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default. pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555. ERROR: Could not install packages due to an OSError: [('/oak/stanford/scg/prj_ENCODE/ChIP/BPNet/bpnet/.git/objects/pack/pack-32901b931750e59eef7e6dadd825eab351eb71b3.pack', '/tmp/pip-req-build-xpg16md7/.git/objects/pack/pack-32901b931750e59eef7e6dadd825eab351eb71b3.pack', "[Errno 13] Permission denied: '/tmp/pip-req-build-xpg16md7/.git/objects/pack/pack-32901b931750e59eef7e6dadd825eab351eb71b3.pack'"), ('/oak/stanford/scg/prj_ENCODE/ChIP/BPNet/bpnet/.git/objects/pack/pack-32901b931750e59eef7e6dadd825eab351eb71b3.idx', '/tmp/pip-req-build-xpg16md7/.git/objects/pack/pack-32901b931750e59eef7e6dadd825eab351eb71b3.idx', "[Errno 13] Permission denied: '/tmp/pip-req-build-xpg16md7/.git/objects/pack/pack-32901b931750e59eef7e6dadd825eab351eb71b3.idx'")]

failed

CondaEnvException: Pip failed

twang15 commented 2 years ago

Model BPNet

modify conda-env.yml as following:

(base) [taowang9@smsh11dsu-srcf-d15-36 bpnet]$ git diff conda-env.yml diff --git a/conda-env.yml b/conda-env.yml index 3c8a087..1d9821c 100644 --- a/conda-env.yml +++ b/conda-env.yml @@ -30,7 +30,7 @@ dependencies:

Reference: Allow pip install flags to be recognized in environment files

(base) [taowang9@smsh11dsu-srcf-d15-36 bpnet]$ conda env create -f conda-env.yml

CondaValueError: prefix already exists: /oak/stanford/scg/prj_ENCODE/miniconda3/envs/bpnet

Solution: conda env update -f conda-env.yml

twang15 commented 2 years ago

Model: BPNet

Installation is successful!

# To activate this environment, use
#
#     $ conda activate bpnet
#
# To deactivate an active environment, use
#
#     $ conda deactivate

(base) [taowang9@smsh11dsu-srcf-d15-36 bpnet]$ conda env list
# conda environments:
#
base                  *  /oak/stanford/scg/prj_ENCODE/miniconda3
basepairmodels           /oak/stanford/scg/prj_ENCODE/miniconda3/envs/basepairmodels
bpnet                    /oak/stanford/scg/prj_ENCODE/miniconda3/envs/bpnet

Usage Example

Follow tutorial: https://colab.research.google.com/drive/1VNsNBfugPJfJ02LBgvPwj-gPK0L_djsD#scrollTo=k_tFgH1y-dq-

Step 1. Get test data

wget 'https://drive.google.com/uc?authuser=0&id=1YX1svzlRnLtvxyzj5O4B1l5-_wArZ9Q6&export=download' -O bpnet.tar.gz
tar xvfz bpnet.tar.gz  > /dev/null

(bpnet) [taowang9@smsh11dsu-srcf-d15-36 bpnet]$ snakemake -d bpnet/examples -s bpnet/examples/Snakefile chip_nexus -j 4 --quiet
ModuleNotFoundError in line 4 of /oak/stanford/scg/prj_ENCODE/ChIP/BPNet/bpnet/bpnet/examples/Snakefile:
No module named 'bpnet'
  File "/oak/stanford/scg/prj_ENCODE/ChIP/BPNet/bpnet/bpnet/examples/Snakefile", line 4, in <module>
twang15 commented 2 years ago

Install Pyenv

curl https://pyenv.run | bash

pyenv.run redirects to the install script in this repository and the invocation above is equivalent to:

curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash

Restart your shell so the path changes take effect:

exec $SHELL

Update:

  pyenv update

Uninstall Pyenv

pyenv is installed within $PYENV_ROOT (default: ~/.pyenv). To uninstall, just remove it:

rm -fr ~/.pyenv

then remove these three lines from .bashrc:

export PATH="$HOME/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

and finally, restart your shell:

exec $SHELL

Reference: https://github.com/pyenv/pyenv-installer

install a python version and ipython, pudb

pyenv install 3.7.0

After installation, check python versions

pyenv versions

system # system default 3.6.0 3.7.0 # newly installed * 3.7.4 (set by /home/taowang9/NLP/.python-version) # current effective version 3.8.0 3.9.0

Switch version to 3.7.0 for current directory and its sub-directories

pyenv local 3.7.0
pyenv versions

system 3.6.0 * 3.7.0 (set by /home/taowang9/NLP/.python-version) # successfully switch to this version 3.7.4 3.8.0 3.9.0

let's double-check 3.7.0 is indeed the current active python

python -V

Python 3.7.0

pip is installed as part of python installation, now let's check its version

pip -V

pip 10.0.1 from /home/taowang9/.pyenv/versions/3.7.0/lib/python3.7/site-packages/pip (python 3.7)

Upgrade pip

pip install --upgrade pip

pip -V

pip 21.2.4 from /home/taowang9/.pyenv/versions/3.7.0/lib/python3.7/site-packages/pip (python 3.7) # upgraded

ipython and pudb are ancillary packages, and have to be installed for each python version explicitly. Otherwise, the one for system python (system version in pyenv) is in use.

Before installation:

ipython -V

3.2.1

After installation:

pip install ipython
ipython -V

7.26.0

The same holds for pudb, except after pip install pudb, the executable is called pudb3 (for python 3 versions) or pudb2 (for python 2 versions), so we have to do the following:

echo "alias pudb='~/.pyenv/shims/pudb3'" >> ~/.bashrc

To uninstall a python version from pyenv, 

pyenv uninstall 3.7.0
pyenv versions

pyenv: version `3.7.0' is not installed (set by /home/taowang9/NLP/.python-version) system 3.6.0 3.7.4 3.8.0 3.9.0

To resolve the above error message:

rm -f /home/taowang9/NLP/.python-version
pyenv versions

system 3.6.0 3.7.4 * 3.8.0 (set by /home/taowang9/.python-version) # automatically set to a new version 3.9.0

twang15 commented 2 years ago

Conflicts between pyenv and Conda

When pyenv takes effect, installation of Conda (see above for Conda installation) can be done but cannot function correctly.

Check which environment is installed and activated

(base) [taowang9@smsh11dsu-srcf-d15-38 ~]$ conda env list

conda environments:

# base * /oak/stanford/scg/prj_ENCODE/miniconda3 basepairmodels /oak/stanford/scg/prj_ENCODE/miniconda3/envs/basepairmodels bpnet /oak/stanford/scg/prj_ENCODE/miniconda3/envs/bpnet

Check environments and their python version

conda env list | grep -v "^$\|#" |awk '{print $1;}'|xargs -I{} -d "\n" sh -c 'printf "Env: {}\t"; conda list -n {} |grep "^python\s";'

Env: base python 3.6.8 h0371630_0 Env: basepairmodels python 3.6.8 h0371630_0 Env: bpnet python 3.6.13 hffdb5ce_0_cpython conda-forge

Chech which python version is activated by pyenv

pyenv versions

system 3.6.0 3.7.4 * 3.8.0 (set by /home/taowang9/.python-version) 3.9.0

Check python version that is really active

python -V

Python 3.6.8 :: Anaconda, Inc.

which python

/oak/stanford/scg/prj_ENCODE/miniconda3/bin/python

In .bashrc, Conda initialization code comes after pyenv initialization code. Therefore, Conda environment wins! If we reverse the order, pyenv environment will win for sure ( I have done the experiment, but details is omitted here)

Therefore, we have to pick either a pyenv or conda environment to use. They should never be activated at the same time. This lesson is learned in a hard way and that's why I document the details here.

twang15 commented 2 years ago

Conda Cheat Sheet: How to use conda

List of all environments

conda env list

conda environments:

base * /oak/stanford/scg/prj_ENCODE/miniconda3 # currently activated basepairmodels /oak/stanford/scg/prj_ENCODE/miniconda3/envs/basepairmodels bpnet /oak/stanford/scg/prj_ENCODE/miniconda3/envs/bpnet

Switch to another environment:

conda activate bpnet
conda env list

conda environments: base /oak/stanford/scg/prj_ENCODE/miniconda3 basepairmodels /oak/stanford/scg/prj_ENCODE/miniconda3/envs/basepairmodels bpnet * /oak/stanford/scg/prj_ENCODE/miniconda3/envs/bpnet

Create a new environment with python 3.8.0 and activate it

conda create --name py38 python==3.8.0
conda activate py38
conda env list

conda environments:

base /oak/stanford/scg/prj_ENCODE/miniconda3 basepairmodels /oak/stanford/scg/prj_ENCODE/miniconda3/envs/basepairmodels bpnet /oak/stanford/scg/prj_ENCODE/miniconda3/envs/bpnet py38 * /oak/stanford/scg/prj_ENCODE/miniconda3/envs/py38

which -a python

/oak/stanford/scg/prj_ENCODE/miniconda3/envs/py38/bin/python /usr/bin/python (system python)

which python

/oak/stanford/scg/prj_ENCODE/miniconda3/envs/py38/bin/python 

python -V # active python

Python 3.8.0

/usr/bin/python -V

Python 2.7.5

pip is built in with python

which pip

/oak/stanford/scg/prj_ENCODE/miniconda3/envs/py38/bin/pip but pudb and ipython are also ancillary packages for conda, and need to install them explicitly

conda install pudb

Collecting package metadata (current_repodata.json): done Solving environment: failed with initial frozen solve. Retrying with flexible solve. Collecting package metadata (repodata.json): done Solving environment: failed with initial frozen solve. Retrying with flexible solve.

PackagesNotFoundError: The following packages are not available from current channels:

Current channels:

Try with pip

pip install pudb
which pudb3

/oak/stanford/scg/prj_ENCODE/miniconda3/envs/py38/bin/pudb3

replace python 3.8.0 to 3.9.0 in the current active version

conda install python==3.9.0
which python

/oak/stanford/scg/prj_ENCODE/miniconda3/envs/py38/bin/python

python -V

Python 3.9.0

remove an environment and everything in it

conda deactivate
conda env remove --name py38

Remove all packages in environment /oak/stanford/scg/prj_ENCODE/miniconda3/envs/py38

conda env list

conda environments:

base /oak/stanford/scg/prj_ENCODE/miniconda3 basepairmodels /oak/stanford/scg/prj_ENCODE/miniconda3/envs/basepairmodels bpnet * /oak/stanford/scg/prj_ENCODE/miniconda3/envs/bpnet