search

dieterich-lab / rp-bp

Rp-Bp is a Bayesian approach to predict, at base-pair resolution, ribosome occupancy and translation.
MIT License
7 stars 5 forks source link

Models not detected in linux #72

Closed bmmalone closed 7 years ago

bmmalone commented 7 years ago

From Zhengtao:

I encounter an error when I run the example dataset following the instruction in https://github.com/dieterich-lab/rp-bp/blob/master/docs/running-example.md

The command and error are below, could you possibly help me solve this problem? BTW, I want to run rp-bp using my alignment file, how should I write the yam file , and what command should I use ?

Best regards,

Zhengtao Xiao 

> run-all-rpbp-instances c-elegans-test.yaml --overwrite --num-cpus 2 --logging-level INFO --keep-intermediate-filesINFO    
> 
> root     2017-06-05 20:08:41,048 : run-rpbp-pipeline /home/yanglab/test_rpbp/c-elegans-chrI-example/c-elegans.test-chrI.rep-1.fastq.gz c-elegans-test.yaml c-elegans-rep-1 --num-cpus 2   --overwrite --logging-level INFO --file-logging-level NOTSET --stderr-logging-level NOTSET --stdout-logging-level NOTSET --star-executable STAR --star-read-files-command zcat   --keep-intermediate-files --mem 2G
> INFO     root     2017-06-05 20:08:41,049 : calling
> INFO     root     2017-06-05 20:08:41,762 : create-orf-profiles /home/yanglab/test_rpbp/c-elegans-chrI-example/c-elegans.test-chrI.rep-1.fastq.gz c-elegans-test.yaml c-elegans-rep-1 --num-cpus 2 --mem 2G  --overwrite --file-logging-level NOTSET --stdout-logging-level NOTSET --logging-level INFO --stderr-logging-level NOTSET --star-read-files-command zcat --star-executable STAR   --keep-intermediate-files
> INFO     root     2017-06-05 20:08:41,762 : calling
> INFO     rpbp.orf_profile_construction.create_orf_profiles 2017-06-05 20:08:42,987 : [create-orf-profiles]: /home/yanglab/anaconda3/envs/rpbp/bin/create-orf-profiles /home/yanglab/test_rpbp/c-elegans-chrI-example/c-elegans.test-chrI.rep-1.fastq.gz c-elegans-test.yaml c-elegans-rep-1 --num-cpus 2 --mem 2G --overwrite --file-logging-level NOTSET --stdout-logging-level NOTSET --logging-level INFO --stderr-logging-level NOTSET --star-read-files-command zcat --star-executable STAR --keep-intermediate-files
> INFO     root     2017-06-05 20:08:43,045 : create-base-genome-profile /home/yanglab/test_rpbp/c-elegans-chrI-example/c-elegans.test-chrI.rep-1.fastq.gz c-elegans-test.yaml c-elegans-rep-1 --num-cpus 2  --overwrite --logging-level INFO --file-logging-level NOTSET --stdout-logging-level NOTSET --stderr-logging-level NOTSET --star-read-files-command zcat --star-executable STAR   --keep-intermediate-files --mem 2G
> INFO     root     2017-06-05 20:08:43,045 : calling
> INFO     rpbp.orf_profile_construction.create_base_genome_profile 2017-06-05 20:08:43,707 : [create-base-genome-profile]: /home/yanglab/anaconda3/envs/rpbp/bin/create-base-genome-profile /home/yanglab/test_rpbp/c-elegans-chrI-example/c-elegans.test-chrI.rep-1.fastq.gz c-elegans-test.yaml c-elegans-rep-1 --num-cpus 2 --overwrite --logging-level INFO --file-logging-level NOTSET --stdout-logging-level NOTSET --stderr-logging-level NOTSET --star-read-files-command zcat --star-executable STAR --keep-intermediate-files --mem 2G
> INFO     root     2017-06-05 20:08:43,764 : flexbar --qtrim-format sanger --max-uncalled 1  --adapters /home/yanglab/test_rpbp/c-elegans-chrI-example/riboseq-adapters.fa -n 2 --zip-output GZ -r /home/yanglab/test_rpbp/c-elegans-chrI-example/c-elegans.test-chrI.rep-1.fastq.gz -t /home/yanglab/test_rpbp/c-elegans-chrI-example/without-adapters/c-elegans-rep-1.test --pre-trim-left 0
> INFO     root     2017-06-05 20:08:43,764 : calling
> INFO     root     2017-06-05 20:08:49,737 : bowtie2 -p 2 --very-fast -x /home/yanglab/test_rpbp/c-elegans-chrI-example/ribosomal-index/X03680_1 -U /home/yanglab/test_rpbp/c-elegans-chrI-example/without-adapters/c-elegans-rep-1.test.fastq.gz -S /dev/null --un-gz /home/yanglab/test_rpbp/c-elegans-chrI-example/without-rrna/c-elegans-rep-1.test.fastq.gz --al-gz /home/yanglab/test_rpbp/c-elegans-chrI-example/with-rrna/c-elegans-rep-1.test.fastq.gz
> INFO     root     2017-06-05 20:08:49,737 : calling
> 30430 reads; of these:
>   30430 (100.00%) were unpaired; of these:
>     20523 (67.44%) aligned 0 times
>     9907 (32.56%) aligned exactly 1 time
>     0 (0.00%) aligned >1 times
> 32.56% overall alignment rate
> INFO     root     2017-06-05 20:08:50,427 : STAR --runThreadN 2 --readFilesCommand zcat --genomeDir /home/yanglab/test_rpbp/c-elegans-chrI-example/WBcel235.79.chrI/STAR --sjdbGTFfile /home/yanglab/test_rpbp/c-elegans-chrI-example/WBcel235.79.chrI.gtf --readFilesIn /home/yanglab/test_rpbp/c-elegans-chrI-example/without-rrna/c-elegans-rep-1.test.fastq.gz --alignIntronMin 20 --alignIntronMax 100000 --outFilterMismatchNmax 1 --outFilterType BySJout --outFilterIntronMotifs RemoveNoncanonicalUnannotated  --outFilterMismatchNoverLmax 0.04 --outSAMattributes AS NH HI nM MD --outFileNamePrefix /home/yanglab/test_rpbp/c-elegans-chrI-example/without-rrna-mapping/c-elegans-rep-1.test --outSAMtype BAM SortedByCoordinate  --limitBAMsortRAM 2147483648
> INFO     root     2017-06-05 20:08:50,427 : calling
> Jun 05 20:08:50 ..... started STAR run
> Jun 05 20:08:50 ..... loading genome
> Jun 05 20:08:51 ..... processing annotations GTF
> Jun 05 20:08:51 ..... inserting junctions into the genome indices
> Jun 05 20:09:16 ..... started mapping
> Jun 05 20:09:20 ..... started sorting BAM
> Jun 05 20:09:20 ..... finished successfully
> INFO     root     2017-06-05 20:09:20,642 : samtools view -h /home/yanglab/test_rpbp/c-elegans-chrI-example/without-rrna-mapping/c-elegans-rep-1.testAligned.sortedByCoord.out.bam > /dev/null
> INFO     root     2017-06-05 20:09:20,643 : calling
> WARNING  root     2017-06-05 20:09:20,721 : [utils.create_symlink]: file already exists at: '/home/yanglab/test_rpbp/c-elegans-chrI-example/without-rrna-mapping/c-elegans-rep-1.test.bam'. It will be removed
> INFO     root     2017-06-05 20:09:20,721 : samtools index -b /home/yanglab/test_rpbp/c-elegans-chrI-example/without-rrna-mapping/c-elegans-rep-1.test.bam
> INFO     root     2017-06-05 20:09:20,721 : calling
> INFO     root     2017-06-05 20:09:20,749 : remove-multimapping-reads /home/yanglab/test_rpbp/c-elegans-chrI-example/without-rrna-mapping/c-elegans-rep-1.test.bam /home/yanglab/test_rpbp/c-elegans-chrI-example/without-rrna-mapping/c-elegans-rep-1.test-unique.bam
> INFO     root     2017-06-05 20:09:20,750 : calling
> INFO     root     2017-06-05 20:09:21,642 : samtools view -h /home/yanglab/test_rpbp/c-elegans-chrI-example/without-rrna-mapping/c-elegans-rep-1.test-unique.bam > /dev/null
> INFO     root     2017-06-05 20:09:21,643 : calling
> INFO     root     2017-06-05 20:09:21,723 : extract-metagene-profiles /home/yanglab/test_rpbp/c-elegans-chrI-example/without-rrna-mapping/c-elegans-rep-1.test-unique.bam /home/yanglab/test_rpbp/c-elegans-chrI-example/WBcel235.79.chrI/WBcel235.79.chrI.annotated.bed.gz /home/yanglab/test_rpbp/c-elegans-chrI-example/metagene-profiles/c-elegans-rep-1.test-unique.metagene-profile.csv.gz --num-cpus 2 --logging-level INFO --file-logging-level NOTSET --stdout-logging-level NOTSET --stderr-logging-level NOTSET  --start-upstream 50 --start-downstream 20 --end-upstream 50 --end-downstream 20
> INFO     root     2017-06-05 20:09:21,724 : calling
> INFO     riboutils.extract_metagene_profiles 2017-06-05 20:09:22,870 : [extract-metagene-profiles]: /home/yanglab/anaconda3/envs/rpbp/bin/extract-metagene-profiles /home/yanglab/test_rpbp/c-elegans-chrI-example/without-rrna-mapping/c-elegans-rep-1.test-unique.bam /home/yanglab/test_rpbp/c-elegans-chrI-example/WBcel235.79.chrI/WBcel235.79.chrI.annotated.bed.gz /home/yanglab/test_rpbp/c-elegans-chrI-example/metagene-profiles/c-elegans-rep-1.test-unique.metagene-profile.csv.gz --num-cpus 2 --logging-level INFO --file-logging-level NOTSET --stdout-logging-level NOTSET --stderr-logging-level NOTSET --start-upstream 50 --start-downstream 20 --end-upstream 50 --end-downstream 20
> 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10163/10163 [00:00<00:00, 121016.10it/s]
> INFO     riboutils.extract_metagene_profiles 2017-06-05 20:09:23,010 : Reading annotations
> INFO     riboutils.extract_metagene_profiles 2017-06-05 20:09:23,041 : Constructing canonical translation initiation ORF data frames
> INFO     riboutils.extract_metagene_profiles 2017-06-05 20:09:23,065 : Constructing canonical translation termination ORF data frames
> INFO     riboutils.extract_metagene_profiles 2017-06-05 20:09:23,086 : Finding translation initiation site matches
> INFO     riboutils.extract_metagene_profiles 2017-06-05 20:09:23,154 : Finding translation termination site matches
> INFO     riboutils.extract_metagene_profiles 2017-06-05 20:09:23,216 : Extracting metagene profiles
> INFO     riboutils.extract_metagene_profiles 2017-06-05 20:09:23,217 : Profiles will be created for lengths: 12,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,40,41,42,48,50
> INFO     riboutils.extract_metagene_profiles 2017-06-05 20:09:23,350 : Writing metagene profiles to disk
> INFO     root     2017-06-05 20:09:23,508 : gunzip -t /home/yanglab/test_rpbp/c-elegans-chrI-example/metagene-profiles/c-elegans-rep-1.test-unique.metagene-profile.csv.gz
> INFO     root     2017-06-05 20:09:23,508 : calling
> INFO     root     2017-06-05 20:09:23,516 : estimate-metagene-profile-bayes-factors /home/yanglab/test_rpbp/c-elegans-chrI-example/metagene-profiles/c-elegans-rep-1.test-unique.metagene-profile.csv.gz /home/yanglab/test_rpbp/c-elegans-chrI-example/metagene-profiles/c-elegans-rep-1.test-unique.metagene-periodicity-bayes-factors.csv.gz --num-cpus 2 --periodic-models  --nonperiodic-models  --periodic-offset-start -20 --periodic-offset-end 0 --metagene-profile-length 21 --seed 8675309 --chains 2 --iterations 500 --logging-level INFO --file-logging-level NOTSET --stdout-logging-level NOTSET --stderr-logging-level NOTSET
> INFO     root     2017-06-05 20:09:23,516 : calling
> usage: estimate-metagene-profile-bayes-factors [-h]
>                                                [--periodic-models PERIODIC_MODELS [PERIODIC_MODELS ...]]
>                                                [--nonperiodic-models NONPERIODIC_MODELS [NONPERIODIC_MODELS ...]]
>                                                [--periodic-offset-start PERIODIC_OFFSET_START]
>                                                [--periodic-offset-end PERIODIC_OFFSET_END]
>                                                [--metagene-profile-length METAGENE_PROFILE_LENGTH]
>                                                [-s SEED] [-c CHAINS]
>                                                [-i ITERATIONS] [-p NUM_CPUS]
>                                                [--type-field TYPE_FIELD]
>                                                [--count-field COUNT_FIELD]
>                                                [--position-field POSITION_FIELD]
>                                                [--log-file LOG_FILE]
>                                                [--log-stdout]
>                                                [--no-log-stderr]
>                                                [--logging-level {NOTSET,DEBUG,INFO,WARNING,ERROR,CRITICAL}]
>                                                [--file-logging-level {NOTSET,DEBUG,INFO,WARNING,ERROR,CRITICAL}]
>                                                [--stdout-logging-level {NOTSET,DEBUG,INFO,WARNING,ERROR,CRITICAL}]
>                                                [--stderr-logging-level {NOTSET,DEBUG,INFO,WARNING,ERROR,CRITICAL}]
>                                                metagene_profiles out
> estimate-metagene-profile-bayes-factors: error: argument --periodic-models: expected at least one argument
> Traceback (most recent call last):
>   File "/home/yanglab/anaconda3/envs/rpbp/bin/create-orf-profiles", line 11, in <module>
>     load_entry_point('rpbp==1.1.8', 'console_scripts', 'create-orf-profiles')()
>   File "/home/yanglab/anaconda3/envs/rpbp/lib/python3.5/site-packages/rpbp/orf_profile_construction/create_orf_profiles.py", line 229, in main
>     file_checkers=file_checkers, overwrite=args.overwrite, call=call)
>   File "/home/yanglab/anaconda3/envs/rpbp/lib/python3.5/site-packages/misc/shell_utils.py", line 241, in call_if_not_exists
>     ret_code = check_call(cmd, call=call, raise_on_error=raise_on_error)
>   File "/home/yanglab/anaconda3/envs/rpbp/lib/python3.5/site-packages/misc/shell_utils.py", line 89, in check_call
>     return check_call_step(cmd, call=call, raise_on_error=raise_on_error)
>   File "/home/yanglab/anaconda3/envs/rpbp/lib/python3.5/site-packages/misc/shell_utils.py", line 72, in check_call_step
>     raise subprocess.CalledProcessError(ret_code, cmd)
> subprocess.CalledProcessError: Command 'estimate-metagene-profile-bayes-factors /home/yanglab/test_rpbp/c-elegans-chrI-example/metagene-profiles/c-elegans-rep-1.test-unique.metagene-profile.csv.gz /home/yanglab/test_rpbp/c-elegans-chrI-example/metagene-profiles/c-elegans-rep-1.test-unique.metagene-periodicity-bayes-factors.csv.gz --num-cpus 2 --periodic-models  --nonperiodic-models  --periodic-offset-start -20 --periodic-offset-end 0 --metagene-profile-length 21 --seed 8675309 --chains 2 --iterations 500 --logging-level INFO --file-logging-level NOTSET --stdout-logging-level NOTSET --stderr-logging-level NOTSET' returned non-zero exit status 2
> Traceback (most recent call last):
>   File "/home/yanglab/anaconda3/envs/rpbp/bin/run-rpbp-pipeline", line 11, in <module>
>     load_entry_point('rpbp==1.1.8', 'console_scripts', 'run-rpbp-pipeline')()
>   File "/home/yanglab/anaconda3/envs/rpbp/lib/python3.5/site-packages/rpbp/run_rpbp_pipeline.py", line 140, in main
>     shell_utils.check_call(cmd)
>   File "/home/yanglab/anaconda3/envs/rpbp/lib/python3.5/site-packages/misc/shell_utils.py", line 89, in check_call
>     return check_call_step(cmd, call=call, raise_on_error=raise_on_error)
>   File "/home/yanglab/anaconda3/envs/rpbp/lib/python3.5/site-packages/misc/shell_utils.py", line 72, in check_call_step
>     raise subprocess.CalledProcessError(ret_code, cmd)
> subprocess.CalledProcessError: Command 'create-orf-profiles /home/yanglab/test_rpbp/c-elegans-chrI-example/c-elegans.test-chrI.rep-1.fastq.gz c-elegans-test.yaml c-elegans-rep-1 --num-cpus 2 --mem 2G  --overwrite --file-logging-level NOTSET --stdout-logging-level NOTSET --logging-level INFO --stderr-logging-level NOTSET --star-read-files-command zcat --star-executable STAR   --keep-intermediate-files' returned non-zero exit status 1
> Traceback (most recent call last):
>   File "/home/yanglab/anaconda3/envs/rpbp/bin/run-all-rpbp-instances", line 11, in <module>
>     load_entry_point('rpbp==1.1.8', 'console_scripts', 'run-all-rpbp-instances')()
>   File "/home/yanglab/anaconda3/envs/rpbp/lib/python3.5/site-packages/rpbp/run_all_rpbp_instances.py", line 168, in main
>     job_id = slurm.check_sbatch(cmd, args=args)
>   File "/home/yanglab/anaconda3/envs/rpbp/lib/python3.5/site-packages/misc/slurm.py", line 182, in check_sbatch
>     shell_utils.check_call(cmd, call=call)
>   File "/home/yanglab/anaconda3/envs/rpbp/lib/python3.5/site-packages/misc/shell_utils.py", line 89, in check_call
>     return check_call_step(cmd, call=call, raise_on_error=raise_on_error)
>   File "/home/yanglab/anaconda3/envs/rpbp/lib/python3.5/site-packages/misc/shell_utils.py", line 72, in check_call_step
>     raise subprocess.CalledProcessError(ret_code, cmd)
> subprocess.CalledProcessError: Command 'run-rpbp-pipeline /home/yanglab/test_rpbp/c-elegans-chrI-example/c-elegans.test-chrI.rep-1.fastq.gz c-elegans-test.yaml c-elegans-rep-1 --num-cpus 2   --overwrite --logging-level INFO --file-logging-level NOTSET --stderr-logging-level NOTSET --stdout-logging-level NOTSET --star-executable STAR --star-read-files-command zcat   --keep-intermediate-files --mem 2G' returned non-zero exit status 1
>
bmmalone commented 7 years ago

This appears to be related to Issue #36.

bmmalone commented 7 years ago

Later from Zhengato

I run this test on Linux. I don’t known which mode file is not found correctly, and which parameters should check in yam file?

bmmalone commented 7 years ago

Hi Zhengato,

For Linux, the Stan model files are typically installed in $HOME/.local/share/rpbp/rpbp_models/*. Please use the following command to check if the files exist:

$ find $HOME/.local/share/rpbp \( -name "*stan" -o -name "*pkl" \)

The expected output is something like:

/home/nle5231/.local/share/rpbp/rpbp_models/periodic/start-high-low-low.pkl
/home/nle5231/.local/share/rpbp/rpbp_models/nonperiodic/start-high-high-low.pkl
/home/nle5231/.local/share/rpbp/rpbp_models/nonperiodic/no-periodicity.pkl
/home/nle5231/.local/share/rpbp/rpbp_models/nonperiodic/start-high-low-high.pkl
/home/nle5231/.local/share/rpbp/rpbp_models/translated/periodic-gaussian-mixture.pkl
/home/nle5231/.local/share/rpbp/rpbp_models/untranslated/gaussian-naive-bayes.pkl

The pickle files contain compiled versions of the respective Stan models described in the paper. If they do not exist, then a problem occurred while installing the program.

If they are not there, please try installing with detailed debug information:

pip3 uninstall rpbp
pip3 install --verbose --log log.rpbp-install.txt -r requirements.txt

The log file should then contain detailed error messages, so we can see if there was a problem compiling the models.

Have a good day, Brandon

bmmalone commented 7 years ago

Later from Zhengato...

Thanks! But I encountered another error when install it using pip3,

/bin/sh: 1: pickle-stan: not found

What package should I install before this step?

bmmalone commented 7 years ago

The pickle-stan script is installed as part of the pymisc-utils package, which is now explicitly included and versioned in the requirements.txt file. That should address this issue.