search

dieterich-lab / rp-bp

Rp-Bp is a Bayesian approach to predict, at base-pair resolution, ribosome occupancy and translation.
MIT License
7 stars 5 forks source link

tests/regression/test_rpbp.py::test_pipeline_part2 fails #144

Closed eboileau closed 1 year ago

eboileau commented 1 year ago

If I run the test locally on the cluster, the call is made using --num-cpus 40

cmd = 'run-all-rpbp-instances /beegfs/scratch/global_tmp/pytest-of-eboileau/pytest-84/data0/c-elegans-chrI-example/c-elegans-test.yaml --num-cpus 40 --merge-replicates --ru
n-replicates --overwrite --keep-intermediate-files --write-unfiltered'

This is due to this change:

this line https://github.com/dieterich-lab/rp-bp/blob/9f40fbb91ea55802749aa76563b1376d5e37e72d/tests/regression/conftest.py#L207

and results in 

BAMoutput.cpp:27:BAMoutput: exiting because of *OUTPUT FILE* error: could not create output file /beegfs/scratch/global_tmp/pytest-of-eboileau/pytest-84/data0/c-elegans-chr
I-example/without-rrna-mapping/c-elegans-rep-1_STARtmp//BAMsort/19/48
SOLUTION: check that the path exists and you have write permission for this file. Also check ulimit -n and increase it to allow more open files.

This is a know issue, running STAR with a large number of threads may be problematic, see e.g. https://github.com/alexdobin/STAR/issues/528 or https://github.com/alexdobin/STAR/issues/1074, etc.

lkeegan commented 1 year ago

This also occurs for me locally (with 12 cores / 24 threads), doing

ulimit -n 10000

before running fixes it for me - maybe this should be added to the docs?

eboileau commented 1 year ago

Or we can just revert to a hard coded number of cpus, e.g. num_cpus = 6 (except for macOS).

eboileau commented 1 year ago

This is unrelated, but I got this as a result of now running the tests successfully.

We now compile models to rpbp_install_dir/models, this is fine in general, but for an -editable install, this will compile models under the local git repository. So after running the test, I got now 

        src/rpbp/models/nonperiodic/no-periodicity                                                                                                                          
        src/rpbp/models/nonperiodic/no-periodicity.hpp                                                                                                                      
        src/rpbp/models/nonperiodic/start-high-high-low                                                                                                                     
        src/rpbp/models/nonperiodic/start-high-high-low.hpp                                                                                                                 
        src/rpbp/models/nonperiodic/start-high-low-high                                                                                                                     
        src/rpbp/models/nonperiodic/start-high-low-high.hpp                                                                                                                 
        src/rpbp/models/periodic/start-high-low-low                                   
        src/rpbp/models/periodic/start-high-low-low.hpp                                                                                                                     
        src/rpbp/models/translated/periodic-gaussian-mixture                                                                                                                
        src/rpbp/models/translated/periodic-gaussian-mixture.hpp                                                                                                            
        src/rpbp/models/untranslated/gaussian-naive-bayes                                                                                                                   
        src/rpbp/models/untranslated/gaussian-naive-bayes.hpp

The easiest solution is to exclude these in the gitignore with 

# compiled Stan models
src/rpbp/models/*
!src/rpbp/models/**/
src/rpbp/models/**/*/*
!src/rpbp/models/**/*.stan