Modulize skipper

[algaebrown]

Hi @byee4 @howardxu520,

please review my code regarding modulizing skipper. Now the single-end ENCODE 4 protocol shared the rules with ENCODE 3. I am also extending the modules to do gkmSVM and variant interpretation. @howardxu520 's containers are merged into this pull request too. let me know what needs clarification.

[algaebrown]

To test, please use /home/hsher/projects/skipper/encode_configs/Skipper_pe_manifest_small.csv and /home/hsher/projects/skipper/encore_configs/Skipper_config_small_test.yaml

# on TSCC 1.0
snakemake -kps Skipper.py \
    -j 30 \
    --cluster "qsub -e {params.error_file} -o {params.out_file} -l walltime={params.run_time} -l nodes=1:ppn={threads} -q home-yeo" \
    --configfile /home/hsher/projects/skipper/encore_configs/Skipper_config_small_test.yaml \
    --conda-prefix /home/hsher/snakeconda \
    --use-conda \
    --use-singularity \
    --singularity-prefix /home/hsher/scratch/singularity \
    --singularity-args "--bind /oasis --bind /projects --bind /scratch" \
    -n

# ENCODE 3
snakemake -kps Skipper.py \
    -j 30 \
    --cluster "qsub -e {params.error_file} -o {params.out_file} -l walltime={params.run_time} -l nodes=1:ppn={threads} -q home-yeo" \
    --configfile /home/hsher/projects/skipper/encode_configs/Skipper_pe_small_test.yaml \
    --conda-prefix /home/hsher/snakeconda \
    --use-conda \
    --use-singularity \
    --singularity-prefix /home/hsher/scratch/singularity \
    --singularity-args "--bind /oasis --bind /projects --bind /scratch" 

[byee4]

@algaebrown I wasn't able to get the SVM rules working possibly due to issues with conflicting conda environments as you discovered earlier. But I did manage to run the modularized components and am checking ENCODE/ENCORE outputs with the previous runs, and once these are checked I'll go ahead and open a new branch to include my fixes (minor things ie. memory/cpu requirement increases), since we'll want to start using this version moving forward.

Also I think @howardxu520 came across this error before, but the scaled bigwigs rule doesn't work with Singularity due to character escapement (see a related? stackoverflow issue).

[algaebrown]

Sounds good to me. I can containerize the python envs in the future and prioritize using docker instead of conda.yaml in the future.

Charlene On Jan 2, 2024, 9:07 AM -0800, Brian Yee @.***>, wrote:

@algaebrown I wasn't able to get the SVM rules working possibly due to issues with conflicting conda environments as you discovered earlier. But I did manage to run the modularized components and am checking ENCODE/ENCORE outputs with the previous runs, and once these are checked I'll go ahead and open a new branch to include my fixes (minor things ie. memory/cpu requirement increases), since we'll want to start using this version moving forward. Also I think @howardxu520 came across this error before, but the scaled bigwigs rule doesn't work with Singularity due to character escapement (see a related? stackoverflow issue). — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

[byee4]

I created a new branch with your modularized changes and beefed up the system reqs that allow us to process all encode/encore data without issue. We can work off this branch?

https://github.com/YeoLab/skipper/tree/modularized

To view these small changes, I drafted a pr but no need to merge this one.

https://github.com/YeoLab/skipper/pull/21/files