search

biocore / oecophylla

shotgun pipeline
MIT License
11 stars 19 forks source link

Some small fixes #120

Closed tanaes closed 6 years ago

tanaes commented 6 years ago

I came across a few errors when running some real datasets on Barnacle, which this PR fixes.

  1. There was an intermittent memory allocation exceeded error caused by the host filtering step. I tracked this down to the piped commands giving bam output to the sort, which I think forced it to load the entire output into memory prior to sorting. Instead outputting to localscratch on disk prior to sort and fastq conversion.

  2. This error would cause a befuddling Snakemake failure where the job would fail, but Snakemake wouldn't notice it. As jobs failed, this eventually caused the Snakemake process to stop submitting any new ones because it thought the failed jobs were still running. To fix, I took advantage of a new functionality in Snakemake ≥ 4.1 which allows a job status script to check in on jobs directly from the scheduler. This way Snakemake can monitor the job without writing temp files + restart as necessary.

Note that this requires a newer version of Snakemake than we were running before, so please reinstall your base oecophylla environment.

tanaes commented 6 years ago

Also note that this follows PR #119 so please merge first!

tanaes commented 6 years ago

@qiyunzhu addressed your comments. This new script status really helps with reliability of execution on the cluster, working on porting over to SLURM too for Comet.

@mortonjt or @sjanssen2 want to take a quick look as well?

qiyunzhu commented 6 years ago

Shall I merge? @tanaes

tanaes commented 6 years ago

Yes please!

On Wed, Oct 25, 2017 at 2:10 PM Qiyun Zhu notifications@github.com wrote:

Shall I merge? @tanaes https://github.com/tanaes

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/biocore/oecophylla/pull/120#issuecomment-339419600, or mute the thread https://github.com/notifications/unsubscribe-auth/AH6JADK2YkryP5YtXlUiEZ2iLJb1Twtxks5sv3j6gaJpZM4QAA_u .