memory issues causes some processes not to complete

[asifzubair]

For 8vs8, normal/senescent analysis, the app seems to be crashing.

It seems to not like system() calls and something like even sessionInfo() might make it crash:

!label: sess_info (with options) 

List of 2
$ echo   : logi FALSE
$ comment: chr ""

.Quitting from lines 603-605 (atacseeker.Rmd) 
YError in system(paste(which, shQuote(names[i])), intern = TRUE, ignore.stderr = TRUE) : 
^  cannot popen '/usr/bin/which 'uname' 2>/dev/null', probable reason 'Cannot allocate memory'
OCalls: <Anonymous> ... withVisible -> eval -> eval -> sessionInfo -> Sys.which

Execution halted
Warning message:
,system call failed: Cannot allocate memory 

Here are some arguments why system() should be avoided.

I should try to make everything use native R as much as possible and avoid system() calls. Some processes I have identified are:

• coping bam files to /data/scratch. please see this SO
• assembly mtDNA genome
• creating bigwigs

The execution of latter two can be moved to the calling script and then use xargs to parallelize.

Finally, here is a page that talks about memory issues in R.

[ttriche]

R is a pig. If you use Rsamtools and/or wrap the worst offenders (memory-wise) so that they run serially, you can loop through relatively easily. Whenever the parent R process runs out of RAM to allocate, it tries to activate the garbage collector, and if that doesn't work the child process that system() may have spawned will also be killed. (This is not unique to R, btw)

You might look into gnu parallel (as opposed to just xargs) if you have multiple cores at your disposal. The bottom line is that if you have a piggy R process spawning a piggy variant caller or other piggy program, it's going to blow up.

"RAM is cheap and thinking hurts" is the underlying philosophy in R.

--t

On Sun, Jul 10, 2016 at 8:30 AM, Asif Zubair notifications@github.com wrote:

For 8vs8, normal/senescent analysis, the app seems to be crashing.

It seems to not like system() calls and something like even sessionInfo() might make it crash:

�!label: sess_info (with options)

List of 2 $ echo : logi FALSE $ comment: chr ""

.Quitting from lines 603-605 (atacseeker.Rmd) YError in system(paste(which, shQuote(names[i])), intern = TRUE, ignore.stderr = TRUE) : ^ cannot popen '/usr/bin/which 'uname' 2>/dev/null', probable reason 'Cannot allocate memory' OCalls: ... withVisible -> eval -> eval -> sessionInfo -> Sys.which

Execution halted �Warning message: ,system call failed: Cannot allocate memory

Here http://stackoverflow.com/questions/19913446/why-should-the-system-function-be-avoided-in-c-and-c are some arguments why system() should be avoided.

I should try to make everything use native R as much as possible and avoid system() calls. Some processes I have identified are:

• coping bam files to /data/scratch. please see this SO http://stackoverflow.com/questions/31655076/copy-folders-from-one-directory-to-another-in-r
• making mtDNA variants
• creating bigwigs

The execution of latter two can be moved to the calling script and then use xargs to parallelize.

Finally, here http://www.matthewckeller.com/html/memory.html is a page that talks about memory issues in R.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/RamsinghLab/atacseeker_app/issues/14, or mute the thread https://github.com/notifications/unsubscribe/AAARImkhB7kPadhA3eWw2RYnQG7jGNI8ks5qURAmgaJpZM4JI20x .

[asifzubair]

Yup. I created this issue to change all system() calls to samtools to Rsamtools.

[asifzubair]

I am using parallel now to make the bigWigs. I've set the number of cores to 4, but I think I should recheck with Jay about specs.

[asifzubair]

I'm going to store the output from sessionInfo in a file and simply cat it to screen. This way I avoid the troublesome system() call. This is a work-around, but I think for now it will ...err... work.