Difficulty using multiple cores-R goes non-responsive

[mleipold]

After last week's success getting Citrus installed (Issue #68, thanks again, Rob), I did a few test cases with FCS files.

However, any time that I try to use more than one core on the "Run!" page, R hangs up.

Specifically, the same FCS files run with "Set Multicore Usage" clicked and "1" core selected will complete and give me output files.

Whereas "2" or "8" cores, R gets through Reading the files, Sampling each file, Clustering. It then reaches

Analyzing condition(s): defaultCondition Building Fold Features Analyzing vs endpoint

and goes no further. Instead, I get the spinning rainbow wheel of death.

1. This job finishes in less than 10 min with one core (like I said, small test case with only 1000 events per file)
2. I let it run overnight (~14 hours) when trying the 8 core job, and it still never finished.
3. If I go to Force Quit, it shows "R (not responding)".
4. The "STOP" sign in R is greyed out and nonresponsive.

This is on a Mac Pro running Mavericks 10.9.4; 3GHz 8-core Intel Xeon E5, 64GB 1867 MHz DDR3.

Mike

[rbruggner]

This sounds bad. Does the job complete if you leave the "Set Multicore Usage" box unchecked?

----- Original Message ----- From: "mleipold" notifications@github.com To: "nolanlab/citrus" citrus@noreply.github.com Sent: Tuesday, August 19, 2014 1:41:13 PM Subject: [citrus] Difficulty using multiple cores-R goes non-responsive (#71)

After last week's success getting Citrus installed (thanks again, Rob), I did a few test cases with FCS files.

However, any time that I try to use more than one core on the "Run!" page, R hangs up.

Specifically, the same FCS files run with "Set Multicore Usage" clicked and "1" core selected will complete and give me output files.

Whereas "2" or "8" cores, R gets through Reading the files, Sampling each file, Clustering. It then reaches

Analyzing condition(s): defaultCondition Building Fold Features Analyzing vs endpoint

and goes no further. Instead, I get the spinning rainbow wheel of death.

1. This job finishes in less than 10 min with one core (like I said, small test case with only 1000 events per file)
2. I let it run overnight (~14 hours) when trying the 8 core job, and it still never finished.
3. If I go to Force Quit, it shows "R (not responding)".
4. The "STOP" sign in R is greyed out and nonresponsive.

This is on a Mac Pro running Mavericks 10.9.4; 3GHz 8-core Intel Xeon E5, 64GB 1867 MHz DDR3.

Mike

---

Reply to this email directly or view it on GitHub: https://github.com/nolanlab/citrus/issues/71

[mleipold]

I tried it again, both ways:

1. 1 core, Set Multicore Usage box UNchecked- did not complete (R not responding; froze at same place)
2. 1 core, box checked: completed in ~ 2 minutes. Output files appear fine.

Also: forgot to say I'm running R 3.1.1....just in case that's an issue.

[rbruggner]

Can you email me the runCitrus.R file for one of the runs that fails?

On Aug 19, 2014, at 2:34 PM, mleipold notifications@github.com wrote:

I tried it again, both ways:

1 core, Set Multicore Usage box UNchecked- did not complete (R not responding; froze at same place)

1 core, box checked: completed in ~ 2 minutes. Output files appear fine.

Also: forgot to say I'm running R 3.1.1....just in case that's an issue.

— Reply to this email directly or view it on GitHub.

[rbruggner]

Something might be a little wonky with multicore usage in R3.1.1 but I haven't quite pinned it down yet.

One more thing to try would be to:

• Directly open the runCitrus.R file from a problematic run (one that froze) in R
• Set the number of Rclusterpp threads (line 9) to say, 4
• Set "mc.cores"=1 (line10).
• Run the entire file (in R, File->Source Document. In Rstudio, Code->Source).
• See if it finishes?

If it does end up working, may just have to cutout general parallel operations, but that shouldn't be a huge problem as most of the runtime is due to the clustering.

[mleipold]

I just tried modifying the runCitrus.R file as described.

The original version of the file that failed, I had designated 2 cores. I changed Line 9 to Rclusterpp "4" rather than "2", and Line 10 "mc.cores" to 1 (it had been 2 as well).

I Saved it, then ran directly in R (File->Source File), and the job did complete, slightly faster than one core (1min vs 1min20sec).

[rbruggner]

There appears to be some problems with mc multicore usage starting in R 3.1.1 or at least, that's my best guess. I've updated the UI to disable multicore usage by default. Update to the latest version in R by running:

library("devtools")
install_github("citrus","nolanlab")

By default, you should leave the "Set Multicore Usage" box unchecked.

Let me know if this new version passes your test and we'll close the issue.

[mleipold]

I have updated Citrus as detailed above.

Before I test on my data set, I want to make sure I understand: the disabling mentioned above means that 1) Multiple cores cannot be used?
2) That I just don't check the "Set Multicore Usage" box if I select 2+ cores? 3) That I'm able to"Set Multicore Usage" by clicking the box and indicating the number of cores, just that the default has changed to now use only one core? 4) None of the above

Basically, if I can use multiple cores, I'd like to (I have 8 cores). It's less critical for my small test-case dataset. But, as an example, I have another one of ~80 files that took 7 hours to run on one core.

[mleipold]

I tried it both ways: 8 core with box checked, and (presumably) 1 core, without box checked.

In each case, I got an error message, with no "Output Folder".

Here is the example from the (presumably) 1 core, no box-check run:

---

48 49 50 Error in dimnames(res.up) = temp.names : length of 'dimnames' [2] not equal to array extent

---

In each case, it got past the steps like

perm= 11 1 cluster abundance perm= 2026 and to "Computing delta table", then gave that error message.

These are the test files, so they have been clustered appropriately by the previous version of Citrus.

[rbruggner]

Option 4: the checkbox limits multicore usage during clustering. By default, the clustering package uses all available cores. If you leave the "set multicore usage" unchecked, it should use all cores for clustering.

I should rename this option to something clearer like "limit multicore usage".

On Aug 21, 2014, at 4:29 PM, mleipold notifications@github.com wrote:

I have updated Citrus as detailed above.

Before I test on my data set, I want to make sure I understand: the disabling mentioned above means that 1) Multiple cores cannot be used?

2) That I just don't check the "Set Multicore Usage" box if I select 2+ cores? 3) That I'm able to"Set Multicore Usage" by clicking the box and indicating the number of cores? 4) None of the above

Basically, if I can use multiple cores, I'd like to (I have 8 cores). It's less critical for my small test-case dataset. But, as an example, I have another one of ~80 files that took 7 hours to run on one core.

— Reply to this email directly or view it on GitHub.

[rbruggner]

Can you run a check of the file parameter consistencies for me as described here: https://github.com/nolanlab/citrus/wiki/Citrus-FAQ#How_do_I_tell_if_the_same_parameters_are_measured_in_all_my_FCS_files:

If the files pass that check, would it be possible to share the testing fcs file and runCitrus.R file with me via stanford.box.com or mss.stanford.edu so I can debug? If not, I can stop by and inspect on your computer.

On Aug 21, 2014, at 5:10 PM, mleipold notifications@github.com wrote:

I tried it both ways: 8 core with box checked, and (presumably) 1 core, without box checked.

In each case, I got an error message, with no "Output Folder".

Here is the example from the (presumably) 1 core, no box-check run:

48 49 50 Error in dimnames(res.up) = temp.names : length of 'dimnames' [2] not equal to array extent

In each case, it got past the steps like

perm= 11 1 cluster abundance perm= 2026 and to "Computing delta table", then gave that error message.

These are the test files, so they have been clustered appropriately by the previous version of Citrus.

— Reply to this email directly or view it on GitHub.

[mleipold]

I ran the consistency test as described:

citrus.checkFileParameterConsistencyUI() Reading parameters in G7-541278-OMRF.fcs Reading parameters in G7-541551-OMRF.fcs Reading parameters in G7-590088-OMRF.fcs Reading parameters in G8-500004-OMRF.fcs Reading parameters in G8-541290-OMRF.fcs Reading parameters in G8-541833-OMRF.fcs Reading parameters in G9-541305-OMRF.fcs Reading parameters in G9-541308-OMRF.fcs Reading parameters in G9-590117-OMRF.fcs [1] "FCS File parameters consistent." [1] 0

As I said, I've successfully run these in the previous version of Citrus, earlier in the week.

I tried it again just now, again with defaults (no box check), and it didn't work.

I also tried it just now, with defaults, but with box checked, one core specified. This time, it did complete.

I put the 9 FCS files into a Stanford Box and invited you to see them. These are FCS files originally obtained from a CyTOFv1, then exported from FlowJo 9.7.5 after gating out debris and removing a few parameters that were no longer necessary.

I also put the runCitrus.R files for 3 different runs of the new Citrus:

1. 8 core-box checked-failed
2. Default-no box checked-failed
3. 1 core-box checked-succeeded

[rbruggner]

Thanks Mike! I will check it out.

----- Original Message ----- From: "mleipold" notifications@github.com To: "nolanlab/citrus" citrus@noreply.github.com Cc: "Robert Bruggner" bruggner@stanford.edu Sent: Friday, August 22, 2014 8:39:31 AM Subject: Re: [citrus] Difficulty using multiple cores-R goes non-responsive (#71)

I ran the consistency test as described:

citrus.checkFileParameterConsistencyUI() Reading parameters in G7-541278-OMRF.fcs Reading parameters in G7-541551-OMRF.fcs Reading parameters in G7-590088-OMRF.fcs Reading parameters in G8-500004-OMRF.fcs Reading parameters in G8-541290-OMRF.fcs Reading parameters in G8-541833-OMRF.fcs Reading parameters in G9-541305-OMRF.fcs Reading parameters in G9-541308-OMRF.fcs Reading parameters in G9-590117-OMRF.fcs [1] "FCS File parameters consistent." [1] 0

As I said, I've successfully run these in the previous version of Citrus, earlier in the week.

I tried it again just now, again with defaults (no box check), and it didn't work.

I also tried it just now, with defaults, but with box checked, one core specified. This time, it did complete.

I put the 9 FCS files into a Stanford Box and invited you to see them. These are FCS files originally obtained from a CyTOFv1, then exported from FlowJo 9.7.5 after gating out debris and removing a few parameters that were no longer necessary.

---

Reply to this email directly or view it on GitHub: https://github.com/nolanlab/citrus/issues/71#issuecomment-53076497

[rbruggner]

Could you throw the runCitrus.R (working or non-working) in the shared directory as well? I just want to do my best to recreate the problematic situation.

----- Original Message ----- From: "mleipold" notifications@github.com To: "nolanlab/citrus" citrus@noreply.github.com Cc: "Robert Bruggner" bruggner@stanford.edu Sent: Friday, August 22, 2014 8:39:31 AM Subject: Re: [citrus] Difficulty using multiple cores-R goes non-responsive (#71)

I ran the consistency test as described:

citrus.checkFileParameterConsistencyUI() Reading parameters in G7-541278-OMRF.fcs Reading parameters in G7-541551-OMRF.fcs Reading parameters in G7-590088-OMRF.fcs Reading parameters in G8-500004-OMRF.fcs Reading parameters in G8-541290-OMRF.fcs Reading parameters in G8-541833-OMRF.fcs Reading parameters in G9-541305-OMRF.fcs Reading parameters in G9-541308-OMRF.fcs Reading parameters in G9-590117-OMRF.fcs [1] "FCS File parameters consistent." [1] 0

As I said, I've successfully run these in the previous version of Citrus, earlier in the week.

I tried it again just now, again with defaults (no box check), and it didn't work.

I also tried it just now, with defaults, but with box checked, one core specified. This time, it did complete.

I put the 9 FCS files into a Stanford Box and invited you to see them. These are FCS files originally obtained from a CyTOFv1, then exported from FlowJo 9.7.5 after gating out debris and removing a few parameters that were no longer necessary.

---

Reply to this email directly or view it on GitHub: https://github.com/nolanlab/citrus/issues/71#issuecomment-53076497

[rbruggner]

After taking a look, what you are seeing is a very specific problem within the samr package and is unrelated to multi-core operations. Basically, the SAM multiclass model has trouble formatting output when exactly one feature is found to be significant. I have notified the package maintainer of the problem and included a bug-fix so hopefully that will get rolled in to future updates.

Until this fix gets rolled in the package and distributed to CRAN, you have a couple of options:

• Don't use the SAM model during analysis
• Continue to use the SAM model during analysis and hope that you have either > 1 or 0 significant features. This is the option I think I would choose.
• Patch the samr package directly. Download the source and in file R/samr.morefuns.R, change line 580 from

 samr.obj$sd[sig$pup], samr.obj$stand.contrasts[sig$pup, ], qvalues$qvalue.up)

to

 samr.obj$sd[sig$pup], samr.obj$stand.contrasts[sig$pup, ,drop=F], qvalues$qvalue.up)

and then install the samr package from source:

shell> R CMD INSTALL /path/to/samr_source

Again, this problem does not relate to multicore usage so I would recommend continuing to run as described before - i.e. leave the "Set Multicore Usage" box unchecked in the GUI.