Error-tSNE continues to be performed even when de-selected in GUI

[mleipold]

Hi,

I'm trying to do a huge analysis (FlowSOM on >400 files without downsampling), and I had issues with tSNE crashing the R session after days of trying.

So, I restarted my analysis with the tSNE button "unchecked". See screenshot.

However, when I hit Submit to start the job, RStudio still gave me this:

• Data merging method: all
• Data transformation method: cytofAsinh
• Dimensionality reduction method: tsne
• Data clustering method(s): Rphenograph FlowSOM
• Data visualization method(s): pca
• Subset progression analysis method: NULL "

So, tSNE is still being performed as a dimensionality reduction.

I didn't think that Rphenograph or FlowSOM require tSNE for clustering (unlike, say, ACCENSE).

Is this tSNE process required in some fashion?

Mike

[SamGG]

Hi Mike, The code is forcing the dimension reduction method to be tsne. There seems to be no infotmation from the GUI that is used to fill parameter. https://github.com/JinmiaoChenLab/cytofkit/blob/c4f93e5d849cf670c3825e5abacd850822c8eca7/R/cytofkit_GUI.R#L531-L533 The cytofkit function allows 3 dimension reduction methods (L136). This dimension reduction is independent from the visualisation (L138). https://github.com/JinmiaoChenLab/cytofkit/blob/c4f93e5d849cf670c3825e5abacd850822c8eca7/R/cytofkit.R#L136-L138 The dimension reduction function itself allows more methods but I stick on the main 3. https://github.com/JinmiaoChenLab/cytofkit/blob/c4f93e5d849cf670c3825e5abacd850822c8eca7/R/cytof_dimensionReduction.R#L34

So, because you do so much on cytoforum, here is for you (PS: it's very late and I didn't check if it makes sense). Run

library(flowCore)
library(cytofkit)
# open and source cytofkit_GUI_mod.R
cytofkit_GUI()

A line for dimension reduction has been added and initialized with tSNE, but change it to PCA and deselect tsne. A very short trial and the modified code are following. Download the text file containing the code from the link at the end of this post and rename it as cytofkit_GUI_mod.R. Hope it will work for you.

Notice the Dim. Red. parameter.

While running PCA is stated as dimension reduction method.

The results in the Shiny app shows the PCA.

The code is in the next link cytofkit_GUI_mod.R.txt

[mleipold]

Hi Samuel,

Thanks for the response.

I downloaded the cytofkit_GUI_mod.R.txt, changed it to cytofkit_GUI_mod.R (no .txt), placed it in the cytofkit R folder, and ran the GUI. I did not get the "Dim. red. method(s)" to show.

I thought maybe the original cytofkit_GUI.R was interfering, so I removed it from the folder and reran the GUI. Still no "Dim.red".

I noticed that the removal of the "txt" in a Mac environment did not give the little "R" type icon. So, I copied the text into an RStudio pane, and saved it cytofkit_GUI.R. This time I got the "R" type icon, so I reran the GUI. Still no "Dim.red" in the GUI.

Is there something I'm leaving out?

BTW, I'm in Mac High Sierra 10.13.4, RStudio 1.1.442, R 3.4.4

Mike

[SamGG]

Hi Mike, Sound over complicated. I didn't explain clearly. I guess you are using RStudio. Load cytofkit library as usual. Before running cytofkit_GUI(), open the cytofkit_GUI_mod.R file with RStudio, then click on Source (Ctrl+Shift+S). This loads the function. Then run the GUI. Let me know.

[mleipold]

Hi Samuel,

I downloaded the "cytofkit_GUI_mod.R.txt" again just to have a clean copy. I removed the ".txt", then loaded it into RStudio as you described.

---

library("cytofkit", lib.loc="/Library/Frameworks/R.framework/Versions/3.4/Resources/library") Loading required package: ggplot2 Loading required package: plyr source('~/Downloads/JinmiaoChenLab-cytofkit-c4f93e5/R/cytofkit_GUI_mod.R') library(flowCore)

Attaching package: ‘flowCore’

The following object is masked from ‘package:base’:

sort

cytofkit_GUI() Error in tclVar(cur_dir) : could not find function "tclVar" source('~/Desktop/cytofkit_GUI_mod.R') cytofkit_GUI() Error in tclVar(cur_dir) : could not find function "tclVar"

I haven't seen this "tclVar" error before.

[SamGG]

Hi Mike, Sounds like you need to load tcltk library explicitly: library(tcltk) This library should have already been loaded transparently. If you get another error at that stage, try to run the standard cytofkit_GUI using cytofkit::cytofkit_GUI() in order to load required packages, quit the GUI and rerun the modified GUI version cytofkit_GUI() . Let me know.

[mleipold]

hi Samuel,

That fixed it. I now get the GUI, and see the Dim.Red. section.

Thanks for your help! Mike

[mleipold]

I tried running the above overnight, and got an error message that killed the operation (note: Traceback info at the very bottom): right after Rphenograph started, this message appeared: "Error in nn2(data, data, k, treetype = "kd", searchtype = "standard") : long vectors (argument 11) are not supported in .C "

• Markers: -Bi209Di<209Bi_CD16> -Dy161Di<161Dy_CD26> -Dy162Di<162Dy_CD8a> -Dy163Di<163Dy_CD33> -Dy164Di<164Dy_CD161> -Er167Di<167Er_CCR7> -Er170Di<170Er_CD3> -Eu153Di<153Eu_CD45RA> -Gd155Di<155Gd_CD27> -Gd156Di<156Gd_CXCR3> -Gd158Di<158Gd_NKG2A> -Gd160Di<160Gd_CD14> -In115Di<115In_HLA-DR> -Nd142Di<142Nd_CD19> -Nd144Di<144Nd_IgG> -Nd145Di<145Nd_CD4> -Nd146Di<146Nd_IgD> -Nd148Di<148Nd_IgA> -Pr141Di<141Pr_CCR6> -Sm147Di<147Sm_CD20> -Sm149Di<149Sm_CD56> -Sm152Di<152Sm_TCRgd> -Sm154Di<154Sm_CD123> -Tb159Di<159Tb_CD11c> -Yb171Di<171Yb_CXCR5> -Yb172Di<172Yb_CD38>
• Data merging method: all
• Data transformation method: cytofAsinh
• Dimensionality reduction method: pca
• Data clustering method(s): Rphenograph FlowSOM
• Data visualization method(s): pca
• Subset progression analysis method: NULL

Extract expression data... 48376032 x 50 data was extracted! Dimension reduction... Running PCA... DONE Run clustering... Running PhenoGraph...Run Rphenograph starts: -Input data of 48376032 rows and 26 columns -k is set to 100 Finding nearest neighbors... Hide Traceback

Rerun with Debug Error in nn2(data, data, k, treetype = "kd", searchtype = "standard") : long vectors (argument 11) are not supported in .C 9. nn2(data, data, k, treetype = "kd", searchtype = "standard") 8. find_neighbors(data, k = k + 1) 7. system.time(neighborMatrix <- find_neighbors(data, k = k + 1)[, -1]) 6. Rphenograph(xdata, k = Rphenograph_k) 5. membership(Rphenograph(xdata, k = Rphenograph_k)) 4. FUN(X[[i]], ...) 3. lapply(clusterMethods, cytof_cluster, ydata = allDimReducedList[[dimReductionMethod]], xdata = exprs_data[, markers], Rphenograph_k = Rphenograph_k, FlowSOM_k = FlowSOM_k, flowSeed = seed) 2. cytofkit(fcsFiles = inputs[["fcsFiles"]], markers = inputs[["markers"]], projectName = inputs[["projectName"]], mergeMethod = inputs[["mergeMethod"]], fixedNum = inputs[["fixedNum"]], transformMethod = inputs[["transformMethod"]], dimReductionMethod = inputs[["dimReductionMethod"]], clusterMethods = inputs[["clusterMethods"]], ... at cytofkit_GUI_mod.R#603 1. cytofkit_GUI()

I tried rerunning it this morning, and got the same error.

[SamGG]

Hi Mike, I think that's a problem of the Phenograph implementation. It's part of the logic of cytofkit and out of the scope of my ability. The nearest neighbor algorithm sounds to have problem in facing 48 10e6 points. I am not sure that it is possible in a reasonable time either. This part could be paralleled, but it was not. I would suggest you to run cytofkit without Phenograph, at least for testing. flowSOM might be OK on that huge dataset. flowSOM (using a custom script) allows one to buid the grid (aka clusterize) using a reduced set of representatives FCS. Then it's possible to map many FCS on that grid. Because representative FCS have been used to define the grid, rare populations that could be found in a unique FCS should be found also with flowSOM. Best.