search

RGLab / CytoML

A GatingML Interface for Cross Platform Cytometry Data Sharing
GNU Affero General Public License v3.0
30 stars 14 forks source link

gatingset_to_flowjo Docker issue #108

Closed Sithara85 closed 4 years ago

Sithara85 commented 4 years ago

Hi,

I am trying to create a FlowJo xml .wsp file from a gating set in R. I had issues with docker and the cytolib/CytoML versions and fixed most of the issues as advised in the ticket - https://github.com/RGLab/CytoML/issues/97

Please see attached error I am getting when I use the sample gating set from CytoMl documentation - https://rdrr.io/bioc/CytoML/man/gatingset_to_flowjo.html.

path <- system.file("extdata",package="flowWorkspaceData")

gs_path <- list.files(path, pattern = "gs_manual",full = TRUE) gs <- load_gs(gs_path)

output to flowJo

outFile <- tempfile(fileext = ".wsp") gatingset_to_flowjo(gs, outFile) INFO: Could not find files for the given pattern(s). Error in gatingset_to_flowjo(gs, outFile) :   'docker' is not running properly! 

This is the first time I am using Docker image. I used admin privilege in Powershell to pull the docker image below:

PS C:\Program Files\Docker\Docker> docker pull rglab/gs-to-flowjo:devel devel: Pulling from rglab/gs-to-flowjo df20fa9351a1: Pull complete                                                                                                                                                                  df33d3a3b5af: Pull complete                                                                                                                                                                  d8b7beb12242: Pull complete                                                                                                                                                                  a4e3f3e3cbb1: Pull complete                                                                                                                                                                  Digest: sha256:5239ba3089a19f50dd985d8b0b29048020feeb16fe808c55a6bb396b9e75748c Status: Downloaded newer image for rglab/gs-to-flowjo:devel docker.io/rglab/gs-to-flowjo:devel

My R sessionInfo() is below to verify package versions:

R version 4.0.2 (2020-06-22) Platform: x86_64-w64-mingw32/x64 (64-bit) Running under: Windows >= 8 x64 (build 9200)

Matrix products: default

Random number generation:  RNG:     Mersenne-Twister  Normal:  Inversion  Sample:  Rounding   locale: [1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252    LC_MONETARY=English_United States.1252 [4] LC_NUMERIC=C                           LC_TIME=English_United States.1252    

attached base packages: [1] stats     graphics  grDevices utils     datasets  methods   base    

other attached packages:  [1] BioInstaller_0.3.7        cytolib_2.1.12            CytoML_2.1.9              scales_1.1.1              ClusterR_1.2.2            [6] gtools_3.8.2              ggcyto_1.17.0             ncdfFlow_2.35.1           BH_1.72.0-3               RcppArmadillo_0.9.900.1.0 [11] ggplot2_3.3.2             flowWorkspace_4.1.7       flowCore_2.1.1            openCyto_2.1.2           

jacobpwagner commented 4 years ago

Alternatively, if you mount it at /gs, as you just did, can you see the contents in /gs? What I'm getting at is gs-to-flowjo probably isn't seeing the directory at all because it's not being mounted at the specified location or mounted at all. I'm just trying to figure out why as that's a singularity issue, not a CytoML issue.

Sithara85 commented 4 years ago

Hi Jake,

I think I figured out how to point to the correct directory. Now I didn't get any domain error but no output in dest directory. I will add -B command before I open the container.

Singularity> ls bin environment gs-to-flowjo lib mnt proc run singularity sys usr dev etc home media opt root sbin srv tmp var

Singularity> ls home/thyagara/svivek/Singularity/src 2017-10-13_PANEL-1_LSR_EC_Group-one_RR_F1631478_024.fcs_gsEM3 2017-10-13_PANEL-1_LSR_EC_Group-two_EC_F1631568_023.fcs_gsEM3 2017-10-13_PANEL-1_LSR_EC_Group-one_RR_F1631761_025.fcs_gsEM3 2017-10-13_PANEL-1_LSR_EC_Group-two_EC_F1631660_021.fcs_gsEM3 2017-10-13_PANEL-1_LSR_EC_Group-one_RR_F1631780_027.fcs_gsEM3 2017-10-13_PANEL-1_LSR_EC_Group-two_EC_F1631889_019.fcs_gsEM3 2017-10-13_PANEL-1_LSR_EC_Group-one_RR_F1631896_026.fcs_gsEM3 2017-10-13_PANEL-1_LSR_EC_Group-two_EC_F1631894_020.fcs_gsEM3 2017-10-13_PANEL-1_LSR_EC_Group-one_RR_F1631898_028.fcs_gsEM3 2017-10-13_PANEL-1_LSR_EC_Group-two_EC_F1638929_022.fcs_gsEM3

Singularity> ./gs-to-flowjo --src=home/thyagara/svivek/Singularity/src/2017-10-13_PANEL-1_LSR_EC_Group-one_RR_F1631478_024.fcs_gsEM3 --dest=home/thyagara/svi vek/Singularity/dest Singularity>

Sithara85 commented 4 years ago

No luck even after I -B to mount directory :( I don't think it is really necessary if we can see the contents without mount. Do you see any obvious mistake?

svivek@ln0004 [/panfs/roc/scratch/svivek/HRS_EM/Helper_EM/flow] % singularity shell ./gs-to-flowjo_devel.sif -B home/thyagara/svivek/Singularity/src -B /thyagara/svivek/Singularity/dest

Singularity> cd /

Singularity> ./gs-to-flowjo --src=home/thyagara/svivek/Singularity/src/2017-10-13_PANEL-1_LSR_EC_Group-one_RR_F1631478_024.fcs_gsEM3 --dest=home/thyagara/svi vek/Singularity/dest

jacobpwagner commented 4 years ago

I don't know what to tell you. I just verified that I can do this fine from a singularity shell:

Outside container in directory with input gs_manual and output directory out, then starting the shell session in the container:

$ sudo singularity shell -B gs_manual:/gs -B out:/out gs-to-flowjo_devel.sif 
Singularity: Invoking an interactive shell within container...

Singularity> ls /gs
63c60e66-779b-4712-ad6e-b9349b965401.gs  CytoTrol_CytoTrol_1.fcs.h5               CytoTrol_CytoTrol_1.fcs.pb
Singularity> ls /out
Singularity> cd /
Singularity> ls
bin           environment   gs            home          media         opt           proc          run           singularity   sys           usr
dev           etc           gs-to-flowjo  lib           mnt           out           root          sbin          srv           tmp           var
Singularity> ./gs-to-flowjo --src=/gs --dest=/out/converted.wsp
Singularity> ls /out
converted.wsp
Singularity> head /out/converted.wsp 
<?xml version="1.0"?>
<!--This file is generated by Community edition of cytolib software (2.1.15-1.0) that is restricted for academic users-->
<Workspace version="20.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:gating="http://www.isac-net.org/std/Gating-ML/v2.0/gating" xmlns:transforms="http://www.isac-net.org/std/Gating-ML/v2.0/transformations" xmlns:data-type="http://www.isac-net.org/std/Gating-ML/v2.0/datatypes" xsi:schemaLocation="http://www.isac-net.org/std/Gating-ML/v2.0/gating http://www.isac-net.org/std/Gating-ML/v2.0/gating/Gating-ML.v2.0.xsd http://www.isac-net.org/std/Gating-ML/v2.0/transformations http://www.isac-net.org/std/Gating-ML/v2.0/gating/Transformations.v2.0.xsd http://www.isac-net.org/std/Gating-ML/v2.0/datatypes http://www.isac-net.org/std/Gating-ML/v2.0/gating/DataTypes.v2.0.xsd">
 <Groups>
  <GroupNode name="All Samples">
   <Group name="All Samples">
    <SampleRefs>
     <SampleRef sampleID="1" />
    </SampleRefs>
   </Group>
Singularity> exit

Confirming that the converted wsp file exists back outside the container:

$ ls out/
converted.wsp
$ head out/converted.wsp 
<?xml version="1.0"?>
<!--This file is generated by Community edition of cytolib software (2.1.15-1.0) that is restricted for academic users-->
<Workspace version="20.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:gating="http://www.isac-net.org/std/Gating-ML/v2.0/gating" xmlns:transforms="http://www.isac-net.org/std/Gating-ML/v2.0/transformations" xmlns:data-type="http://www.isac-net.org/std/Gating-ML/v2.0/datatypes" xsi:schemaLocation="http://www.isac-net.org/std/Gating-ML/v2.0/gating http://www.isac-net.org/std/Gating-ML/v2.0/gating/Gating-ML.v2.0.xsd http://www.isac-net.org/std/Gating-ML/v2.0/transformations http://www.isac-net.org/std/Gating-ML/v2.0/gating/Transformations.v2.0.xsd http://www.isac-net.org/std/Gating-ML/v2.0/datatypes http://www.isac-net.org/std/Gating-ML/v2.0/gating/DataTypes.v2.0.xsd">
 <Groups>
  <GroupNode name="All Samples">
   <Group name="All Samples">
    <SampleRefs>
     <SampleRef sampleID="1" />
    </SampleRefs>
   </Group>
jacobpwagner commented 4 years ago

Also, if I put the bind options after the image as you did, I get an error, presumably again because those are getting passed to the entry point (in this case /bin/sh because it's a shell session) rather than the singularity shell command:

$ sudo singularity shell gs-to-flowjo_devel.sif -B gs_manual:/gs -B out:/out
Singularity: Invoking an interactive shell within container...

/bin/sh: illegal option -B

From the official docs, the bind options have to come first:

singularity shell [shell options...] <container>
Sithara85 commented 4 years ago

This all makes sense! I will pull the image again and redo these steps. I totally agree it’s something with the container. Thanks again for checking all the steps with me. I very much appreciate it.

Sithara

Sent from my iPhone

On Aug 6, 2020, at 2:16 PM, Jake Wagner notifications@github.com wrote:

 Also, if I put the bind options after the image as you did, I get an error, presumably again because those are getting passed to the entry point (in this case /bin/sh because it's a shell session) rather than the singularity shell command:

$ sudo singularity shell gs-to-flowjo_devel.sif -B gs_manual:/gs -B out:/out Singularity: Invoking an interactive shell within container...

/bin/sh: illegal option -B — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

Sithara85 commented 4 years ago

Jake,

It worked, the issue was I was mounting the directory where I had separate directories for each samples. I just tried to mount and listed the contents as you have detailed and it helped. Now I will have to run it with an Rscript and loop through samples to make sure it works fine. Thanks a million!!!

jacobpwagner commented 4 years ago

No problem. And yeah, one call to gs-to-flowjo converts one GatingSet to one wsp. You can loop in the Rscript as you mentioned or if you were using it in a shell session in the container you could make repeated calls there. Up to you.

jacobpwagner commented 4 years ago

@Sithara85 , so all is well now? Can I re-close this?

Sithara85 commented 4 years ago

Yes please. I was working to update my Rscript to use system2() to invoke singularity container. Thanks a lot for your patience and hard work. I will close the ticket for you.

jacobpwagner commented 4 years ago

In the long run we could also maybe look at trying Docker, then Singularity within the body of CytoML::gatingset_to_flowjo as well ,as opposed to just Docker.

Sithara85 commented 4 years ago

Sure! That might be easier to implement. Thank you! Have a great weekend.

Sithara85 commented 4 years ago

Hi,

As you know we have been working on ~10,000 samples FCS files and generated gating set and workspace files. Now we were trying to open the wsp files in FlowJO and it is not opening properly. It shows a red 'X' on the filename in FlowJo and can't find associated fcs so we can't see the populations gated.

Do you have any idea what's going on? We were able to visualize the gates in FlowJo once we used openCyto 3 years ago.

I have used the below code to create the gating set and then ran docker/ singularity to create wsp files.

template = gs_add_gating_method( gs_EM3, alias = "CD28Gate", pop = "+/-", parent = "root", dims = "CD28", gating_method = "mindensity", gating_args = "gate_range = c(1000,15000)") print(gs_pop_get_count_fast(gs_EM3)) fcs_file <- basename(f)

outputRoot = paste0(opt$outputDir, fcs_file) addedWSP = paste0(outputRoot,"_CTEM3") addedWSP = gsub(" ", "", addedWSP) print(paste0("adding and renaming nodes and writing to ", addedWSP)) save_gs(gs_EM3,addedWSP) print (file.exists(addedWSP))

}

Thank you, Sithara