search

RGLab / CytoML

A GatingML Interface for Cross Platform Cytometry Data Sharing
GNU Affero General Public License v3.0
29 stars 14 forks source link

diva_to_gatingset: option required to *not* use compensation matrix from xml #128

Open RPdavies opened 3 years ago

RPdavies commented 3 years ago

Hi,

In cases where a compensation matrix was not created in Diva, diva_to_gatingset attempts to find one in the xml file and returns an error: Error in solve.default(comp) : 'a' (2 x 4) must be square.

This makes importing xml gates from Diva impossible in many situations. Let me know if you require an example xlm / fcs set.

mikejiang commented 3 years ago

yeah, send the link or data to mike@ozette.ai. Thanks!

RPdavies commented 3 years ago

No problem, I've sent it over just now. Please let me know if the link didn't work for some reason. Thank you!

mikejiang commented 3 years ago

There is currently no elegant way to decide whether xml compensation should/can be used since compensation for this xml is actually present, but as dummy setting, thus caused the parse to be confused.

<compensation>
<compensation_coefficient>0.0</compensation_coefficient>
<compensation_coefficient>1.0</compensation_coefficient>
</compensation>

I've pushed the easy workaround as you suggested, i.e. optionally disable it 

  gs <- diva_to_gatingset(ws, name = 1, xml_compensation_enabled = FALSE)
RPdavies commented 3 years ago

Awesome, that does the job. Look forward to using CytoML!

nvannest-absci commented 3 years ago

I also have a dummy compensation issue in my xml file.

Screen Shot 2021-05-26 at 11 40 16 AM

I tried setting xml_compensation_enabled to either TRUE or FALSE and both times I received the same error. Are there alternative solutions to this problem?

Screen Shot 2021-05-24 at 2 58 30 PM

I am wondering if it might be easier to create a function that removes dummy compensation data from diva xml files. I tried removing the code listed above from my xml file but I received the following error.

Screen Shot 2021-05-26 at 11 42 48 AM
mikejiang commented 3 years ago

Can you show the error and traceback() output when you set xml_compensation_enabled = F ?

nvannest-absci commented 3 years ago
Screen Shot 2021-05-27 at 2 19 23 PM

I don't know much about these functions, but it seems like diva_to_gatingset is not passing xml_compensation_enabled = F to .parseDivaWorkspace when it is called. I could be misinterpreting things however.

mikejiang commented 3 years ago

You may need to pull the latest version from github.

nvannest-absci commented 3 years ago

I am currently having trouble running the diva_to_gatingset function from the CytoML package

I am using CytoML version 2.4.0. available through biocManager My experiment lacks any compensation, but my XML from Diva includes placeholder values. An example is shown below:

> library(flowWorkspace)
> library(CytoML)
> setwd("/Users/user1/Desktop/DummyProject_Folder/DummyExp")
> flowData <- getwd()

> gatingFile <- list.files(flowData, pattern=".xml",full = TRUE)
> ws <- open_diva_xml(gatingFile)
> ws
> diva_get_sample_groups(ws)
> gs <- diva_to_gatingset(ws)
> sampleNames(gs)
> gs_get_pop_paths(gs)
> plotGate(gs[[1]])

> sessionInfo()

Diva Workspace Version  Version 9.1.2 
File location:  /Users/user1/Desktop/DummyProject_Folder/DummyExp
File name:  DummyExp.xml 
Workspace is open. 

Groups in Workspace
> diva_get_sample_groups(ws)
> gs <- diva_to_gatingset(ws)
Choose which group of samples to import:

1: DummySpecimen1

> Selection: 1
Parsing 2 samples
loading data: /Users/nvannest/Desktop/ACE openCyto Test/DummyProject_Folder/DummyExp_Exp_Export/DummyExp_DummySpecimen1_DummyTube1.fcs
swap cols: FSC-H:FSC-W
swap cols: SSC-H:SSC-W
Compensating
Error in solve.default(comp) : 'a' (7 x 14) must be square )

When I try to install the version of CytoML available through github I get an installation error

> devtools::install_github("RGLab/CytoML")
Downloading GitHub repo RGLab/CytoML@HEAD
These packages have more recent versions available.
It is recommended to update all of them.
Which would you like to update?

1: All                                  
2: CRAN packages only                   
3: None                                 
4: robustbase  (0.93-7 -> 0.93-8) [CRAN]
5: matrixStats (0.58.0 -> 0.59.0) [CRAN]
6: ks          (1.13.0 -> 1.13.1) [CRAN]

> Enter one or more numbers, or an empty line to skip updates: 1
robustbase  (0.93-7 -> 0.93-8) [CRAN]
matrixStats (0.58.0 -> 0.59.0) [CRAN]
ks          (1.13.0 -> 1.13.1) [CRAN]
Installing 3 packages: robustbase, matrixStats, ks

  There are binary versions available but the source versions are later:
            binary source needs_compilation
robustbase  0.93-7 0.93-8              TRUE
matrixStats 0.58.0 0.59.0              TRUE
ks          1.13.0 1.13.1              TRUE

> Do you want to install from sources the packages which need compilation? (Yes/no/cancel) no
trying URL 'https://cran.rstudio.com/bin/macosx/contrib/4.1/robustbase_0.93-7.tgz'
Content type 'application/x-gzip' length 3123297 bytes (3.0 MB)
==================================================
downloaded 3.0 MB

trying URL 'https://cran.rstudio.com/bin/macosx/contrib/4.1/matrixStats_0.58.0.tgz'
Content type 'application/x-gzip' length 2019790 bytes (1.9 MB)
==================================================
downloaded 1.9 MB

trying URL 'https://cran.rstudio.com/bin/macosx/contrib/4.1/ks_1.13.0.tgz'
Content type 'application/x-gzip' length 1851632 bytes (1.8 MB)
==================================================
downloaded 1.8 MB

The downloaded binary packages are in
    /var/folders/_g/xp8hp7mn02z122v2092lzw499zps22/T//RtmpcPvBBy/downloaded_packages
✓  checking for file ‘/private/var/folders/_g/xp8hp7mn02z122v2092lzw499zps22/T/RtmpcPvBBy/remotes132e432e8405d/RGLab-CytoML-d544a00/DESCRIPTION’ ...
─  preparing ‘CytoML’:
✓  checking DESCRIPTION meta-information ...
─  cleaning src
─  checking for LF line-endings in source and make files and shell scripts
─  checking for empty or unneeded directories
   Omitted ‘LazyData’ from DESCRIPTION
     NB: this package now depends on R (>= 3.5.0)
     WARNING: Added dependency on R >= 3.5.0 because serialized objects in
     serialize/load version 3 cannot be read in older versions of R.
     File(s) containing such objects:
       ‘CytoML/tests/testthat/expect_result/flowJoWorkspace_expect.rds’
─  building ‘CytoML_2.5.1.tar.gz’
   Warning in sprintf(gettext(fmt, domain = domain), ...) :
     one argument not used by format 'invalid uid value replaced by that for user 'nobody''
   Warning: invalid uid value replaced by that for user 'nobody'
   Warning in sprintf(gettext(fmt, domain = domain), ...) :
     one argument not used by format 'invalid gid value replaced by that for user 'nobody''
   Warning: invalid gid value replaced by that for user 'nobody'

* installing *source* package ‘CytoML’ ...
** using staged installation
checking whether the C++ compiler works... yes
checking for C++ compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C++ compiler... yes
checking whether clang++ -mmacosx-version-min=10.13 -std=gnu++11 accepts -g... yes
checking for gcc... clang -mmacosx-version-min=10.13
checking whether we are using the GNU C compiler... yes
checking whether clang -mmacosx-version-min=10.13 accepts -g... yes
checking for clang -mmacosx-version-min=10.13 option to accept ISO C89... none needed
configure: setting xml2 flags...
configure: No directory was specified for --with-xml2. Trying to find xml2 using other methods.
checking for xml2-config... /usr/bin/xml2-config
configure: setting cytolib-ml commandline tool path...
configure: Using the following compilation and linking flags
configure:    PKG_CPPFLAGS=-I/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include -ftemplate-depth=900
configure:    PKG_LIBS=-L/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/lib -lxml2 -lz -lpthread -licucore -lm 
configure:    CYTOLIBML_BIN=/usr/local/bin
configure: creating ./config.status
config.status: creating src/Makevars
config.status: creating R/cytolibml_bin_path.R
** libs
clang++ -mmacosx-version-min=10.13 -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DROUT -I../inst/include/ -I/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include -ftemplate-depth=900 -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.1/Resources/library/Rcpp/include' -I'/Library/Frameworks/R.framework/Versions/4.1/Resources/library/BH/include' -I'/Library/Frameworks/R.framework/Versions/4.1/Resources/library/RProtoBufLib/include' -I'/Library/Frameworks/R.framework/Versions/4.1/Resources/library/cytolib/include' -I'/Library/Frameworks/R.framework/Versions/4.1/Resources/library/Rhdf5lib/include' -I'/Library/Frameworks/R.framework/Versions/4.1/Resources/library/RcppArmadillo/include' -I'/Library/Frameworks/R.framework/Versions/4.1/Resources/library/RcppParallel/include' -I'/Library/Frameworks/R.framework/Versions/4.1/Resources/library/flowWorkspace/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c RcppExports.cpp -o RcppExports.o
In file included from RcppExports.cpp:4:
In file included from ./../inst/include/CytoML.h:4:
In file included from /Library/Frameworks/R.framework/Versions/4.1/Resources/library/flowWorkspace/include/flowWorkspace.h:5:
In file included from /Library/Frameworks/R.framework/Versions/4.1/Resources/library/cytolib/include/cytolib/GatingSet.hpp:14:
In file included from /Library/Frameworks/R.framework/Versions/4.1/Resources/library/cytolib/include/cytolib/GatingHierarchy.hpp:17:
In file included from /Library/Frameworks/R.framework/Versions/4.1/Resources/library/cytolib/include/cytolib/populationTree.hpp:14:
In file included from /Library/Frameworks/R.framework/Versions/4.1/Resources/library/BH/include/boost/graph/adjacency_list.hpp:255:
In file included from /Library/Frameworks/R.framework/Versions/4.1/Resources/library/BH/include/boost/graph/detail/adjacency_list.hpp:37:
In file included from /Library/Frameworks/R.framework/Versions/4.1/Resources/library/BH/include/boost/graph/adjacency_iterator.hpp:13:
/Library/Frameworks/R.framework/Versions/4.1/Resources/library/BH/include/boost/detail/iterator.hpp:13:1: warning: This header is deprecated. Use <iterator> instead. [-W#pragma-messages]
BOOST_HEADER_DEPRECATED("<iterator>")
^
/Library/Frameworks/R.framework/Versions/4.1/Resources/library/BH/include/boost/config/header_deprecated.hpp:23:37: note: expanded from macro 'BOOST_HEADER_DEPRECATED'
# define BOOST_HEADER_DEPRECATED(a) BOOST_PRAGMA_MESSAGE("This header is deprecated. Use " a " instead.")
                                    ^
/Library/Frameworks/R.framework/Versions/4.1/Resources/library/BH/include/boost/config/pragma_message.hpp:24:34: note: expanded from macro 'BOOST_PRAGMA_MESSAGE'
# define BOOST_PRAGMA_MESSAGE(x) _Pragma(BOOST_STRINGIZE(message(x)))
                                 ^
<scratch space>:12:2: note: expanded from here
 message("This header is deprecated. Use " "<iterator>" " instead.")
 ^
In file included from RcppExports.cpp:4:
In file included from ./../inst/include/CytoML.h:4:
In file included from /Library/Frameworks/R.framework/Versions/4.1/Resources/library/flowWorkspace/include/flowWorkspace.h:6:
In file included from /Library/Frameworks/R.framework/Versions/4.1/Resources/library/flowWorkspace/include/flowWorkspace/pairVectorRcppWrap.h:15:
In file included from /Library/Frameworks/R.framework/Versions/4.1/Resources/library/RcppArmadillo/include/RcppArmadillo.h:34:
In file included from /Library/Frameworks/R.framework/Versions/4.1/Resources/library/Rcpp/include/Rcpp.h:57:
/Library/Frameworks/R.framework/Versions/4.1/Resources/library/Rcpp/include/Rcpp/DataFrame.h:136:18: warning: unused variable 'data' [-Wunused-variable]
            SEXP data = Parent::get__();
                 ^
In file included from RcppExports.cpp:4:
In file included from ./../inst/include/CytoML.h:5:
In file included from ../inst/include/CytoML/openWorkspace.hpp:12:
../inst/include/CytoML/winFlowJoWorkspace.hpp:435:30: warning: '/*' within block comment [-Wcomment]
           * "*[local-name()='edge']/*[local-name()='vertex']" is for ellipsoidGate
                                    ^
../inst/include/CytoML/winFlowJoWorkspace.hpp:866:18: error: no member named 'setDataOnlyFlag' in 'cytolib::scaleTrans'
                                    curTran->setDataOnlyFlag(true);
                                    ~~~~~~~  ^
../inst/include/CytoML/winFlowJoWorkspace.hpp:882:16: error: no member named 'setDataOnlyFlag' in 'cytolib::scaleTrans'
                                  curTran->setDataOnlyFlag(false);
                                  ~~~~~~~  ^
3 warnings and 2 errors generated.
make: *** [RcppExports.o] Error 1
ERROR: compilation failed for package ‘CytoML’
* removing ‘/Library/Frameworks/R.framework/Versions/4.1/Resources/library/CytoML’
* restoring previous ‘/Library/Frameworks/R.framework/Versions/4.1/Resources/library/CytoML’
Warning message:
In i.p(...) :
  installation of package ‘/var/folders/_g/xp8hp7mn02z122v2092lzw499zps22/T//RtmpcPvBBy/file132e439dffd5e/CytoML_2.5.1.tar.gz’ had non-zero exit status

Here is additional information about my setup:

R version 4.1.0 (2021-05-18) -- "Camp Pontanezen"
Copyright (C) 2021 The R Foundation for Statistical Computing
Platform: x86_64-apple-darwin17.0 (64-bit)

BiocManager version 1.30.15

> tools:::.BioC_version_associated_with_R_version()
[1] ‘3.13’

Any help would be very appreciated. Thanks!

mikejiang commented 3 years ago

try to update cytolib and flowWorkspace from github first

nvannest-absci commented 3 years ago

I've updated cytolib and flowWorkspace directly from github and continue to get the same error

Input:

library(flowWorkspace)
library(CytoML)

setwd("/Users/user1/Desktop/Flow_Data_Import_Test/DummyProject_Folder/DummyExp_Diva_Export/DummyExp_Diva_Exp_Export")
flowData <- getwd()

gatingFile <- list.files(flowData, pattern=".xml",full = TRUE)
ws <- open_diva_xml(gatingFile)
ws
diva_get_sample_groups(ws)
gs <- diva_to_gatingset(ws)
sampleNames(gs)
gs_get_pop_paths(gs)
plotGate(gs[[1]])

Output:

> setwd("/Users/user1/Desktop/Flow_Data_Import_Test/DummyProject_Folder/DummyExp_Diva_Export/DummyExp_Diva_Exp_Export")
The working directory was changed to /Users/user1/Desktop/Flow_Data_Import_Test/DummyProject_Folder/DummyExp_Diva_Export/DummyExp_Diva_Exp_Export inside a notebook chunk. The working directory will be reset when the chunk is finished running. Use the knitr root.dir option in the setup chunk to change the working directory for notebook chunks.> flowData <- getwd()
> 
> gatingFile <- list.files(flowData, pattern=".xml",full = TRUE)
> ws <- open_diva_xml(gatingFile)
> ws
Diva Workspace Version  Version 9.1.2 
File location:  /Users/user1/Desktop/Flow_Data_Import_Test/DummyProject_Folder/DummyExp_Diva_Export/DummyExp_Diva_Exp_Export 
File name:  DummyExp.xml 
Workspace is open. 

Groups in Workspace
> diva_get_sample_groups(ws)
> gs <- diva_to_gatingset(ws)
Choose which group of samples to import:

1: DummySpecimen1

Selection: 1
Parsing 2 samples
loading data: /Users/user1/Desktop/Flow_Data_Import_Test/DummyProject_Folder/DummyExp_Diva_Export/DummyExp_Diva_Exp_Export/DummyExp_DummySpecimen1_DummyTube1.fcs
swap cols: FSC-H:FSC-W
swap cols: SSC-H:SSC-W
Compensating
Error in solve.default(comp) : 'a' (7 x 14) must be square
mikejiang commented 3 years ago

how about set xml_compensation_enabled = F?

nvannest-absci commented 3 years ago

Success!!

Wow what a relief. Thanks so much for your help! 

Selection: 1
Parsing 2 samples
loading data: /Users/user1/Desktop/Flow_Data_Import_Test/DummyProject_Folder/DummyExp_Diva_Export/DummyExp_Diva_Exp_Export/DummyExp_DummySpecimen1_DummyTube1.fcs
swap cols: FSC-H:FSC-W
swap cols: SSC-H:SSC-W
Compensating
computing data range
transforming ...
parsing gates ...
loading data: /Users/user1/Desktop/Flow_Data_Import_Test/DummyProject_Folder/DummyExp_Diva_Export/DummyExp_Diva_Exp_Export/DummyExp_DummySpecimen1_DummyTube2.fcs
swap cols: FSC-H:FSC-W
swap cols: SSC-H:SSC-W
Compensating
computing data range
transforming ...
parsing gates ...
done!
> sampleNames(gs)
[1] "DummyExp_DummySpecimen1_DummyTube1.fcs"
[2] "DummyExp_DummySpecimen1_DummyTube2.fcs"
> gs_get_pop_paths(gs)
[1] "root"         "/P1"          "/P1/P2"       "/P1/P2/P3"    "/P1/P2/P3/P4"
[6] "/P1/P2/P3/P5" "/P1/P2/P3/P6" "/P1/P2/P3/P8"