Open baj12 opened 3 years ago
Can you share the minimum reproducible example?
Can you try the file I sent you around the 18th?
gs = load_gs(sampleGS)
> gatingset_to_flowjo(gs, "test")
Using docker image rglab/gs-to-flowjo:devel to write FlowJo workspace...
Error in gatingset_to_flowjo(gs, "test") :
terminate called after throwing an instance of 'std::runtime_error' what(): Can't find any non-boolean children node under CD4_P CD127_P
In addition: Warning message:
In gatingset_to_flowjo(gs, "test") :
docker image 'rglab/gs-to-flowjo:devel' is built with different cytolib version of from R package: 2.3.0 vs 2.3.7
> sessionInfo()
R version 4.0.3 (2020-10-10)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Mojave 10.14.6
Matrix products: default
BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRlapack.dylib
locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] flowWorkspace_4.3.7 CytoML_2.3.3
loaded via a namespace (and not attached):
[1] Biobase_2.50.0 httr_1.4.2 tidyr_1.1.2.9000 ConsensusClusterPlus_1.54.0 jsonlite_1.7.2 RcppParallel_5.0.3
[7] assertthat_0.2.1 stats4_4.0.3 latticeExtra_0.6-29 RBGL_1.66.0 yaml_2.2.1 pillar_1.5.1
[13] lattice_0.20-41 glue_1.4.2 digest_0.6.27 RColorBrewer_1.1-2 colorspace_2.0-0 ggcyto_1.18.0
[19] htmltools_0.5.1.1 plyr_1.8.6 XML_3.99-0.5 pkgconfig_2.0.3 tsne_0.1-3 calibrate_1.7.7
[25] zlibbioc_1.36.0 purrr_0.3.4 flowCore_2.3.2 scales_1.1.1 jpeg_0.1-8.1 pracma_2.3.3
[31] tibble_3.1.0 aws.s3_0.3.21 generics_0.1.0 ggplot2_3.3.3 ellipsis_0.3.1 DT_0.17
[37] BiocGenerics_0.36.0 hexbin_1.28.2 magrittr_2.0.1 crayon_1.4.1 evaluate_0.14 fansi_0.4.2
[43] MASS_7.3-53 xml2_1.3.2 graph_1.68.0 tools_4.0.3 data.table_1.14.0 ncdfFlow_2.36.0
[49] lifecycle_1.0.0 matrixStats_0.58.0 S4Vectors_0.28.1 munsell_0.5.0 cluster_2.1.0 compiler_4.0.3
[55] rlang_0.4.10 debugme_1.1.0 grid_4.0.3 htmlwidgets_1.5.3 aws.signature_0.6.0 igraph_1.2.6
[61] base64enc_0.1-3 rmarkdown_2.6 cytolib_2.3.7 gtable_0.3.0 DBI_1.1.1 curl_4.3
[67] R6_2.5.0 RProtoBufLib_2.3.4 gridExtra_2.3 knitr_1.31 dplyr_1.0.5 utf8_1.1.4
[73] FlowSOM_1.22.0 Rgraphviz_2.34.0 stringi_1.5.3 parallel_4.0.3 Rcpp_1.0.6 vctrs_0.3.6
[79] png_0.1-7 tidyselect_1.1.0 xfun_0.20
>
I think I know where the problem is. Let me tackle this and get back to you once we have a solution.
I've pushed the fix. You will need to pull the latest CytoML from github master and pull the new docker image
docker pull public.ecr.aws/x4k5d9i7/cytoverse/gs-to-wsp:latest
I have pulled the image but cannot run it:
> system2("docker", paste0("run ", docker.image, " --cytolib-version"), stdout = TRUE)
character(0)
attr(,"status")
[1] 132
attr(,"errmsg")
[1] "Resource temporarily unavailable"
Warning message:
In system2("docker", paste0("run ", docker.image, " --cytolib-version"), :
running command ''docker' run public.ecr.aws/x4k5d9i7/cytoverse/gs-to-wsp:latest --cytolib-version' had status 132
I am not a docker expert and don't know what to do with status 132 or "Resource temporarily unavailable"
Originally I ran into this problem:
> gatingset_to_flowjo(fp, "test.wsp")
Using docker image public.ecr.aws/x4k5d9i7/cytoverse/gs-to-wsp:latest to write FlowJo workspace...
Error: '/var/folders/d_/g1jsd9v96mvc9n0ds29v7m8m0000gn/T//RtmpOtPBY0/filee6481c5c4ea0' does not exist.
In addition: Warning message:
In system2("docker", paste0("run ", docker_img, " --cytolib-version"), :
running command ''docker' run public.ecr.aws/x4k5d9i7/cytoverse/gs-to-wsp:latest --cytolib-version' had status 132
I am not sure why the temp file doesn't exist as well.
Thx for looking into this.
I don't know how to reproduce your error, here is what you should get when you run the command at termainal
CytoML$ docker run public.ecr.aws/x4k5d9i7/cytoverse/gs-to-wsp:latest --cytolib-version
2.3.2
Hi Mike, I ran quite a few tests on quite a few machines. Bottom line, on some computers the docker file works on others not. Most importantly, it doesn't work on any of the machines I have access to. Within the docker file I tracked down the problem to the following:
#0 0x0000563c3876ac45 in google::protobuf::internal::ReadSizeFallback(char const*, unsigned int) ()
(gdb) up
#1 0x0000563c385d7b55 in google::protobuf::internal::ReadSize(char const**) ()
(gdb) up
#2 0x0000563c386ccd7a in char const* google::protobuf::internal::ParseContext::ParseMessage<google::protobuf::DescriptorProto>(google::protobuf::DescriptorProto*, char const*) ()
(gdb) up
#3 0x0000563c3869abb4 in google::protobuf::FileDescriptorProto::_InternalParse(char const*, google::protobuf::internal::ParseContext*) ()
(gdb) up
#4 0x0000563c38771e9b in bool google::protobuf::internal::MergeFromImpl<false>(google::protobuf::StringPiece, google::protobuf::MessageLite*, google::protobuf::MessageLite::ParseFlags)
()
(gdb) up
#5 0x0000563c38772c88 in bool google::protobuf::MessageLite::ParseFrom<(google::protobuf::MessageLite::ParseFlags)1, google::protobuf::StringPiece>(google::protobuf::StringPiece const&) ()
(gdb) up
#6 0x0000563c3877067d in google::protobuf::MessageLite::ParseFromArray(void const*, int) ()
(gdb) up
#7 0x0000563c386d7733 in google::protobuf::EncodedDescriptorDatabase::Add(void const*, int) ()
(gdb) up
#8 0x0000563c3863a065 in google::protobuf::DescriptorPool::InternalAddGeneratedFile(void const*, int) ()
(gdb) up
#9 0x0000563c385f79e4 in google::protobuf::(anonymous namespace)::AddDescriptorsImpl(google::protobuf::internal::DescriptorTable const*) ()
(gdb) up
#10 0x0000563c385f7a71 in google::protobuf::internal::AddDescriptors(google::protobuf::internal::DescriptorTable const*) ()
(gdb) up
#11 0x0000563c385d672e in __static_initialization_and_destruction_0(int, int) ()
(gdb) up
#12 0x0000563c385d674f in _GLOBAL__sub_I_GatingSet.pb.cc ()
(gdb) up
#13 0x0000563c3969844d in __libc_csu_init ()
(gdb) up
#14 0x00007fa0c9d7c040 in __libc_start_main (main=0x563c38402a81 <main>, argc=2, argv=0x7ffdf7ec71c8, init=0x563c39698400 <__libc_csu_init>, fini=<optimized out>,
rtld_fini=<optimized out>, stack_end=0x7ffdf7ec71b8) at ../csu/libc-start.c:264
264 ../csu/libc-start.c: No such file or directory.
(gdb) up
#15 0x0000563c384025ee in _start ()
But, again, this doesn't mean much to me. I have updated and upgraded everything but the OS. (I am still working on Mojave, 10.14.6 )
Before upgrading the OS, which I really like to push to some time with less stress, I would like to ask you for the script to create the docker file. Maybe I can get this working.
Thanks for you consideration.
docker is os and machine independent , shouldn't behave differently. Here is a reproducible example see if it works for you
library(flowWorkspace)
library(CytoML)
dataDir <- system.file("extdata",package="flowWorkspaceData")
gs_dir <- list.files(dataDir, pattern = "gs_manual",full = TRUE)
gs <- load_gs(gs_dir)
outFile <- tempfile(fileext = ".wsp")
gatingset_to_flowjo(gs, outFile)
library(flowWorkspace)
> library(CytoML)
> dataDir <- system.file("extdata",package="flowWorkspaceData")
> gs_dir <- list.files(dataDir, pattern = "gs_manual",full = TRUE)
> gs <- load_gs(gs_dir)
> outFile <- tempfile(fileext = ".wsp")
> gatingset_to_flowjo(gs, outFile)
Using docker image public.ecr.aws/x4k5d9i7/cytoverse/gs-to-wsp:latest to write FlowJo workspace...
Error: '/var/folders/d_/g1jsd9v96mvc9n0ds29v7m8m0000gn/T//RtmpFSgzEg/file268f7640fcc' does not exist.
In addition: Warning message:
In system2("docker", paste0("run ", docker_img, " --cytolib-version"), :
running command ''docker' run public.ecr.aws/x4k5d9i7/cytoverse/gs-to-wsp:latest --cytolib-version' had status 132
Unfortunately, there are some specific issues with MAC as it is running a different VM. https://www.reddit.com/r/docker/comments/bltx02/container_exits_with_132_with_no_logs/ https://bugzilla.redhat.com/show_bug.cgi?id=1795574
These sites are not 100% matching, but they show some aspects of the issue. I can access the executable within the docker file, but it would be helpful to see the docker file to create the image such that I can verify the machine and then the executable. I guess there are certain libraries that are responsible for the interaction between the VM, mac OS and Linux which are not behaving.
Btw. I was able to verify that the mounting of files works, i.e. I can access the the files from my Mac from within the docker image. (at /gs)
here is the dockerfile
#set up running environment for gs-to-wsp
FROM ubuntu:20.04
#prepare build env
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get update
RUN apt-get install -y libblas-dev liblapack-dev libgomp1
COPY gs-to-wsp /
# Pre-create mount points for I/O
RUN mkdir /gs /out
ENTRYPOINT ["/gs-to-wsp"]
CMD ['--help']
I am not sure how much help you can get from it, since the gs-to-wsp
executable is built and copied from the private host machine. It shouldn't really matter since I expect the docker image has all the environment it needs to run the executable.
One thing you can do is get inside of container and cp your gs to it, and run the gs-to-wsp
in the container to see if it works
docker run -it --entrypoint=/bin/bash public.ecr.aws/x4k5d9i7/cytoverse/gs-to-ws
root@50202e6faf2c:/# ./gs-to-wsp --version
here is the dockerfile
#set up running environment for gs-to-wsp
FROM ubuntu:20.04
#prepare build env
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get update
RUN apt-get install -y libblas-dev liblapack-dev libgomp1
COPY gs-to-wsp /
# Pre-create mount points for I/O
RUN mkdir /gs /out
ENTRYPOINT ["/gs-to-wsp"]
CMD ['--help']
I am not sure how much help you can get from it, since the gs-to-wsp
executable is built and copied from the private host machine. It shouldn't really matter since I expect the docker image has all the environment it needs to run the executable.
One thing you can do is get inside of container and cp your gs to it, and run the gs-to-wsp
in the container to see if it works
docker run -it --entrypoint=/bin/bash public.ecr.aws/x4k5d9i7/cytoverse/gs-to-ws
root@50202e6faf2c:/# ./gs-to-wsp --version
Sorry, still not working.
As a last resort, I even updated the OS to Big Sur.
I installed gdb within the VM and traced back the illegal instruction to:
0x563c325d4c45 <_ZN6google8protobuf8internal16ReadSizeFallbackEPKcj+67> shlx %eax,%ecx,%edx
for the shlx command I found out that it might be that my computer is too old (2013, Ivy Bridge EP, Xeon E5) is not supporting AVX2.
(I also tried copying the executable into a new VM, without luck.)
Would it be possible to compile it on my side within the VM or even directly on MacOS?
Does it mean that docker in general isn't working in your mac vm? Can you enter bash mode by docker run -it --entrypoint=/bin/bash public.ecr.aws/x4k5d9i7/cytoverse/gs-to-ws
at all?
docker is working. I can run R within Docker. Version 2.0 is running normally:
docker run rglab/gs-to-flowjo:2.0 --help
usage: gs-to-flowjo [--version] [--cytolib-version] [--help] <src> <dest> [--showHidden=no]
src GatingSet archive directory
dest output flowjo wsp file path
showHidden whether to export the hidden populations
docker run rglab/gs-to-flowjo:2.0 --cytolib-version
2.0.2
Other docker images are running as well. I would assume that you compiled the executable within the image on a machine with avx2 support. I don't know which switch would disable this during compilation. But since I checked this on several machines and only the older ones are broken I would conclude that the CPUs are not ready for this. Windows and Mac by the way.
Mike, I moved for this task to a newer CPU (from 2017) and docker works, but I still get the following error:
[1] /Library/Frameworks/R.framework/Versions/4.0/Resources/library
> gatingset_to_flowjo(fp, "test.wsp")
Using docker image public.ecr.aws/x4k5d9i7/cytoverse/gs-to-wsp:latest to write FlowJo workspace...
Error in gatingset_to_flowjo(fp, "test.wsp") :
terminate called after throwing an instance of 'std::domain_error' what(): Treg_Memory- not found!
the docker image produces the correct output:
docker run public.ecr.aws/x4k5d9i7/cytoverse/gs-to-wsp:latest --cytolib-version
2.3.2
Do you have any idea what could be the cause of the problem? Since it is working with the sample data I might be able to change my data such that it works. Thx
Hi Mike,
not sure if you understood, that the VM is working and I found a more recent computer that runs the program. But, I still get the same error message. Can you reproduce this problem on your side?
thanks,
Bernd
I've updated the docker image, with latest pull, your previous error should be gone. But this time, you should see
> library(CytoML)
> gs = load_gs("~/Downloads/sampleGS/")
> tmp = tempfile()
> gatingset_to_flowjo(gs, tmp, docker_img = 'public.ecr.aws/x4k5d9i7/cytoverse/gs-to-wsp')
Using docker image public.ecr.aws/x4k5d9i7/cytoverse/gs-to-wsp to write FlowJo workspace...
Error in gatingset_to_flowjo(gs, tmp, docker_img = "public.ecr.aws/x4k5d9i7/cytoverse/gs-to-wsp") :
F0331 22:35:12.266316 1 flowjo_xml_node.cpp:507] grp.0.1: logical gate is not supported***
which tells you that your gs has the logical gate from grp.0.1
that is not supported by flowJo. Can you tell me how it is generated? Is it from flowjo or added by you with our tooling?
Just as a side-note: yesterday I received a new computer, a MacBook pro. Guess what happens when trying to install Docker:
Incompatible CPU detected
We are sorry, but your hardware is incompatible with Docker Desktop.
This version of Docker Desktop requires an Intel processor.
The chip is now called: "Apple M1"
There is only a RC (version 2) with quite a few unresolved issues.
This whole thing is turning into a nightmare.
I will try with a windows based computer from my son
I hope we can get closer with this. It is not a "minimal" example but has all the elements I am using.
devtools::install_github("RGLab/CytoML")
install.packages("devtools")
library(devtools) #load it
devtools::install_github("RGLab/RProtoBufLib")
devtools::install_github("RGLab/cytolib")
install_github("RGLab/flowWorkspace")
devtools::install_github("RGLab/flowCore")
devtools::install_github("RGLab/flowWorkspaceData")
BiocManager::install("openCyto")
devtools::install_github("RGLab/openCyto")
devtools::install_github("RGLab/CytoML")
library(openCyto)
library(flowCore)
library(flowWorkspace)
library(CytoML)
fcsfiles <- list.files(pattern = "CytoTrol",
system.file("extdata",
package = "flowWorkspaceData"),
full = TRUE)
(fs <- read.flowSet(fcsfiles))
gs <- GatingSet(fs)
gs
saveName = tempfile()
save_cytoset(gs, saveName)
cs = load_cytoset(saveName)
cs_unlock(cs)
for (sampleIdx in 1:length(gs)) {
col2add = rep(0, gh_pop_get_count(gs[[sampleIdx]],"root"))
#modify some col2add elements
mm = matrix(ncol = 1, col2add, dimnames = list(NULL, "fsom.0.1"))
mm[,1] = sample(1:10, replace = T, size = nrow(mm))
cf_append_cols(cs[[sampleIdx]], mm)
}
gs_cyto_data(gs) <- cs
gsFile = "testGS"
save_gs(gs, path=gsFile, backend_opt = "copy")
gs_cleanup_temp(gs)
gs = load_gs(path=gsFile)
save_cytoset(gs, saveName)
cs = load_cytoset(saveName)
cs_unlock(cs)
metaGate <- function(fr, pp_res, channels = NA, metaNr, ...) {
exprs(fr)[,"fsom.0.1"] %in% metaNr
}
register_plugins(fun=metaGate, methodName="metaGate", type="gating")
parentGate = "root"
for (idx in 1:10) {
gs_add_gating_method(gs, alias = paste0("grp.0.",idx), parent = parentGate,
gating_args = list(metaNr=c(idx)),
dims = "fsom.0.1", gating_method = "metaGate")
}
recompute(gs)
save_gs(gs = gs, path = saveName, backend_opt = "copy")
cs_cleanup_temp(cs)
gatingset_to_flowjo(gs, outFile = "outtest.jo")
with the following error message:
> gatingset_to_flowjo(gs, outFile = "outtest.jo")
Informationÿ: impossible de trouver des fichiers pour le(s) modŠle(s) sp‚cifi‚(s).
Using docker image public.ecr.aws/x4k5d9i7/cytoverse/gs-to-wsp:latest to write FlowJo workspace...
Error in gatingset_to_flowjo(gs, outFile = "outtest.jo") :
terminate called after throwing an instance of 'std::domain_error' what(): undefined getParam function!
First of all, if you pull the latest docker image , you should see
gatingset_to_flowjo(gs, outFile = "outtest.jo")
Using docker image public.ecr.aws/x4k5d9i7/cytoverse/gs-to-wsp:latest to write FlowJo workspace...
Error in gatingset_to_flowjo(gs, outFile = "outtest.jo") :
F0406 19:29:28.604566 1 flowjo_xml_node.cpp:507] grp.0.1: logical gate is not supported*
Secondly, you were creating the logical
gate (which is simply a boolean vector) from opencyto, which is fine in itself within opencyto context. But flowJo doesn't have the equivalent gate type to represent these independent logical gate.
I think what you may want to do is to add these bool vector as a single factor , which will be treated the clustering results. And it can be handled by flowjo properly
g <- lapply(cs, function(fr) {
as.factor(exprs(fr)[,"fsom.0.1"])# this will be split into multiple logical gates under the hood, but they still belong to the same clustering method
})
gs_pop_add(gs, g, name = "cluster", parent = "root")
gs_pop_get_children(gs, "root")
[1] "/cluster_1" "/cluster_2" "/cluster_3" "/cluster_4" "/cluster_5" "/cluster_6" "/cluster_7" "/cluster_8" "/cluster_9"
[10] "/cluster_10"
recompute(gs)
gatingset_to_flowjo(gs, outFile = "outtest.wsp")
Using docker image public.ecr.aws/x4k5d9i7/cytoverse/gs-to-wsp:latest to write FlowJo workspace
I am getting the following error on M1 Pro chip Macbook Pro. I installed docker, got the latest dev version of flowCore, CytoML. But I am getting this weird msg on specifying image platform, for which i dont see any options to specify it. Please advise.
cyto_export(gsf, "ba48_1e5_flowjo_au.wsp" ) Saving GatingSet to flowJo workspace file... WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested Using docker image public.ecr.aws/x4k5d9i7/cytoverse/gs-to-wsp:latest to write FlowJo workspace... Error in CytoML::gatingset_to_flowjo(x, save_as, ...) : WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
I was able to fix the issue by adding --platform linux/amd64
to the system2
call in the original gatingset_to_flowjo
function. The docker image was built for linux/amd64
and Apple Silicone uses linux/aarch64
.
❯ docker inspect -f '{{.Os}}/{{.Architecture}}' public.ecr.aws/ozette/cytoverse/gs-to-wsp
linux/amd64
❯ docker info -f '{{.OSType}}/{{.Architecture}}'
linux/aarch64
As a temporary fix, copy and source the following
gatingset_to_flowjo <- function (gs, outFile, showHidden = FALSE, docker_img = NULL, ...) {
res <- CytoML:::check_binary_status()
if (res != "binary_ok") {
res <- CytoML:::check_docker_status(docker_img)
}
if (!(res[1] %in% c("binary_ok", "docker_ok")))
stop(res)
if (is(gs, "GatingSet")) {
tmp <- tempfile()
suppressMessages(save_gs(gs, tmp, ...))
}
else tmp <- gs
outFile <- gsub(" ", "\\\\ ", file.path(normalizePath(dirname(outFile)),
basename(outFile)))
tmpfile <- tempfile()
if (res[1] == "binary_ok") {
message("Using local gs-to-flowjo binary to write FlowJo workspace...")
res <- suppressWarnings(system2("gs-to-flowjo", paste0(" --src=",
tmp, " --dest=", tmpfile, " --showHidden=", showHidden),
stderr = TRUE))
}
else {
docker_img <- res[2]
v1 <- packageVersion("cytolib")
v2 <- system2("docker", paste0("run --platform linux/amd64", docker_img, " --cytolib-version"),
stdout = TRUE)
message(paste0("Using docker image ", docker_img, " to write FlowJo workspace..."))
res <- suppressWarnings(system2("docker", paste0("run --platform linux/amd64",
" -v ", tmp, ":/gs", " -v ", normalizePath(dirname(tmpfile)),
":/out ", docker_img, " --src=/gs --dest=/out/",
basename(tmpfile), " --showHidden=", showHidden),
stderr = TRUE))
}
if (length(res) > 0)
stop(res)
else {
tree <- xml2::read_xml(tmpfile)
CytoML:::add_version_info(tree)
invisible(xml2::write_xml(tree, file = outFile))
}
}
This will allow Docker to use emulation. If it still fails, make sure to install Rosetta 2.
Long term, I think it should be rebuilt for multiple-architectures. @mikejiang
I want to export a gatingset to FlowJo. (the data origniates from a flowJo workspace, is a bit modified (added a gate and two markers)
I get the error message below, complaining about a population ("Treg_Memory-") that is not found.
I cannot find this population myself either:
I am also wondering if this has to do with the warning about 2.3.0 vs. 2.3.7.
Not sure what to test next. Any help appreciated.
Thanks,
Bernd
PS. not sure if these are related, but with the same data set, only a bit more modified, I get the following error message: