search

kevinrue / velociraptor

Toolkit for Single-Cell Velocity
https://kevinrue.github.io/velociraptor/
Other
54 stars 10 forks source link

scvelo() "cannot open connection" #32

Open Laura19993 opened 3 years ago

Laura19993 commented 3 years ago

When I try to run the scvelo() function I get the following error message (see attached picture). I am pretty new to R and Python and would appreciate any help to solve that issue. grafik

LTLA commented 3 years ago

From the warnings, one of the calls has a 137 status, and this is ultimately causing the error. Some Googling suggests that this occurs when a Docker container runs out of memory, though I don't know enough about your system to know if this is applicable.

rrydbirk commented 3 years ago

I get the same error on a machine with 128 cores and 2 TB memory.

scvelo(dat) Registered S3 methods overwritten by 'dbplyr': method from print.tbl_lazy
print.tbl_sql
adding rname 'https://repo.anaconda.com/miniconda/Miniconda3-py37_4.8.3-Linux-x86_64.sh' |==========================================================================================================================================================| 100%

PREFIX=/home/rasmusr/.cache/basilisk/1.2.1/0 Unpacking payload ... Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done

Package Plan

environment location: /home/rasmusr/.cache/basilisk/1.2.1/0

added / updated specs:

The following NEW packages will be INSTALLED:

_libgcc_mutex pkgs/main/linux-64::_libgcc_mutex-0.1-main ca-certificates pkgs/main/linux-64::ca-certificates-2020.1.1-0 certifi pkgs/main/linux-64::certifi-2020.4.5.1-py37_0 cffi pkgs/main/linux-64::cffi-1.14.0-py37he30daa8_1 chardet pkgs/main/linux-64::chardet-3.0.4-py37_1003 conda pkgs/main/linux-64::conda-4.8.3-py37_0 conda-package-han~ pkgs/main/linux-64::conda-package-handling-1.6.1-py37h7b6447c_0 cryptography pkgs/main/linux-64::cryptography-2.9.2-py37h1ba5d50_0 idna pkgs/main/noarch::idna-2.9-py_1 ld_impl_linux-64 pkgs/main/linux-64::ld_impl_linux-64-2.33.1-h53a641e_7 libedit pkgs/main/linux-64::libedit-3.1.20181209-hc058e9b_0 libffi pkgs/main/linux-64::libffi-3.3-he6710b0_1 libgcc-ng pkgs/main/linux-64::libgcc-ng-9.1.0-hdf63c60_0 libstdcxx-ng pkgs/main/linux-64::libstdcxx-ng-9.1.0-hdf63c60_0 ncurses pkgs/main/linux-64::ncurses-6.2-he6710b0_1 openssl pkgs/main/linux-64::openssl-1.1.1g-h7b6447c_0 pip pkgs/main/linux-64::pip-20.0.2-py37_3 pycosat pkgs/main/linux-64::pycosat-0.6.3-py37h7b6447c_0 pycparser pkgs/main/noarch::pycparser-2.20-py_0 pyopenssl pkgs/main/linux-64::pyopenssl-19.1.0-py37_0 pysocks pkgs/main/linux-64::pysocks-1.7.1-py37_0 python pkgs/main/linux-64::python-3.7.7-hcff3b4d_5 readline pkgs/main/linux-64::readline-8.0-h7b6447c_0 requests pkgs/main/linux-64::requests-2.23.0-py37_0 ruamel_yaml pkgs/main/linux-64::ruamel_yaml-0.15.87-py37h7b6447c_0 setuptools pkgs/main/linux-64::setuptools-46.4.0-py37_0 six pkgs/main/linux-64::six-1.14.0-py37_0 sqlite pkgs/main/linux-64::sqlite-3.31.1-h62c20be_1 tk pkgs/main/linux-64::tk-8.6.8-hbc83047_0 tqdm pkgs/main/noarch::tqdm-4.46.0-py_0 urllib3 pkgs/main/linux-64::urllib3-1.25.8-py37_0 wheel pkgs/main/linux-64::wheel-0.34.2-py37_0 xz pkgs/main/linux-64::xz-5.2.5-h7b6447c_0 yaml pkgs/main/linux-64::yaml-0.1.7-had09818_2 zlib pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3

Preparing transaction: ...working... done Executing transaction: ...working... done installation finished. Killed running command '. '/d0/home/rasmusr/.cache/basilisk/1.2.1/0/etc/profile.d/conda.sh' && conda activate && /usr/lib/R/bin/Rscript --default-packages=NULL -e "con <- socketConnection(port=11552, open='wb', blocking=TRUE);serialize(Sys.getenv(), con);close(con)"' had status 137problem in accepting connections on this socketError in socketAccept(soc, blocking = TRUE, open = "a+b") : cannot open the connection

traceback() 10: socketAccept(soc, blocking = TRUE, open = "a+b") 9: .activate_condaenv(output, envpath) 8: .activateEnvironment() 7: setupBasiliskEnv(envpath, packages = .getPackages(env), channels = .getChannels(env), pip = .getPipPackages(env)) 6: .obtainEnvironmentPath(env) 5: basiliskStart(env, fork = fork, shared = shared) 4: basiliskRun(env = velo.env, fun = .run_scvelo, X = X, spliced = spliced, unspliced = unspliced, use.theirs = use.theirs, mode = mode, scvelo.params = scvelo.params, dimred = dimred) 3: .local(x, ...) 2: scvelo(dat) 1: scvelo(dat)

sessionInfo() R version 4.0.3 (2020-10-10) Platform: x86_64-pc-linux-gnu (64-bit) Running under: Ubuntu 18.04.4 LTS

Matrix products: default BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1 LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1

locale: [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C LC_ADDRESS=C LC_TELEPHONE=C LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages: [1] parallel stats4 stats graphics grDevices utils datasets methods base

other attached packages: [1] velociraptor_1.1.0 dutchmasters_0.1.0 ggbeeswarm_0.6.0 ggplot2_3.3.3 sccore_0.1.2
[6] cacoa_0.1 dplyr_1.0.3 magrittr_2.0.1 pagoda2_0.1.1 conos_1.3.2
[11] igraph_1.2.6 SingleCellExperiment_1.12.0 SummarizedExperiment_1.20.0 DelayedArray_0.16.1 MatrixGenerics_1.2.0
[16] Matrix_1.2-18 matrixStats_0.57.0 Biobase_2.50.0 GenomicRanges_1.42.0 GenomeInfoDb_1.26.2
[21] IRanges_2.24.1 S4Vectors_0.28.1 BiocGenerics_0.36.0

loaded via a namespace (and not attached): [1] rappdirs_0.3.1 scattermore_0.7 ggthemes_4.2.4 tidyr_1.1.2 bit64_4.0.5 knitr_1.30
[7] irlba_2.3.3 Rook_1.1-1 data.table_1.13.6 rpart_4.1-15 RCurl_1.98-1.2 generics_0.1.0
[13] cowplot_1.1.1 RSQLite_2.2.3 shadowtext_0.0.7 RANN_2.6.1 future_1.21.0 bit_4.0.4
[19] enrichplot_1.11.1 spatstat.data_1.7-0 httpuv_1.5.5 assertthat_0.2.1 viridis_0.5.1 xfun_0.20
[25] hms_1.0.0 promises_1.1.1 fansi_0.4.2 dbplyr_2.0.0 DBI_1.1.1 geneplotter_1.66.0
[31] htmlwidgets_1.5.3 purrr_0.3.4 ellipsis_0.3.1 annotate_1.66.0 deldir_0.2-9 sparseMatrixStats_1.2.0
[37] vctrs_0.3.6 Cairo_1.5-12.2 ROCR_1.0-11 abind_1.4-5 withr_2.4.0 ggforce_0.3.2
[43] grr_0.9.5 triebeard_0.3.0 sctransform_0.3.2 scran_1.16.0 zellkonverter_1.0.0 goftest_1.2-2
[49] cluster_2.1.0 DOSE_3.16.0 ape_5.4-1 lazyeval_0.2.2 crayon_1.3.4 basilisk.utils_1.2.1
[55] genefilter_1.70.0 edgeR_3.30.3 pkgconfig_2.0.3 tweenr_1.0.1 nlme_3.1-150 vipor_0.4.5
[61] rlang_0.4.10 globals_0.14.0 lifecycle_0.2.0 miniUI_0.1.1.1 downloader_0.4 filelock_1.0.2
[67] BiocFileCache_1.12.1 rsvd_1.0.3 polyclip_1.10-0 lmtest_0.9-38 urltools_1.7.3 zoo_1.8-8
[73] Matrix.utils_0.9.8 base64enc_0.1-3 beeswarm_0.2.3 ggridges_0.5.3 GlobalOptions_0.1.2 pheatmap_1.0.12
[79] png_0.1-7 viridisLite_0.3.0 rjson_0.2.20 bitops_1.0-6 KernSmooth_2.23-18 blob_1.2.1
[85] DelayedMatrixStats_1.12.2 shape_1.4.5 stringr_1.4.0 qvalue_2.22.0 brew_1.0-6 parallelly_1.23.0
[91] readr_1.4.0 beachmat_2.6.4 scales_1.1.1 memoise_1.1.0 plyr_1.8.6 ica_1.0-2
[97] zlibbioc_1.36.0 compiler_4.0.3 scatterpie_0.1.5 dqrng_0.2.1 tinytex_0.29 RColorBrewer_1.1-2
[103] clue_0.3-58 DESeq2_1.28.1 fitdistrplus_1.1-3 cli_2.2.0 XVector_0.30.0 listenv_0.8.0
[109] patchwork_1.1.1 pbapply_1.4-3 MASS_7.3-53 mgcv_1.8-33 tidyselect_1.1.0 stringi_1.5.3
[115] yaml_2.2.1 GOSemSim_2.16.1 BiocSingular_1.6.0 locfit_1.5-9.4 ggrepel_0.9.1 pbmcapply_1.5.0
[121] grid_4.0.3 fastmatch_1.1-0 tools_4.0.3 future.apply_1.7.0 circlize_0.4.12 rstudioapi_0.13
[127] gridExtra_2.3 farver_2.0.3 Rtsne_0.15 ggraph_2.0.4 digest_0.6.27 rvcheck_0.1.8
[133] BiocManager_1.30.10 shiny_1.6.0 Rcpp_1.0.6 scuttle_1.0.4 later_1.1.0.1 RcppAnnoy_0.0.18
[139] httr_1.4.2 AnnotationDbi_1.52.0 ComplexHeatmap_2.7.1 colorspace_2.0-0 XML_3.99-0.5 tensor_1.5
[145] reticulate_1.18 splines_4.0.3 uwot_0.1.10 statmod_1.4.35 spatstat.utils_2.0-0 scater_1.16.2
[151] graphlayouts_0.7.1 basilisk_1.2.1 plotly_4.9.3 xtable_1.8-4 jsonlite_1.7.2 spatstat_1.64-1
[157] tidygraph_1.2.0 R6_2.5.0 pillar_1.4.7 htmltools_0.5.1.1 mime_0.9 glue_1.4.2
[163] fastmap_1.1.0 clusterProfiler_3.16.1 BiocParallel_1.24.1 BiocNeighbors_1.6.0 codetools_0.2-18 fgsea_1.17.1
[169] lattice_0.20-41 tibble_3.0.5 curl_4.3 leiden_0.3.6 GO.db_3.12.1 survival_3.2-7
[175] limma_3.44.3 munsell_0.5.0 DO.db_2.9 GetoptLong_1.0.5 GenomeInfoDbData_1.2.4 reshape2_1.4.4
[181] gtable_0.3.0 Seurat_3.2.3

LTLA commented 3 years ago

Hm. I don't know why this happens. Suggest trying to run the minimal example in a fresh R session.

# Copy-pasted from the error message above. Note the escape of the double quotes.
act.cmd <- ". '/d0/home/rasmusr/.cache/basilisk/1.2.1/0/etc/profile.d/conda.sh' && conda activate && /usr/lib/R/bin/Rscript --default-packages=NULL -e \"con <- socketConnection(port=11552, open='wb', blocking=TRUE);serialize(Sys.getenv(), con);close(con)\""

# Taken from the port above.
p <- 11552 

soc <- serverSocket(p)
system(act.cmd, intern = TRUE)
listener <- socketAccept(soc, blocking = TRUE, open = "a+b")
activated <- unserialize(listener)
close(soc)

Turning off intern=TRUE might also provide some more information w.r.t. an error message.

Laura19993 commented 3 years ago

Thanks for your suggestions. I ran it in a fresh R session using the paths from my error message. And I still get the same error "status 137".

#copy-pasted from my error message. Note the escape of the double quotes.
act.cmd <- ". '/d0/home/lwolbeck/.cache/basilisk/1.2.1/0/etc/profile.d/conda.sh' && conda activate '/d0/home/lwolbeck/.cache/basilisk/1.2.1/velociraptor-1.0.0/env' && /usr/local/R/R-4.0.3/lib/R/bin/Rscript --default-packages=NULL -e \"con <- socketConnection(port=11656, open='wb', blocking=TRUE);serialize(Sys.getenv(), con);close(con)\""
#Taken from the port above
p <- 11656  
soc <- serverSocket(p)
system(act.cmd, intern = TRUE)
Killed
character(0)
attr(,"status")
[1] 137
Warning message:
In system(act.cmd, intern = TRUE) :
running command '. '/d0/home/lwolbeck/.cache/basilisk/1.2.1/0/etc/profile.d/conda.sh' && conda activate '/d0/home/lwolbeck/.cache/basilisk/1.2.1/velociraptor-1.0.0/env' && /usr/local/R/R-4.0.3/lib/R/bin/Rscript --default-packages=NULL -e "con <- socketConnection(port=11656, open='wb', blocking=TRUE);serialize(Sys.getenv(), con);close(con)"' had status 137

listener <- socketAccept(soc, blocking = TRUE, open = "a+b")
Error in socketAccept(soc, blocking = TRUE, open = "a+b") : 
cannot open the connection
In addition: Warning message:
In socketAccept(soc, blocking = TRUE, open = "a+b") :
  problem in accepting connections on this socket

activated <- unserialize(listener)
Error in unserialize(listener) : object 'listener' not found
close(soc)
LTLA commented 3 years ago

Do you get more informative error messages if you remove intern=TRUE in the system() call?

Edit: After some more Googling, it turns out that 137 is a combination of the status codes 128 (fatal signal) and 9 (sigkill), see docs here. This is consistent with an examination of your error logs - see the Killed message above - which indicates that some external process is killing the system call. This is usually caused by resource controllers on clusters, cloud, etc. - an event like this would be fairly typical on the cluster at my workplace where the memory limits requested by each job are strictly enforced.

Edit 2: I would be curious to see what happens for the simpler:

act.cmd <- ". '/d0/home/rasmusr/.cache/basilisk/1.2.1/0/etc/profile.d/conda.sh' && conda activate"
system(act.cmd, intern = TRUE)

Or even just:

act.cmd <- ". '/d0/home/rasmusr/.cache/basilisk/1.2.1/0/etc/profile.d/conda.sh'"
system(act.cmd, intern = TRUE)
Laura19993 commented 3 years ago

If I remove intern=TRUE I receive a shorter error message, it only says Killed then

For the two other calls you suggested I get character(0) as a respond

LTLA commented 3 years ago

I have no idea what's going on here. My best guess is that your system is configured to kill any process that tries to connect to a port. While I could add a workaround... I don't want to, unless I get some more information about what's going wrong. My workaround won't solve the underlying port issue, which seems like it would break any socket-based parallelization via parallel.

kevinrue commented 3 years ago

@shijianasdf Please open a new GitHub issue for your problem. There is no guarantee - and it is rather unlikely - that your problem is related to this one.

As a general rule, if your situation is not obviously the same as an existing one, it is always better to create a new issue, so that users and developers can track and communicate on each issue separately.

shijianasdf commented 3 years ago

@shijianasdf Please open a new GitHub issue for your problem. There is no guarantee - and it is rather unlikely - that your problem is related to this one.

As a general rule, if your situation is not obviously the same as an existing one, it is always better to create a new issue, so that users and developers can track and communicate on each issue separately.

Thank you for your advice