Error in socketAccept for writeH5AD

[Richard0146]

Hi, when I running writeH5AD for storing sce object in ubuntu system, it reported socketAccept error

> sce <- read10xCounts(file.path("./C9_R/filtered_feature_bc_matrix"))
> writeH5AD(sce,file='sce.h5ad')
sh: 4: /home/labw/anaconda3/envs/py38/etc/conda/deactivate.d/deactivate-r-base.sh: [[: not found
sh: 5: /home/labw/anaconda3/envs/py38/etc/conda/deactivate.d/deactivate-gxx_linux-64.sh: Syntax error: "(" unexpected
Error in socketAccept(soc, blocking = TRUE, open = "a+b") : 
  cannot open the connection
In addition: Warning messages:
1: In system(paste(act.cmd, collapse = " "), intern = TRUE) :
  running command '. '/home/labw/.cache/basilisk/1.2.1/0/etc/profile.d/conda.sh' && conda activate && /home/labw/anaconda3/envs/py38/lib/R/bin/Rscript --default-packages=NULL -e "con <- socketConnection(port=11566, open='wb', blocking=TRUE);serialize(Sys.getenv(), con);close(con)"' had status 2
2: In socketAccept(soc, blocking = TRUE, open = "a+b") :
  problem in accepting connections on this socket

for SessionInfo:

> sessionInfo()
R version 4.0.3 (2020-10-10)
Platform: x86_64-conda-linux-gnu (64-bit)
Running under: Ubuntu 18.04.5 LTS

Matrix products: default
BLAS/LAPACK: /home/labw/anaconda3/envs/py38/lib/libopenblasp-r0.3.12.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=zh_CN.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=zh_CN.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=zh_CN.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=zh_CN.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] parallel  stats4    stats     graphics  grDevices utils     datasets 
[8] methods   base     

other attached packages:
 [1] DropletUtils_1.10.3         SingleCellExperiment_1.12.0
 [3] SummarizedExperiment_1.20.0 Biobase_2.50.0             
 [5] GenomicRanges_1.42.0        GenomeInfoDb_1.26.4        
 [7] IRanges_2.24.1              S4Vectors_0.28.1           
 [9] BiocGenerics_0.36.0         MatrixGenerics_1.2.1       
[11] matrixStats_0.58.0          zellkonverter_1.0.3        

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.6                compiler_4.0.3           
 [3] XVector_0.30.0            DelayedMatrixStats_1.12.3
 [5] R.methodsS3_1.8.1         basilisk.utils_1.2.2     
 [7] bitops_1.0-6              rhdf5filters_1.2.0       
 [9] R.utils_2.10.1            tools_4.0.3              
[11] zlibbioc_1.36.0           jsonlite_1.7.2           
[13] rhdf5_2.34.0              lattice_0.20-41          
[15] Matrix_1.3-2              DelayedArray_0.16.2      
[17] filelock_1.0.2            GenomeInfoDbData_1.2.4   
[19] rappdirs_0.3.3            locfit_1.5-9.4           
[21] grid_4.0.3                scuttle_1.0.4            
[23] reticulate_1.18           HDF5Array_1.18.1         
[25] basilisk_1.2.1            BiocParallel_1.24.1      
[27] limma_3.46.0              Rhdf5lib_1.12.1          
[29] edgeR_3.32.1              sparseMatrixStats_1.2.1  
[31] beachmat_2.6.4            dqrng_0.2.1              
[33] RCurl_1.98-1.3            R.oo_1.24.0 

I run the R under conda (Python 3.8.8) environment. For my windows system, same sce object can be stored by writeH5AD successfully without any errors. Can you help me to see what the problem is? Is it a linux system problem or a conda environment problem? Thank you.

[LTLA]

zellkonverter itself uses its own Conda environment to manage the various Python packages. So I'm guessing that your Conda is fighting with zellkonverter's Conda, leading to the conda-related warnings (and ultimately the error).

You could try telling it to use your Conda instead of installing its own. This is done by setting export BASILISK_EXTERNAL_CONDA=/home/labw/anaconda3 somewhere, usually in your ~/.Renviron, reinstalling basilisk and zellkonverter, and hoping for the best. This functionality is not particularly well-tested so YMMV.

[Richard0146]

Unfortunately, I still get the same error after I adjust BASILISK_EXTERNAL_CONDA=/home/labw/anaconda3. This error seems to have two parts. firstly,

sh: 4: /home/labw/anaconda3/envs/py38/etc/conda/deactivate.d/deactivate-r-base.sh: [[: not found
sh: 5: /home/labw/anaconda3/envs/py38/etc/conda/deactivate.d/deactivate-gxx_linux-64.sh: Syntax error: "(" unexpected

this seems like an error in executing the conda-related commands in the server environment, but the fatal error is like the following socketAccept error

Error in socketAccept(soc, blocking = TRUE, open = "a+b") : 
  cannot open the connection

I updated Basilisk with reference to the solution in Issue #36 but still haven't solved it. Warnings and Errors are very similar to issue #36

>Sys.getenv()
BASILISK_EXTERNAL_CONDA                                                
                        /home/labw/anaconda3

>library(basilisk.utils)
>getCondaDir(installed = TRUE)
[1] "/home/labw/anaconda3"

[LTLA]

The first failure causes the second, so that's not surprising.

I forgot to mention that, as part of the reinstallation, you should also delete the directory at basilisk.utils::getExternalDir(). That contains the environment generated by basilisk's conda; you want to make sure that the environment is instead generated by your system conda. Whether this makes a difference or not is anyone's guess, but one would hope that two environments generated by the same conda instance are at least compatible with each other.

If that doesn't work... ¯\_(ツ)_/¯. The first failing command is run on the command line, and looks like:

. '/home/luna/.cache/basilisk/1.3.7/0/etc/profile.d/conda.sh' && conda activate && /home/luna/Software/R/trunk/bin/Rscript --no-save --no-restore --no-site-file --no-init-file --default-packages=NULL -e "con <- socketConnection(port=11886, open='wb', blocking=TRUE);serialize(Sys.getenv(), con);close(con)"

You can find the equivalent command for your systemby running debug(basilisk.utils:::.activate_condaenv), calling any basilisk-powered function like writeH5AD(), stepping through the function line by line, and then printing out the value of act.cmd when you get to the line where it is defined. Once you get that command, it would be instructive to see what happens when you try to run it outside of R, just to make sure that it's not an error with the R set-up itself.

[lazappi]

@Richard0146 Were you able to make any progress with this?

[Richard0146]

I got results like following:

debug: system(paste(act.cmd, collapse = " "), intern = TRUE)
Browse[3]> 
sh: 4: /home/labw/anaconda3/envs/py38/etc/conda/deactivate.d/deactivate-r-base.sh: [[: not found
sh: 5: /home/labw/anaconda3/envs/py38/etc/conda/deactivate.d/deactivate-gxx_linux-64.sh: Syntax error: "(" unexpected
debug: listener <- socketAccept(soc, blocking = TRUE, open = "a+b")
Warning message:
In system(paste(act.cmd, collapse = " "), intern = TRUE) :
  running command '. '/home/labw/anaconda3/etc/profile.d/conda.sh' && conda activate && /home/labw/anaconda3/envs/py38/lib/R/bin/Rscript --default-packages=NULL -e "con <- socketConnection(port=11640, open='wb', blocking=TRUE);serialize(Sys.getenv(), con);close(con)"' had status 2

Browse[3]> print(act.cmd)
 [1] "."                                                                                                        
 [2] "'/home/labw/anaconda3/etc/profile.d/conda.sh'"                                                            
 [3] "&&"                                                                                                       
 [4] "conda"                                                                                                    
 [5] "activate"                                                                                                 
 [6] "&&"                                                                                                       
 [7] "/home/labw/anaconda3/envs/py38/lib/R/bin/Rscript"                                                         
 [8] "--default-packages=NULL"                                                                                  
 [9] "-e"                                                                                                       
[10] "\"con <- socketConnection(port=11640, open='wb', blocking=TRUE);serialize(Sys.getenv(), con);close(con)\""

I have no idea what need to do next.

[Richard0146]

When I run the code outside, it shows

(py38) labw@labw-GPU:~$ . '/home/labw/anaconda3/etc/profile.d/conda.sh'
(py38) labw@labw-GPU:~$ conda activate
(base) labw@labw-GPU:~$ /home/labw/anaconda3/envs/py38/lib/R/bin/Rscript --default-packages=NULL -e "con <- socketConnection(port=11566, open='wb', blocking=TRUE);serialize(Sys.getenv(), con);close(con)"
Error in socketConnection(port = 11566, open = "wb", blocking = TRUE) : 
  cannot open the connection
In addition: Warning message:
In socketConnection(port = 11566, open = "wb", blocking = TRUE) :
  localhost:11566 cannot be opened
Execution halted

[Richard0146]

I tried to change the original code in basilisk to remove conda activate, and the top two lines that sh4 and sh5 was disappeared, however there is still socket problem:

Browse[2]> 
debug: listener <- socketAccept(soc, blocking = TRUE, open = "a+b")
Browse[2]> 
Error in socketAccept(soc, blocking = TRUE, open = "a+b") : 
  cannot open the connection
In addition: Warning message:
In socketAccept(soc, blocking = TRUE, open = "a+b") :
  problem in accepting connections on this socket

> writeH5AD(sce,'test.h5ad')
Error in socketAccept(soc, blocking = TRUE, open = "a+b") : 
  cannot open the connection
In addition: Warning message:
In socketAccept(soc, blocking = TRUE, open = "a+b") :
  problem in accepting connections on this socket

>print(act.cmd)
[1] "."                                                                                                        
[2] "'/home/labw/anaconda3/etc/profile.d/conda.sh'"                                                            
[3] "/home/labw/anaconda3/envs/py38/lib/R/bin/Rscript"                                                         
[4] "--default-packages=NULL"                                                                                  
[5] "-e"                                                                                                       
[6] "\"con <- socketConnection(port=11150, open='wb', blocking=TRUE);serialize(Sys.getenv(), con);close(con)\""

[LTLA]

After scratching my head for some time, I just skipped the failing activation in basilisk.utils (1.3.9). So it'll just give a warning and try to keep going. Maybe this will allow you to proceed with the rest of the function, I'm not sure; the failing activation may be a symptom of other problems in the environment that precludes the correct use of conda/reticulate.

@lazappi we should make #46 (and the corresponding writer) a priority to get it in the next release.

[Richard0146]

I'm not sure what I should do now.... Should I skip listener <- line in debug mode?

[LTLA]

For the time being, you can try to install basilisk.utils from GitHub:

devtools::install_github("basilisk.utils")
devtools::install_github("basilisk")

This also necessitates an install of the development version of basilisk, because those two are tied together.

In this development version, the problematic activation step is just skipped if it fails, allowing you to continue on with the rest of the function. This... may or may not work, if the underlying issue also affects the rest of reticulate's behavior.

But you might as well give it a shot, because I'm out of ideas about how why it fails on your system.

[lazappi]

I'm going to close this because it seems to be somewhat system specific and I'm hoping has been fixed in more recent release. Please comment/reopen if needed.