Segfaults with CRAN R, BioC 3.18, and fresh Anaconda Install

[berniejmulvey]

Hi,

After several months of not utilizing the zellkonverter package for no particular reason, I came back to some code I had previously used and began encountering segfaults on my personal computer. The datasets I was trying to load were moderately large (though well within the 32GB RAM + 500GB swap allocated to my R session).

I have already tried the following things, most comprehensively in this order: •Run basilisk.utils::clearExternalDir() •remove.packages(c("basilisk","reticulate","zellkonverter") •reinstalling baslisk, reticulate, zellkonverter individually, with or without type set to "source" •reinstalling only zellkonverter and letting the installer handle the reticulate/basilisk dependencies •remove ~/anaconda3 , ~/.anaconda , ~/.conda , ~/.miniconda , ~/mambaforge directories and reinstall Anaconda for Mac, followed by both tries at R package reinstallations above •manually initializing a basilisk environment before running zellkonverter for the "first" time using basilisk::setupBasiliskEnv() •removing opt/homebrew/bin from $PATH entirely to prevent conflicts with brew's python •placing opt/homebrew/bin at the very end of $PATH to include homebrew but guarantee other python binaries supercede homebrew

Regardless, I get this segfault at most or a total crash of R and RStudio (I can get the segfault messages reliably running R from a command line window):

Minimal example:

library(zellkonverter)
example(SCE2AnnData)

Example output (after package loading messages, etc):

*** caught segfault ***
address 0x40, cause 'invalid permissions'
 *** caught segfault ***
 *** caught segfault ***
address 0x18, cause 'invalid permissions'
 *** caught segfault ***
address 0x30, cause 'invalid permissions'
 *** caught segfault ***
address 0x0, cause 'invalid permissions'
 *** caught segfault ***
address 0x48, cause 'invalid permissions'
Error in match(attrnames, specials) : 
 2 arguments passed to .Internal(match) which requires 4
 *** caught segfault ***
address 0x58, cause 'invalid permissions'
 *** caught segfault ***
 *** caught segfault ***
address 0x50, cause 'invalid permissions'
address 0x10, cause 'invalid permissions'
address 0x28, cause 'invalid permissions'
 *** caught segfault ***
Error: no more error handlers available (recursive errors?); invoking 'abort' restart
Error: no more error handlers available (recursive errors?); invoking 'abort' restart
 *** caught segfault ***
Error: no more error handlers available (recursive errors?); invoking 'abort' restart
Error: no more error handlers available (recursive errors?); invoking 'abort' restart
address 0x38, cause 'invalid permissions'
address 0x8, cause 'invalid permissions'
Traceback:
 1: Error: no more error handlers available (recursive errors?); invoking 'abort' restart
Traceback:
Traceback:
 1: structure(list(message = as.character(message), call = call),   class = class)
 2: simpleError(msg, call)
 3: tryCatchOne(expr, names, parentenv, handlers[[1L]])
 4: tryCatchList(expr, classes, parentenv, handlers)
 5: tryCatch({  oldpythonpath <- Sys.getenv("PYTHONPATH")  newpythonpath <- Sys.getenv("RETICULATE_PYTHONPATH", unset = paste(config$pythonpath,     system.file("python", package = "reticulate"), sep = .Platform$path.sep))  local({
 *** caught segfault ***
 1: example(SCE2AnnData)
Possible actions:
1: abort (with core dump, if enabled)
2: normal R exit
3: exit R without saving workspace
4: exit R saving workspace
    Sys.setenv(PYTHONPATH = newpythonpath)structure(list(message = as.character(message), call = call),   class = class)
 2: simpleError(msg, call)
 3: pairlist(NULL, NULL, NULL)
 4: tryCatchList(expr, classes, parentenv, handlers)
 5: tryCatch({  oldpythonpath <- Sys.getenv("PYTHONPATH")  newpythonpath <- Sys.getenv("RETICULATE_PYTHONPATH", unset = paste(config$pythonpath,     system.file("python", package = "reticulate"), sep = .Platform$path.sep))  local({    Sys.setenv(PYTHONPATH = newpythonpath)    on.exit(Sys.setenv(PYTHONPATH = oldpythonpath), add = TRUE)    py_initialize(config$python, config$libpython, config$pythonhome, address 0x20, cause 'invalid permissions'
    on.exit(Sys.setenv(PYTHONPATH = oldpythonpath), add = TRUE)      config$virtualenv_activate, config$version >= "3.0", 
Traceback:
 1: example(SCE2AnnData)
Possible actions:
1: abort (with core dump, if enabled)
2: normal R exit
3: exit R without saving workspace
4: exit R saving workspace
Traceback:
      interactive(), numpy_load_error)  })Selection:  1:     py_initialize(config$python, config$libpython, config$pythonhome,       config$virtualenv_activate, config$version >= "3.0",       interactive(), numpy_load_error)example(SCE2AnnData)
Selection: 
Possible actions:
1: abort (with core dump, if enabled)
2: normal R exit
3: exit R without saving workspace
4: exit R saving workspace
Selection:   })}, error = function(e) {  Sys.setenv(PATH = oldpath)  if ([is.na](http://is.na/)(curr_session_env)) {    Sys.unsetenv("R_SESSION_INITIALIZED")  }  else {    Sys.setenv(R_SESSION_INITIALIZED = curr_session_env)  }  stop(e)})
 6: initialize_python()
}, error = function(e) {  Sys.setenv(PATH = oldpath)  if ([is.na](http://is.na/)(curr_session_env)) {    Sys.unsetenv("R_SESSION_INITIALIZED")  }  else {    Sys.setenv(R_SESSION_INITIALIZED = curr_session_env)  }  stop(e)})
 6: initialize_python()
 7: ensure_python_initialized()
 8: py_config()
 9: useBasiliskEnv("python", package = "reticulate")
10: basiliskStart(env, full.activation = full.activation, fork = fork,   shared = shared, testload = testload)
11: basiliskRun(fun = function(sce) {  adata <- zellkonverter::SCE2AnnData(sce)  zellkonverter::AnnData2SCE(adata)}, env = zellkonverterAnnDataEnv(), sce = seger)
12: eval(ei, envir)
13: eval(ei, envir)
14: withVisible(eval(ei, envir))
15: source(tf, local, echo = echo, prompt.echo = paste0(prompt.prefix,   getOption("prompt")), continue.echo = paste0(prompt.prefix,   getOption("continue")), verbose = verbose, max.deparse.length = Inf,   encoding = "UTF-8", skip.echo = skips, keep.source = TRUE)
16: example(SCE2AnnData)
Possible actions:
1: abort (with core dump, if enabled)
2: normal R exit
3: exit R without saving workspace
4: exit R saving workspace
Selection:  7: ensure_python_initialized()
 8: py_config()
 9: useBasiliskEnv(envpath, full.activation)
10: basiliskStart(env, full.activation = full.activation, fork = fork,   shared = shared, testload = testload)
11: basiliskRun(fun = function(sce) {  adata <- zellkonverter::SCE2AnnData(sce)  zellkonverter::AnnData2SCE(adata)}, env = zellkonverterAnnDataEnv(), sce = seger)
12: eval(ei, envir)
13: eval(ei, envir)
14: withVisible(eval(ei, envir))
15: source(tf, local, echo = echo, prompt.echo = paste0(prompt.prefix,   getOption("prompt")), continue.echo = paste0(prompt.prefix,   getOption("continue")), verbose = verbose, max.deparse.length = Inf,   encoding = "UTF-8", skip.echo = skips, keep.source = TRUE)
16: example(SCE2AnnData)

Then, bizarrely at this point, I get several errors before/while I try and choose a “possible action” before I can actually send a response:

Possible actions:
1: abort (with core dump, if enabled)
2: normal R exit
3: exit R without saving workspace
4: exit R saving workspace
Selection: 
 *** caught bus error ***
address 0x411000006, cause 'invalid alignment'
Traceback:
 1: example(SCE2AnnData)
Possible actions:
1: abort (with core dump, if enabled)
2: normal R exit
3: exit R without saving workspace
4: exit R saving workspace
Selection: 
 *** caught bus error ***
address 0x156031808, cause 'invalid alignment'
Traceback:
 1: example(SCE2AnnData)
Possible actions:
1: abort (with core dump, if enabled)
2: normal R exit
3: exit R without saving workspace
4: exit R saving workspace
Selection: 
 *** caught bus error ***
address 0xfffffe67ae584000, cause 'invalid alignment'
Traceback:
 1: example(SCE2AnnData)
Possible actions:
1: abort (with core dump, if enabled)
2: normal R exit
3: exit R without saving workspace
4: exit R saving workspace
Selection: 3
 *** caught bus error ***
address 0x11c7bc000, cause 'invalid alignment'
Traceback:
 1: example(SCE2AnnData)
Possible actions:
1: abort (with core dump, if enabled)
2: normal R exit
3: exit R without saving workspace
4: exit R saving workspace
Selection: 3
 *** caught bus error ***
address 0x10a574000, cause 'invalid alignment'
Traceback:
 1: example(SCE2AnnData)
Possible actions:
1: abort (with core dump, if enabled)
2: normal R exit
3: exit R without saving workspace
4: exit R saving workspace
Selection: 3

Finally, some miscellany: sessionInfo()

R version 4.3.2 (2023-10-31)
Platform: aarch64-apple-darwin20 (64-bit)
Running under: macOS Ventura 13.6

Matrix products: default
BLAS:   /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/lib/libRblas.0.dylib 
LAPACK: /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/lib/libRlapack.dylib;  LAPACK version 3.11.0

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

time zone: America/Chicago
tzcode source: internal

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] zellkonverter_1.12.1 rlang_1.1.3         

loaded via a namespace (and not attached):
 [1] crayon_1.5.2                cli_3.6.2                   png_0.1-8                   DelayedArray_0.28.0        
 [5] jsonlite_1.8.8              dir.expiry_1.10.0           SummarizedExperiment_1.32.0 S4Vectors_0.40.2           
 [9] RCurl_1.98-1.14             stats4_4.3.2                MatrixGenerics_1.14.0       Biobase_2.62.0             
[13] grid_4.3.2                  abind_1.4-5                 filelock_1.0.3              bitops_1.0-7               
[17] SingleCellExperiment_1.24.0 IRanges_2.36.0              basilisk_1.14.1             GenomeInfoDb_1.38.5        
[21] compiler_4.3.2              Rcpp_1.0.12                 XVector_0.42.0              lattice_0.22-5             
[25] SparseArray_1.2.3           reticulate_1.34.0           parallel_4.3.2              GenomeInfoDbData_1.2.11    
[29] GenomicRanges_1.54.1        Matrix_1.6-4                tools_4.3.2                 matrixStats_1.2.0          
[33] zlibbioc_1.48.0             S4Arrays_1.2.0              basilisk.utils_1.14.1       BiocGenerics_0.48.1

Makevars (which have not changed since I originally installed and successfully used zellkonverter many moons ago):

LOC = /opt/gfortran
CC=$(LOC)/bin/aarch64-apple-darwin20.0-gcc -fopenmp
CXX=/usr/bin/g++ #-fopenmp
CXX11 = /usr/bin/g++ #-fopenmp # for fst package

SHLIB_OPENMP_CFLAGS= -fopenmp
CFLAGS=-g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe
CXXFLAGS=-g -O3 -Wall -pedantic -std=c++11 -mtune=native -pipe
LDFLAGS=-L$(LOC)/lib -Wl,-rpath,$(LOC)/lib
CPPFLAGS=-I$(LOC)/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX12.3.sdk/usr/include/c++/v1
LDFLAGS += -L/opt/homebrew/opt/gettext/lib -L/opt/homebrew/opt/libomp/lib -lomp
CPPFLAGS += -I/opt/homebrew/opt/gettext/include -I/opt/homebrew/opt/libomp/include -Xpreprocessor -fopenmp

System $PATH as reported from within R.app / Rstudio (which is something R devs are aware of being discrepant/redundant relative to the command line reported $PATH on M-family Macs). Currently is excluding homebrew altogether.

"/Users/bmulvey/anaconda3/condabin:/opt/gfortran/bin:/Library/Frameworks/R.framework/Resources:/opt/local/bin:/opt/local/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/opt/X11/bin:/Library/Apple/usr/bin:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/opt/homebrew/bin:/usr/local/bin"

[berniejmulvey]

Alright, a complete wipe of my R, Rstudio, and anaconda, and re-installing all of my previously installed packages fixed things, but I think the workaround that was really key was the following. I was getting errors about the absence of zlib.1.dylib, even when masking out my custom $PATH variables from the R session, when zellkonverter tried to build its environment via basilisk. (This was an issue before wiping the packages as well, after I removed the entire basilisk directory from Caches per below).

1. For a Mac user, remove any subdirectories in /Users/YOU/Library/Caches/org.R-project.R/R/basilisk

2. Open an R session and run remove.packages(c("reticulate","basilisk","zellkonverter","basilisk.utils"))

3. Close the R session

4. In a new session, run: BiocManager::install("zellkonverter") library(reticulate) reticulate::install_miniconda()

5. In Rstudio, go to Tools --> Global Options --> Python, and click Select...

6. Choose the Conda Environments tab and choose the r-reticulate environment that was built by the commands in step 4 (mine was at ~/Library/r-miniconda-arm64/envs/r-reticulate/bin/python3.9 ).

7. At this point, basilisk.utils::install_conda was still returning an error. I incidentally found something that I was hoping would give me a more explanatory error message, but instead, the conda installation finally worked when commanded this way, but ONLY if run in a fresh command line / R.app session (!): library(basilisk.installConda()

8. NOW load zellkonverter and try example(SCE2AnnData), which should successfully create the conda environment and yield a new object in the workspace, roundtrip.

9. Confirm that other functions, e.g. sce <- readH5AD work as expected.

Why it worked is for someone above my level of knowledge, but hopefully this is a useful reference if someone else gets stuck with this issue.

[lazappi]

Thanks for sharing what you tried. Unfortunately things sometimes get mixed up, particularly when you are also using things like homebrew.

One note for anyone who reads this later. SCE2AnnData() operates in whichever Python environment is detected by {reticulate} unlike readH5AD() which operates in a special {basilisk} environment. Which to use depends on how experienced you are with Python but we recommend readH5AD() for new users. Doing work using {reticulate} and then calling readH5AD() can sometimes cause issues because there are two environments being used.