Error in interactive R session on HPC that does not come up when running in rstudio server on the same HPC setup

[JesseRop]

Dear developers,

Thank you very much for the very useful package for large datasets.

I'm able to use it smoothly through rstudio server on the HPC but I want to submit some of the seurat steps as independent jobs to our cluster since I am handling many samples. I'm therefore running the package directly on R after opening an interactive R session and encountering some errors. When I try to read in the h5 on an interactive R session on the server i get an error

library(Seurat)
library(BPCells)
h5 <- open_matrix_10x_hdf5("outs/filtered_feature_bc_matrix.h5")

caught illegal operation address 0x14def0cf1771, cause 'illegal operand'

Traceback: 1: dims_matrix_10x_hdf5_cpp(path, buffer_size) 2: open_matrix_10x_hdf5("/lustre/scratch126/tol/teams/lawniczak/users/jr35/phd/Mali2/data/raw/Pf/MSC33/outs/filtered_feature_bc_matrix.h5")

Possible actions: 1: abort (with core dump, if enabled) 2: normal R exit 3: exit R without saving workspace 4: exit R saving workspace

when I try to skip the above steps and perform normalization on the RDS object created through R on rstudio server (which work smoothly), I get the following error.

library(Seurat)
library(BPCells)

Seurat_object <- readRDS("seu_obj/bpc_seu.RDS")
Seurat_object = NormalizeData(Seurat_object, verbose = F)

caught illegal operation address 0x14650f1cf771, cause 'illegal operand'

Traceback: 1: iter_function(x@dir, x@buffer_size, x@dimnames[[1]], x@dimnames[[2]], nrow(x)) 2: iterate_matrix(x@matrix) 3: iterate_matrix(x@matrix) 4: iterate_matrix(x@matrix) 5: iterate_matrix(x@matrix) 6: iterate_matrix(convert_matrix_type(x, "double")) 7: iterate_matrix(convert_matrix_type(x, "double")) 8: .local(x, ...) 9: colSums(data) 10: colSums(data) 11: BPCells::t(BPCells::t(data)/colSums(data)) 12: LogNormalize.IterableMatrix(data = object, scale.factor = scale.factor, margin = cmargin, verbose = verbose, ...) 13: LogNormalize(data = object, scale.factor = scale.factor, margin = cmargin, verbose = verbose, ...) 14: NormalizeData.default(object = LayerData(object = object, layer = l, fast = NA), normalization.method = normalization.method, scale.factor = scale.factor, margin = margin, verbose = verbose, ...) 15: NormalizeData(object = LayerData(object = object, layer = l, fast = NA), normalization.method = normalization.method, scale.factor = scale.factor, margin = margin, verbose = verbose, ...) 16: NormalizeData.StdAssay(object = object[[assay]], normalization.method = normalization.method, scale.factor = scale.factor, verbose = verbose, margin = margin, ...) 17: NormalizeData(object = object[[assay]], normalization.method = normalization.method, scale.factor = scale.factor, verbose = verbose, margin = margin, ...) 18: NormalizeData.Seurat(Seurat_object, verbose = F) 19: NormalizeData(Seurat_object, verbose = F)

Possible actions: 1: abort (with core dump, if enabled) 2: normal R exit 3: exit R without saving workspace 4: exit R saving workspace

The output of sessionInfo is

R version 4.3.1 (2023-06-16) Platform: x86_64-pc-linux-gnu (64-bit) Running under: Ubuntu 22.04.3 LTS

Matrix products: default BLAS: /opt/software/spack_path_placeholder/spack_path_placeholder/spack_path_placeholder/spack_path_placeholder/spac/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/r-4.3.1-5g4geac7a3tp337e3vjsfwphwtylve5c/rlib/R/lib/libRblas.so LAPACK: /opt/software/spack_path_placeholder/spack_path_placeholder/spack_path_placeholder/spack_path_placeholder/spac/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/r-4.3.1-5g4geac7a3tp337e3vjsfwphwtylve5c/rlib/R/lib/libRlapack.so; LAPACK version 3.11.0

locale: [1] LC_CTYPE=en_GB.utf8 LC_NUMERIC=C
[3] LC_TIME=en_GB.utf8 LC_COLLATE=en_GB.utf8
[5] LC_MONETARY=en_GB.utf8 LC_MESSAGES=en_GB.utf8
[7] LC_PAPER=en_GB.utf8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_GB.utf8 LC_IDENTIFICATION=C

time zone: UTC tzcode source: internal

attached base packages: [1] stats graphics grDevices utils datasets methods base

other attached packages: [1] Seurat_5.0.3 SeuratObject_5.0.1 sp_2.1-3 BPCells_0.1.0

loaded via a namespace (and not attached): [1] deldir_2.0-4 pbapply_1.7-2 gridExtra_2.3
[4] rlang_1.1.3 magrittr_2.0.3 RcppAnnoy_0.0.22
[7] matrixStats_1.2.0 ggridges_0.5.6 compiler_4.3.1
[10] spatstat.geom_3.2-9 png_0.1-8 vctrs_0.6.5
[13] reshape2_1.4.4 stringr_1.5.1 pkgconfig_2.0.3
[16] fastmap_1.1.1 utf8_1.2.4 promises_1.2.1
[19] purrr_1.0.2 jsonlite_1.8.8 goftest_1.2-3
[22] later_1.3.2 spatstat.utils_3.0-4 irlba_2.3.5.1
[25] parallel_4.3.1 cluster_2.1.6 R6_2.5.1
[28] ica_1.0-3 stringi_1.8.3 RColorBrewer_1.1-3
[31] spatstat.data_3.0-4 reticulate_1.35.0 parallelly_1.37.1
[34] lmtest_0.9-40 scattermore_1.2 Rcpp_1.0.12
[37] tensor_1.5 future.apply_1.11.2 zoo_1.8-12
[40] sctransform_0.4.1 httpuv_1.6.15 Matrix_1.6-5
[43] splines_4.3.1 igraph_2.0.3 tidyselect_1.2.1
[46] abind_1.4-5 spatstat.random_3.2-3 codetools_0.2-20
[49] miniUI_0.1.1.1 spatstat.explore_3.2-7 listenv_0.9.1
[52] lattice_0.22-6 tibble_3.2.1 plyr_1.8.9
[55] shiny_1.8.1.1 ROCR_1.0-11 Rtsne_0.17
[58] future_1.33.2 fastDummies_1.7.3 survival_3.5-8
[61] polyclip_1.10-6 fitdistrplus_1.1-11 pillar_1.9.0
[64] KernSmooth_2.23-22 plotly_4.10.4 generics_0.1.3
[67] RcppHNSW_0.6.0 ggplot2_3.5.0 munsell_0.5.1
[70] scales_1.3.0 globals_0.16.3 xtable_1.8-4
[73] glue_1.7.0 lazyeval_0.2.2 tools_4.3.1
[76] data.table_1.15.4 RSpectra_0.16-1 RANN_2.6.1
[79] leiden_0.4.3.1 dotCall64_1.1-1 cowplot_1.1.3
[82] grid_4.3.1 tidyr_1.3.1 colorspace_2.1-0
[85] nlme_3.1-164 patchwork_1.2.0 cli_3.6.2
[88] spatstat.sparse_3.0-3 spam_2.10-0 fansi_1.0.6
[91] viridisLite_0.4.2 dplyr_1.1.4 uwot_0.1.16
[94] gtable_0.3.4 digest_0.6.35 progressr_0.14.0
[97] ggrepel_0.9.5 htmlwidgets_1.6.4 htmltools_0.5.8
[100] lifecycle_1.0.4 httr_1.4.7 mime_0.12
[103] MASS_7.3-60.0.1

The output of gcc --version is

gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 Copyright (C) 2021 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[bnprks]

Hi @JesseRop, what appears to be happening here is that BPCells was compiled on a machine with more advanced CPU features than the machine it ran on later. (The 'illegal operation' error is saying the CPU doesn't recognize that CPU instruction)

The quick fix for this would be to re-install BPCells from one of the computers where you get crashes, so that BPCells will compile only with the lowest-common-denominator instructions available for machines on your cluster.

I have a branch set up with changes that will eliminate this problem regardless of where BPCells was compiled (named highway-simd). Those changes will make it into the main branch eventually, and it should also be possible to specify installing from that branch in install_github() if you don't mind missing out on a couple fixes/features from the last two months that aren't yet present in that branch.

[JesseRop]

Hi @bnprks , I was able to liaise with our hpc administrators to solve the issue as advised. Thanks!