Issue when running large dataset

[cnk113]

Hello,

I was running TF activity inference on my large dataset but I seem to ran into an obscure error which I can't debug. Any ideas why this might be occurring? This works on a subset of the data.

network2 <- get_dorothea(organism = "human",
+                                    levels = c("A", "B", "C"))
[2022-11-05 12:42:33] [SUCCESS] [OmnipathR] Loaded 278482 interactions from cache.
> activities2 <- run_wmean(mat = as.matrix(second@assays[["SCT"]]@data),
+                                    network = network2,
+                                    .source = "source",
+                                    .targe = "target",
+                                    .mor = "mor",
+                                    times = 100,
+                                    minsize = 5)
Error in `dplyr::bind_rows()`:
! attempt to set index 210941704/210941704 in SET_STRING_ELT
Run `rlang::last_error()` to see where the error occurred.
Warning message:
In asMethod(object) :
  sparse->dense coercion: allocating vector of size 37.3 GiB

Thanks, Chang

[PauBadiaM]

Hi @cnk113,

Thanks for checking out the package! It looks like you are running into memory problems due to the size of your dataset. Here are some alternatives that you can try:

• Run the activity inference per "batches of cells", meaning, split your dataset into chunks and then merge the obtained activities.
• If you are running it in a cluster, increase the memory limit.
• Alternatively, switch to the python version of decoupler, which is far more memory and computationally efficient than its R counterpart: https://decoupler-py.readthedocs.io/en/latest/

Hope this is helpful!