search

drisso / zinbwave

Clone of the Bioconductor repository for the zinbwave package, see https://bioconductor.org/packages/zinbwave
43 stars 10 forks source link

maximal number of DLLs reached #25

Closed bio-la closed 6 years ago

bio-la commented 6 years ago

Hi Davide, this tool is just amazing. It's 10 times better than CC alignment, on my data. I managed to make it work with a lot of pain on 3 different machines (a macOs and 2 linux servers) but there are a couple of annoying issues and I don't know if you can do something about it:

I often (but not always and I know it depends a lot on what's going on on the cluster at the moment the task is performed!) get an error on the parallelization Error in serialize(data, node$con, xdr = FALSE) : error writing to connection Calls: zinbFit ... .send_EXEC -> -> sendData.SOCK0node -> serialize In addition: Warning message: In simpute.als(x, J, thresh, lambda, maxit, trace.it, warm.start, : Convergence not achieved by 100 iterations

and most importantly maximal number of DLLs reached

Error: package or namespace load failed for ‘Seurat’ in dyn.load(file, DLLpath = DLLpath, ...): unable to load shared object '/gfs/apps/apps/R-3.4.2/lib64/R/library/Seurat/libs/Seurat.so': `maximal number of DLLs reached...

sessionInfo() Error in La_library() : LAPACK routines cannot be loaded In addition: Warning message: In La_library() : unable to load shared object '/gfs/apps/apps/R-3.4.2/lib64/R/modules//lapack.so': `maximal number of DLLs reached...

I tried setting the relevant environment variables (R_MAX_NUM_DLLS for example) but I understand that it's quite different on each different machine and it does depend on how the memory is allocated (for example I can't run zinbwave on Rstudio on my macbook but I do run it via terminal) I eventually end up saving intermediate steps and aggregate the results on a new R session. is there a way to deal with the number of dependencies? (reduce them, ideally)

When it does work, it's just so great.

Many thanks!

drisso commented 6 years ago

Hi,

thanks for the feedback!

Regarding your questions:

Error in serialize(data, node$con, xdr = FALSE) :
error writing to connection
Calls: zinbFit ... .send_EXEC -> -> sendData.SOCK0node -> serialize

My guess is that this is simply because you are running out of memory. One thing that is still puzzling me is that it seems that increasing the number of cores has a big impact on memory footprint (which should not be since multicore should use shared memory...), so trying a smaller number of cores might help here.

Error: package or namespace load failed for ‘Seurat’ in dyn.load(file, DLLpath = DLLpath, ...):
unable to load shared object '/gfs/apps/apps/R-3.4.2/lib64/R/library/Seurat/libs/Seurat.so':
`maximal number of DLLs reached...

This as you said can be fixed by changing the R_MAX_NUM_DLLS variable. I also understand that it's annoying to set this up on every machine, but if you add / edit a .Renviron file in your home and set the R_MAX_NUM_DLLS variable there, at least you only have to do it once for each machine. As far as I know this will affect all the subsequent R sessions (both started from the terminal and through RStudio).

drisso commented 6 years ago

Can I close this?

bio-la commented 6 years ago

yes, i'm closing it now, thank you.