search

GangLiLab / genekitr

🧬 Gene analysis toolkit based on R
https://www.genekitr.fun
GNU General Public License v3.0
53 stars 7 forks source link

new version is memory hungry #7

Closed MEladawi closed 1 year ago

MEladawi commented 1 year ago

Hello--

In the previous version, I used to convert 30k gene symbols in one command on my machine with 32GB and never had a problem. Now, when I try to run the same command (transId) on the same symbols, even a machine with 128GN will kill the process as the memory is not enough.

reedliu commented 1 year ago

Hi, The new version didn't change main functions. It just utilized new backend data. So i think it may not be the real problem since you tested on last machine successfully. For your new machine, please check R memory limit setting following the link, https://stackoverflow.com/questions/1395229/increasing-or-decreasing-the-memory-available-to-r-processes. If things happens, please try to use small sets to run transId on the new machine and see if it works.

MEladawi commented 1 year ago

Thanks,

Last time I tested was 6 weeks ago. I updated the symbols using transID(). I am running the same code using the same machine on the same data now and it is taking 28GB and freezes. I tried on HPC system with 128GB and the used memory keeps increasing until all the 128GB are full and the process is killed.

reedliu commented 1 year ago

I have tested ~56k mouse ensembl IDs (from gencode.vM32.annotation.gtf) and used transId to convert to symbols. Seems that it did not consume much memory.

image
MEladawi commented 1 year ago

From ensembl IDs no problem, but from symbol to symbol (to get updated symbols), this problem occurs.

MEladawi commented 1 year ago

symbols.csv

Please try these symbols for example. I use the following code: updated_sym <- transId(id =symbols, transTo= "symbol", org = "mouse", keepNA = FALSE, unique = FALSE)

reedliu commented 1 year ago

I have tested your sample file and no memory problem.

image
MEladawi commented 1 year ago

Hmm this is very weird.

MEladawi commented 1 year ago

are you using v1.1.3?

reedliu commented 1 year ago

Ah, I was running on another R 4.0 machine (genekitr v1.1.0). After changing to v1.1.3, I found this problem because of a typo in symbol IDs converting function(merge() => imerge() 😖). Anyway, it is fixed now and you could try to install the v1.1.4 developmental version remotes::install_github("GangLiLab/genekitr"). The new version will be available soon on CRAN. Thanks for your feedback and sorry for any inconvenience.

MEladawi commented 1 year ago

Yes, it is fixed now. Thanks.