search

caitiecollins / treeWAS

treeWAS: A Phylogenetic Tree-Based Tool for Genome-Wide Association Studies in Microbes
Other
92 stars 18 forks source link

Memory problems #51

Closed GaryNapier closed 2 years ago

GaryNapier commented 4 years ago

Hi, I was using treeWAS on our server, but I kept running out of memory in the final treeWAS() function.

It was taking about 12 hours to run before finally crashing. I do have a large dataset, but I thought I'd let you know anyway:

I have ~17k samples and ~10k SNPs, and I applied the get.binary.snps() to cut the columns down in the final matrix.

After the first crash I cut down the samples by half, but still got the memory error.

The error messages for the crash are:

17k samples - so 17k x 10k matrix: Error: cannot allocate vector of size 25.5 Gb

8 k samples - so 8k x 10k matrix: [1] "treeWAS snps sim done @ 2020-07-01 10:05:51" Error: cannot allocate vector of size 16.0 Gb

I'm puzzled by the second one where it says it's done (or is it referring to something else?) then says it's run out of memory.

Thanks, Gary

hollygene commented 3 years ago

Hi Gary,

Have you had any luck resolving this issue? I'm running into the same problem.

Thanks! -Holly

GaryNapier commented 3 years ago

Hi Holly,

I haven't tried treeWAS for a while as I've started new projects, but I'll probably be using it again in the coming few months. I will post here if there are still memory problems or I hear about any kind of solutions.

Thanks,

Gary