search

pearselab / MADcomm

Make A Database of ecological communities
Other
1 stars 15 forks source link

Hitting memory capacity #14

Open bweedop opened 6 years ago

bweedop commented 6 years ago

Currently, loading some of the data in downloads.R can cause issues with system memory if other tasks are being performed. We may need to consider different data structures (e.g. data tables vs. data frames) and combining functions that use the same data sets to reduce stress on system memory.

willpearse commented 6 years ago

Confirmed by @mrhelmus. I'm moving this to a "bug", because if Matt can't load the files then that's clearly a problem. As wonderful as it is to have a demonstration of the power of Linux, this isn't very helpful behavior...

willpearse commented 4 years ago

Hey past Will, it's Will From The Future - a related problem is that some of these functions take a very long time to download because they're grabbing so much data (e.g., biotime.2018). You need to write something that flags these (and, in so-doing, would solve the memory problem in a stupid way).