search

CSCI-4830-002-2014 / challenge-week-11

0 stars 13 forks source link

overly sized ebird data #13

Open peymanmortazavi opened 9 years ago

peymanmortazavi commented 9 years ago

It took me 50 minutes to import the data in the MongoDB. I think it would've been more productive and efficient if the size of the file was more reasonable considering that we need do these things on our laptops not on servers capable of handling real big data tasks.

dawsbot commented 9 years ago

Same thing happened to me. Not enough space to do this assignment on my computer or my roomates'. Going to take the grade hit on this one, we all gave it a fair share of effort.

ianks commented 9 years ago

Make sure you give your VM a hunk of memory if you have it. As far as converting the data goes, you also need to pay attention to things like spatial locality etc, or you will just constantly evicting blocks from cache, and doing too many memory reads.

dawsbot commented 9 years ago

Can't allocate more memory since neither Peyman nor I are in a VM. Natively I have already allocated my partition the maximum amount. He is also running natively and has reached max capacity on his SSD. ᐧ

Dawson Botsford

http://hackcu.org/

Dawson Botsford | Co-Founder HackCU dawsonbotsford@gmail.com dawsonbotsford@gmail.com | 865-804-9305 | dawsonbotsford.com

[image: Facebook] https://www.facebook.com/dawson.botsford [image: Twitter] https://twitter.com/DawsonBotsford [image: Linkedin] https://www.linkedin.com/pub/dawson-botsford/9a/91b/94b

On Sun, Nov 9, 2014 at 6:29 PM, Ian Ker-Seymer notifications@github.com wrote:

Make sure you give your VM a hunk of memory if you have it. As far as converting the data goes, you also need to pay attention to things like spatial locality etc, or you will just constantly evicting blocks from cache, and doing too many memory reads.

— Reply to this email directly or view it on GitHub https://github.com/CSCI-4830-002-2014/challenge-week-11/issues/13#issuecomment-62330274 .