search

CSCI-4830-002-2014 / challenge-week-11

0 stars 13 forks source link

overly sized ebird data #13

Open peymanmortazavi opened 10 years ago

peymanmortazavi commented 10 years ago

It took me 50 minutes to import the data in the MongoDB. I think it would've been more productive and efficient if the size of the file was more reasonable considering that we need do these things on our laptops not on servers capable of handling real big data tasks.

dawsbot commented 10 years ago

Same thing happened to me. Not enough space to do this assignment on my computer or my roomates'. Going to take the grade hit on this one, we all gave it a fair share of effort.

ianks commented 10 years ago

Make sure you give your VM a hunk of memory if you have it. As far as converting the data goes, you also need to pay attention to things like spatial locality etc, or you will just constantly evicting blocks from cache, and doing too many memory reads.

dawsbot commented 10 years ago

Can't allocate more memory since neither Peyman nor I are in a VM. Natively I have already allocated my partition the maximum amount. He is also running natively and has reached max capacity on his SSD. ᐧ

Dawson Botsford

http://hackcu.org/

Dawson Botsford | Co-Founder HackCU dawsonbotsford@gmail.com dawsonbotsford@gmail.com | 865-804-9305 | dawsonbotsford.com

[image: Facebook] https://www.facebook.com/dawson.botsford [image: Twitter] https://twitter.com/DawsonBotsford [image: Linkedin] https://www.linkedin.com/pub/dawson-botsford/9a/91b/94b

On Sun, Nov 9, 2014 at 6:29 PM, Ian Ker-Seymer notifications@github.com wrote:

Make sure you give your VM a hunk of memory if you have it. As far as converting the data goes, you also need to pay attention to things like spatial locality etc, or you will just constantly evicting blocks from cache, and doing too many memory reads.

— Reply to this email directly or view it on GitHub https://github.com/CSCI-4830-002-2014/challenge-week-11/issues/13#issuecomment-62330274 .