search

crccheck / bikeme

Apache License 2.0
3 stars 0 forks source link

omg scraper performance is terrible! #13

Open crccheck opened 9 years ago

crccheck commented 9 years ago

select/insert/update blah blah so many transactions. no wonder the scraper takes so long to run.

$ time docker run --rm --env-file=/envs/mapbike crccheck/mapbike python manage.py scrape
real    3m26.475s
user    0m0.114s
sys 0m0.192s
crccheck commented 9 years ago

after #14 

real    0m26.803s
user    0m0.078s
sys 0m0.107s
crccheck commented 9 years ago

after #15 

real    0m21.738s
user    0m0.033s
sys 0m0.103s
crccheck commented 9 years ago

if bulk_create returns the instances, it might be possible to save a huge chunk of queries by inserting all the snapshots at once and then doing the denormalize in another step