search

linwoodc3 / gdeltPyR

Python based framework to retreive Global Database of Events, Language, and Tone (GDELT) version 1.0 and version 2.0 data.
https://linwoodc3.github.io/gdeltPyR/
GNU General Public License v3.0
203 stars 54 forks source link

Add "get.data" function to download master list #2

Closed linwoodc3 closed 8 years ago

linwoodc3 commented 8 years ago

This will reduce the load time and the run time of the search function. Right now, for GDELT Version 2.0, a single day query takes 45-50 s. With this new functionality, we'll only make calls to the last 15 minute query or the historical get data master list.

linwoodc3 commented 8 years ago

This is OBE; removed download of master list completely. We build the urls based on the dates pass in now and use multiprocessing to run simultaneous downloads. Big speed up.