Regularly scrape GDELT sources into a database

[karimbahgat]

NOTE: Depends on #13

Working folder:

/resources/gdelt_source_urls/

Purpose:

In main.py:

Create function scrape_gdelt_sources (no args):

• [X] Connect to the scopesourcedata mysql database (same login info as the scope django website).
• [X] Read master file list (http://data.gdeltproject.org/gdeltv2/masterfilelist.txt)
• [x] Loop through "mentions" csv files.
• [X] If file doesn't already exist in database (check if exists or not).
  • [X] Loop through source rows using iter_gdelt_sources in #13
  • [x] If row key 'MentionIdentifier' does not already exist in the database:
    • [x] Add to database (table name: 'TestGdeltSource'). The dict row should match exactly with the table column names:
    • file_url
    • MentionType
    • MentionSourceName
    • MentionIdentifier
    • MentionDocTranslationInfo

Run scrape_gdelt_sources and make sure that it works:

• [X] Check when finished

Implement GDELT script into Django pipeline module

• [ ] Get GDELT script from Lanond
• [ ] Run to see if it works properly/make it more robust or cleaner if possible
• [ ] Create a pipeline module in Django that puts GDELT sources into the MySQL database every 15 minutes
• [ ] Create a keyword search on all the sources that were pulled during a single iteration

[karimbahgat]

I added some comments to your previous commit @lclime812 , and updated this issue with more information about the database and table/columns.

If you don't already know SQL, the relevant parts you'll need is select where, and insert into.