mikemccllstr / dominionstats

The code behind councilroom.com.
http://councilroom.com
11 stars 3 forks source link

Update the isotropic game scraper to leverage the new approach #7

Closed mikemccllstr closed 11 years ago

mikemccllstr commented 12 years ago

The scraper script should now:

  1. Retrieve the day's archive from isotropic
  2. Store a verbatim copy in the S3 bucket
  3. Create a Celery task instance to repackage it

The Celery task should:

  1. Retrieve the file from S3
  2. Decompress it
  3. Recompress each game, individually
  4. Insert the compressed game into MongoDB
  5. Create a Celery task instance to parse and insert the game
mikemccllstr commented 11 years ago

Appears to be functioning as of 91f2b4460.