This PR adds a Makefile that downloads, compresses, and stores contribution and expenditure data, and a management command to import contribution data.
The management command takes 60-90 seconds locally, but 15-30 minutes on Heroku. As it will not be run frequently in a deployment environment, I feel trading off longer run time for the legibility of the ORM, rather than writing raw SQL, is more than worth it.
There are some errors resulting from missing expenditure data, but I'm going to wait to handle them until I extend the new script to import expenditures.
There are some weird edge cases to handle, e.g., the same candidate sometimes has a different OrgID across records, resulting in duplicates. We should figure out why this is and address it.
Testing Instructions
Tested extensively locally and against the review app.
Overview
This PR adds a Makefile that downloads, compresses, and stores contribution and expenditure data, and a management command to import contribution data.
The management command takes 60-90 seconds locally, but 15-30 minutes on Heroku. As it will not be run frequently in a deployment environment, I feel trading off longer run time for the legibility of the ORM, rather than writing raw SQL, is more than worth it.
Notes
Some outstanding issues:
Testing Instructions
Tested extensively locally and against the review app.