Add new data for fall 2019. In the process, upgrade a few dependencies that were raising errors, notably raven (the Sentry SDK) and openpyxl (Excel reading library). In the later case, make some adjustments to the import script to match new APIs.
Notes
I had to do some manual cleaning on Cam_ContribExpenditure to remove improperly-formatted Excel carriage returns (which show up as the string _x000d_). It's likely that we'll continue to see these carriage returns in future dumps if the SOS delivers us another Excel file for Cam_ContribExpenditure instead of the zipped CSV that they used to send us.
Testing instructions
Pull down this branch
Start your virtualenv and run pip install -U -r requirements.txt
Run ./manage.py import_data and confirm it updates all data (N.B. this takes a few hours on my machine)
Overview
Add new data for fall 2019. In the process, upgrade a few dependencies that were raising errors, notably
raven(the Sentry SDK) andopenpyxl(Excel reading library). In the later case, make some adjustments to the import script to match new APIs.Notes
I had to do some manual cleaning on
Cam_ContribExpenditureto remove improperly-formatted Excel carriage returns (which show up as the string_x000d_). It's likely that we'll continue to see these carriage returns in future dumps if the SOS delivers us another Excel file forCam_ContribExpenditureinstead of the zipped CSV that they used to send us.Testing instructions
pip install -U -r requirements.txt./manage.py import_dataand confirm it updates all data (N.B. this takes a few hours on my machine)