Our data pipeline stores incoming school program data in s3; this allows
the load_programs script to load these files directly fro s3 rather than
requiring the moving and handling of files.
Additions
--s3 option for the load_programs script and management command
related tests
Testing
In standalone, try running load_programs with an s3 URL:
Coverage remained the same at 100.0% when pulling 1b0205e74387df6d537672696f844184f6df5699 on load-s3 into 503ad3882338e8f1120b9fa2b3aa88b4d3b165fe on master.
Our data pipeline stores incoming school program data in s3; this allows the load_programs script to load these files directly fro s3 rather than requiring the moving and handling of files.
Additions
--s3
option for theload_programs
script and management commandTesting
If you run
./manage.py purge programs
first, it should load 1,522 programs.If you run it without purging, it should report updating 1,522 programs.
Review