This changeset modifies the download of observation data from Data Garrison to explicitly hit a download.php script on the Data Garrison server to make sure that the observations TSV file has the latest readings.
Why was it changed?
These was an issue where station observation data from Data Garrison would not update on the remote server, causing the Data Transloader ETL to not download any new observations. The TSV on the remote server is only updated when a download.php webpage is loaded on Data Garrison.
Additional Notes
Additional metadata is stored locally for Data Garrison TSV files in order to use the download.php script correctly
A utility script was added (cassette_cleaner.rb) to solve an issue with content-length headers in the test fixtures
The "timecop" gem is now used to "freeze" the time when the tests are run, as the download.php script requires including the current timestamp
I used git rebase to modify the commits in the PR to reduce the file sizes of the HTTP test fixture files, as they were getting too large and slowing down test runs. This is the main reason for the large number of lines removed in this PR
What changed?
This changeset modifies the download of observation data from Data Garrison to explicitly hit a
download.phpscript on the Data Garrison server to make sure that the observations TSV file has the latest readings.Why was it changed?
These was an issue where station observation data from Data Garrison would not update on the remote server, causing the Data Transloader ETL to not download any new observations. The TSV on the remote server is only updated when a
download.phpwebpage is loaded on Data Garrison.Additional Notes
download.phpscript correctlycassette_cleaner.rb) to solve an issue withcontent-lengthheaders in the test fixturesdownload.phpscript requires including the current timestamp