This adds two checks for the situation that caused Milton Keynes to fail during final import and bring the API down with it. The root of the problem was that there was no population in the imported blocks within the neighborhood boundary, but the way that error manifested was not helpful. It ran the whole analysis and imported the broken score results, then failed because the blank population value caused an error on the database level for any query that tried to load the record.
The two new pieces of error-checking:
one early in the analysis, as soon as the blocks are imported, so that it will raise a more informative error and avoid running a whole analysis with bad data
a second in the command that imports analysis results into the database, to raise an error if it encounters an empty population value, rather than importing it and causing the database to start throwing errors.
Resolves #911
Notes
This is based on the branch for PR #912 because it needs those fixes, but I didn't want to make it all into one even-bigger PR.
Testing Instructions
Kick off an analysis for Milton Keynes (milton-keynes.zip) using https://pfb-public-documents.s3.amazonaws.com/population/tabblock2010_99_pophu.zip for the "Population File URL" and checking the "Omit jobs data in analysis" box. It should get as far as importing "census" blocks and truncating them to the boundary, but then instead of continuing it should print the message "Error: Total population of included census blocks is zero." and stop.
Try to load this problematic overall scores file, neighborhood_overall_scores.csv, onto a job in your local instance. I.e. download it into the src/django/ directory so it will be visiblel to the Django container then run
replacing JOB_UUID with the UUID of one of the analysis jobs in your database. It should be one you don't care too much about, since I think it will overwrite some of the scores before it hits the crash. But the result of the command should be that it throws an exception and does not mess up the record and cause API calls to start crashing.
Overview
This adds two checks for the situation that caused Milton Keynes to fail during final import and bring the API down with it. The root of the problem was that there was no population in the imported blocks within the neighborhood boundary, but the way that error manifested was not helpful. It ran the whole analysis and imported the broken score results, then failed because the blank population value caused an error on the database level for any query that tried to load the record.
The two new pieces of error-checking:
Resolves #911
Notes
This is based on the branch for PR #912 because it needs those fixes, but I didn't want to make it all into one even-bigger PR.
Testing Instructions
https://pfb-public-documents.s3.amazonaws.com/population/tabblock2010_99_pophu.zipfor the "Population File URL" and checking the "Omit jobs data in analysis" box. It should get as far as importing "census" blocks and truncating them to the boundary, but then instead of continuing it should print the message "Error: Total population of included census blocks is zero." and stop.src/django/directory so it will be visiblel to the Django container then runreplacing JOB_UUID with the UUID of one of the analysis jobs in your database. It should be one you don't care too much about, since I think it will overwrite some of the scores before it hits the crash. But the result of the command should be that it throws an exception and does not mess up the record and cause API calls to start crashing.
Checklist