The questionnaire used to collect data for ZOA projects contained used incorrect field names for location and party resources. The files are uploaded to the server and attached to the project but weren't attached to the corresponding location or parties.
The original submissions are available, which we use in the script to attach the resources to the corresponding locations and parties.
Submissions are stored on XFormSubmission; the json_submission field contains the raw data of the original submission.
Field names used in the form are:
renco_resource_photo for location photos
id_resource_photo for party photos
The mappings of submissions to locations and parties are in XFormSubmission.spatial_units and XFormSubmission.parties
Description of the change
I implemented a one-off command to that we can run manually to clean up the data.
Iterate through a list of projects.
For each project, get the submission
For each submission, check whether renco_resource_photo or id_resource_photo are set.
If the fields are set, find the corresponding resource object as well as location or party entity and create a ContentObject, which links the resource with the entity.
How someone else can test the change
On your local machine, use a fresh install of the platform, or reset the database (./manage.py reset database). Before running the reset command, I diabled the creation of triggers in cadasta/organization/migrations/0006_add_project_area_trigger.py and cadasta/spatial/migrations/0007_make_area_not_null.py because they where creating problems when importing the dump and the areas are not relevant for the task at hand.
Load the production dump. I dropped the dump on S3, the link is posted on Slack (psql -U cadasta -d cadasta -h localhost < dump.sql)
Run the command ./manage.py fix_zoa
When should this PR be merged
When approved.
Risks
We'll need to test the script on staging against the production DB.
Follow-up actions
Test on staging
Applly to production
Checklist (for reviewing)
General
Is this PR explained thoroughly? All code changes must be accounted for in the PR description.
[ ] Review 1
[ ] Review 2
Is the PR labeled correctly? It should have the migration label if a new migration is added.
[ ] Review 1
[ ] Review 2
Is the risk level assessment sufficient? The risks section should contain all risks that might be introduced with the PR and which actions we need to take to mitigate these risks. Possible risks are database migrations, new libraries that need to be installed or changes to deployment scripts.
[ ] Review 1
[ ] Review 2
Functionality
Are all requirements met? Compare implemented functionality with the requirements specification.
[ ] Review 1
[ ] Review 2
Does the UI work as expected? There should be no Javascript errors in the console; all resources should load. There should be no unexpected errors. Deliberately try to break the feature to find out if there are corner cases that are not handled.
[ ] Review 1
[ ] Review 2
Code
Do you fully understand the introduced changes to the code? If not ask for clarification, it might uncover ways to solve a problem in a more elegant and efficient way.
[ ] Review 1
[ ] Review 2
Does the PR introduce any inefficient database requests? Use the debug server to check for duplicate requests.
[ ] Review 1
[ ] Review 2
Are all necessary strings marked for translation? All strings that are exposed to users via the UI must be marked for translation.
[ ] Review 1
[ ] Review 2
Is the code documented sufficiently? Large and complex classes, functions or methods must be annotated with comments following our code-style guidelines.
[ ] Review 1
[ ] Review 2
Has the scalability of this change been evaluated?
[ ] Review 1
[ ] Review 2
Is there a maintenance plan in place?
[ ] Review 1
[ ] Review 2
Tests
Are there sufficient test cases? Ensure that all components are tested individually; models, forms, and serializers should be tested in isolation even if a test for a view covers these components.
[ ] Review 1
[ ] Review 2
If this is a bug fix, are tests for the issue in place? There must be a test case for the bug to ensure the issue won’t regress. Make sure that the tests break without the new code to fix the issue.
[ ] Review 1
[ ] Review 2
If this is a new feature or a significant change to an existing feature? has the manual testing spreadsheet been updated with instructions for manual testing?
[ ] Review 1
[ ] Review 2
Security
Confirm this PR doesn't commit any keys, passwords, tokens, usernames, or other secrets.
[ ] Review 1
[ ] Review 2
Are all UI and API inputs run through forms or serializers?
[ ] Review 1
[ ] Review 2
Are all external inputs validated and sanitized appropriately?
[ ] Review 1
[ ] Review 2
Does all branching logic have a default case?
[ ] Review 1
[ ] Review 2
Does this solution handle outliers and edge cases gracefully?
[ ] Review 1
[ ] Review 2
Are all external communications secured and restricted to SSL?
[ ] Review 1
[ ] Review 2
Documentation
Are changes to the UI documented in the platform docs? If this PR introduces new platform site functionality or changes existing ones, the changes must be documented in the Cadasta Platform Documentation.
[ ] Review 1
[ ] Review 2
Are changes to the API documented in the API docs? If this PR introduces new API functionality or changes existing ones, the changes must be documented in the API docs.
[ ] Review 1
[ ] Review 2
Are reusable components documented? If this PR introduces components that are relevant to other developers (for instance a mixin for a view or a generic form) they should be documented in the Wiki.
Proposed changes in this pull request
Why I made this change
The questionnaire used to collect data for ZOA projects contained used incorrect field names for location and party resources. The files are uploaded to the server and attached to the project but weren't attached to the corresponding location or parties.
The original submissions are available, which we use in the script to attach the resources to the corresponding locations and parties.
Submissions are stored on
XFormSubmission
; thejson_submission
field contains the raw data of the original submission.Field names used in the form are:
renco_resource_photo
for location photosid_resource_photo
for party photosThe mappings of submissions to locations and parties are in
XFormSubmission.spatial_units
andXFormSubmission.parties
Description of the change
I implemented a one-off command to that we can run manually to clean up the data.
renco_resource_photo
orid_resource_photo
are set.ContentObject
, which links the resource with the entity.How someone else can test the change
./manage.py reset database
). Before running the reset command, I diabled the creation of triggers incadasta/organization/migrations/0006_add_project_area_trigger.py
andcadasta/spatial/migrations/0007_make_area_not_null.py
because they where creating problems when importing the dump and the areas are not relevant for the task at hand.psql -U cadasta -d cadasta -h localhost < dump.sql
)./manage.py fix_zoa
When should this PR be merged
When approved.
Risks
We'll need to test the script on staging against the production DB.
Follow-up actions
Checklist (for reviewing)
General
Is this PR explained thoroughly? All code changes must be accounted for in the PR description.
Is the PR labeled correctly? It should have the
migration
label if a new migration is added.Is the risk level assessment sufficient? The risks section should contain all risks that might be introduced with the PR and which actions we need to take to mitigate these risks. Possible risks are database migrations, new libraries that need to be installed or changes to deployment scripts.
Functionality
Are all requirements met? Compare implemented functionality with the requirements specification.
Does the UI work as expected? There should be no Javascript errors in the console; all resources should load. There should be no unexpected errors. Deliberately try to break the feature to find out if there are corner cases that are not handled.
Code
Do you fully understand the introduced changes to the code? If not ask for clarification, it might uncover ways to solve a problem in a more elegant and efficient way.
Does the PR introduce any inefficient database requests? Use the debug server to check for duplicate requests.
Are all necessary strings marked for translation? All strings that are exposed to users via the UI must be marked for translation.
Is the code documented sufficiently? Large and complex classes, functions or methods must be annotated with comments following our code-style guidelines.
Has the scalability of this change been evaluated?
Is there a maintenance plan in place?
Tests
Are there sufficient test cases? Ensure that all components are tested individually; models, forms, and serializers should be tested in isolation even if a test for a view covers these components.
If this is a bug fix, are tests for the issue in place? There must be a test case for the bug to ensure the issue won’t regress. Make sure that the tests break without the new code to fix the issue.
If this is a new feature or a significant change to an existing feature? has the manual testing spreadsheet been updated with instructions for manual testing?
Security
Confirm this PR doesn't commit any keys, passwords, tokens, usernames, or other secrets.
Are all UI and API inputs run through forms or serializers?
Are all external inputs validated and sanitized appropriately?
Does all branching logic have a default case?
Does this solution handle outliers and edge cases gracefully?
Are all external communications secured and restricted to SSL?
Documentation
Are changes to the UI documented in the platform docs? If this PR introduces new platform site functionality or changes existing ones, the changes must be documented in the Cadasta Platform Documentation.
Are changes to the API documented in the API docs? If this PR introduces new API functionality or changes existing ones, the changes must be documented in the API docs.
Are reusable components documented? If this PR introduces components that are relevant to other developers (for instance a mixin for a view or a generic form) they should be documented in the Wiki.