Django Backend Re-write

[davidswarbrick]

After the discussion of #49, many of the features this project may require going forwards such as

• An administrator/moderator login page,
• Fixed models for data stored in the project's database
• An understandable & well documented information flow from database to HTML
• Extendability to a REST framework

These are all already features of Django, or can be easily added through extension while retaining a well-defined information flow and representation. Although the Flask backend is coping well at the moment, its flexibility leads to the personal opinion of developers directing information flow rather than a defined framework, ultimately resulting in difficult-to-maintain code as only those who have already contributed fully understand the operations taking place.

Thus, this PR and associated branch aims to recreate the current functionality on the master branch using Django as a backend Python web framework, replacing Flask. To ease the transition the template engine is kept as Jinja2.

Before ready to merge into development, the following tasks must be completed:

• [X] Configure development server to serve static files
• [X] Jinja2 setup for Django
• [X] Model descriptions for EmailTemplates and other objects in models.py
• [X] Describing user inputs as Django Forms in forms.py
• [x] Re-formatting logic for filling out unfilled EmailTemplate objects
• [x] Re-formatting logic for parsing user-submitted email templates
• [x] /postcode/ endpoint for Javascript lookup of addresses
• [ ] Storing existing templates in Django database
• [ ] Configuring Heroku development deployment for a Django application, and investigating changes needed to master Heroku deployment

Additionally, some information flows could be streamlined during this re-write:

• [x] Rewrite landing page template so that the form is created dynamically from the Django Form object
• [x] Perform a single lookup of the user's postcode to get both addresses and constituency (if an API is available)
• [x] Investigate a single template-filling stage, is there a benefit to using render() for the email bodies themselves?

[davidswarbrick]

The force push was to rebase this branch on feature/db-admin-ui so that I could replicate that functionality in the Django swap/upgrade. Tests must be reconfigured for the Django layout.

[davidswarbrick]

The Django implementation of emaildrafter is now working:

1. A user is asked for their name and postcode in a similar form as the Flask implementation, but now it is automatically generated by Django's Form API thus some CSS hooks are missing/broken.
2. The /postcode/ endpoint now returns both the addresses and constituency of the postcode, which are inserted into relevant fields in the form, initially hidden to the user by the Javacript function.
3. The form is validated on submission, and then passed to a draft_templates() function as previously, however the templates are now stored and accessed as Django Model objects. The filling function either uses the specific target of the template, stored in a separate database table, or an MP which is looked up in its own database table using the constituency of the user as submitted in the initial form.
4. Template bodies are stored as TextFields in the database. To fill them, an EmailBody object is intialised (defined in emailbody.py), which extends the Python Template class to implement %TONAME, %YOURNAME, etc. shortcuts to information about the target and user. Also defined in emailbody.py is a validator to check submitted template bodies only contain fields which can be filled by this app. Between EmailTemplate in models.py and EmailBody in emailbody.py most of the functionality of emailtemplates.py is recreated, although there are some extra validation steps which may be useful and have not yet been recreated.
5. Filled templates are returned and displayed in results.html as before.

Updated To Do:

• [ ] Adjust CSS on landing page to account for different form definition
• [ ] Remove leading whitespace in text area widget for rendered email bodies in results.html
• [ ] Select only templates that are public and admin_approved for display on the homepage.
• [ ] Switch database to PostgreSQL - speed improvement future-proofs the project against REST APIs or scaling, other developers have shown a preference. MySQL has been used so far for ease of testing locally, but once a database has been deployed to staging or production I suspect the complexity increase will be equivalent for any choice of DB, thus it makes sense to go with a familiar option to other members of the team.
• [ ] Recreate /topic/ and /template/ endpoints.
• [ ] Rewrite MP scraping script to store MP objects in the new database.
• [ ] Remove Flask implementation code (currently still present on the branch as a reference).
• [ ] Rewrite tests.
• [ ] Handle 404 errors.
• [ ] Ensure settings are production ready.

Carried forward:

• [ ] Store existing templates in Django database
• [ ] Configure Heroku development deployment for a Django application, and investigating changes needed to master Heroku deployment

Identified extensions:

• [ ] Template submission from non-admins - Django's forms.ModelForm should ease this considerably.
• [ ] Moderator accounts with differing levels of access to the administrator interface.
• [ ] Selection box for user of the templates they wish to be filled out - could add a forms.ChoiceField() with the choices set to a query of EmailTemplate.objects()
• [ ] Responsive CSS, for example on the box surrounding user input.
• [ ] Additional tests such as on the Template regex functions.

Hopefully most of the complex & confusing bits of the Flask implementation have been addressed, so if anyone wishes to contribute please do! The relevant files to the Django implementation are in the emaildrafter and writeforchange subdirectories, corresponding to the Django App and Project respectively. Aside from emaildrafter/emailbody.py the app information flow follows the same structure as the Django tutorial - this file only implements the template filling functionality, which has been re-written to use Python's inbuilt template setup, so it should be far clearer than emailtemplates.py. All the external API calls have been grouped together in emaildrafter/externalapis.py. Feel free to post questions here or comment on specific pieces of code!