Upgrade to Django 1.5.x (then 1.6) so that Salesforce can post images to us

[stevenday]

Django 1.4 has this bug: https://code.djangoproject.com/ticket/19036 with decoding big base64 encoded files in multipart forms (it miscalculates the size of chunks and then claims they're invalid or something like that).

This is a problem because NHSD's Salesforce integration partners are limited to using the Salesforce Apex language, which can't send files in raw binary form very easily (or at all they claim) and must send them base64 encoded.

Therefore, we need to upgrade to get the fix.

Issues I can think of immediately:

• [ ] General package compatibility, at least one of our packages was pinned to a specific version for django 1.4 (django-concurrency I think) so this will need to change

But generally we need to review: https://docs.djangoproject.com/en/dev/releases/1.5/#backwards-incompatible-changes-in-1-5 and fix anything that comes up..

[BenJam]

note: this is in response to #1198.

[evdb]

Extracted from https://docs.djangoproject.com/en/dev/releases/1.5/

Backwards incompatible changes in 1.5

• [ ] ALLOWED_HOSTS required in production
• [ ] Managers on abstract models
• [ ] Context in year archive class-based views
• [ ] Context in year and month archive class-based views
• [ ] Context in TemplateView
• [ ] Non-form data in HTTP requests
• [ ] request_finished signal
• [ ] OPTIONS, PUT and DELETE requests in the test client
• [ ] System version of simplejson no longer used
• [ ] String types of hasher method parameters
• [ ] Validation of previous_page_number and next_page_number
• [ ] Behavior of autocommit database option on PostgreSQL changed
• [ ] Session not saved on 500 responses
• [ ] Email checks on failed admin login
• [ ] Changes in tests execution
• [ ] Database flushing in django.test.TransactionTestCase
• [ ] No more implicit DB sequences reset
• [ ] Ordering of tests
• [ ] cleaned_data dictionary kept for invalid forms
• [ ] Behavior of syncdb with multiple databases
• [ ] XML deserializer will not parse documents with a DTD
• [ ] Formsets default max_num
• [ ] Miscellaneous

Features deprecated in 1.5

• [ ] django.contrib.localflavor
• [ ] django.contrib.markup
• [ ] AUTH_PROFILE_MODULE
• [ ] Streaming behavior of HttpResponse
• [ ] django.utils.simplejson
• [ ] django.utils.encoding.StrAndUnicode
• [ ] django.utils.itercompat.product
• [ ] cleanup management command
• [ ] daily_cleanup.py script
• [ ] depth keyword argument in select_related

[evdb]

On hold pending confirmation that the image upload related fix is still required.

[stevenday]

https://github.com/mysociety/citizenconnect/tree/issues/1204-upgrade-django contains the work towards this. I've upgraded Django and other packages which required an upgrade to function (as determined by failing tests) and worked through the failing tests resolving incompatibilities one by one.

The current status is that there seems to be one outstanding issue with the api where posting values like:

form_data = {
    requires_second_tier_moderation: 0,
}

To the api endpoint used to evaluate as False, but now evaluate as True unless you actually specify False or leave it empty. This is a problem because our published api says to send 0 or 1, which used to work in Django 1.4

Perhaps we need to explicitly specify the field or widget type on the form, or explicitly coerce the values of any Booleans, I'm not sure.

After this, I think all the tests will pass, but we need to do a bit more manually UAT to make sure everything is still working as expected, and perhaps a manual review of every package in requirements.txt to make sure we have suitable versions for each (only upgrading if they're not stated as being compatible with 1.5).

[BenJam]

holding off on this for now although steve has taken an initial look at it. Seems that there may some refactoring necessary in the API provided to NHSD but ultimately should be on estimate for it.

[stevenday]

This is not required by the NHS, but it would be nice to do at some point. Also, there's 1.6 now! (Which does some things to Transactions that may cause us more hassle)