Update Django to 1.5

[btbonval]

It looks like Django has moved on to developing 1.6.x presently.

According to reqs/common.txt, we're running the largest 1.4.x that can be found.

Since 1.5 is no longer in development, it is worth attempting to upgrade to 1.5.x from 1.4.x. This upgrade would probably fix #89 although who knows how many problems it would create.

That's what VMs are good for :)

[btbonval]

https://docs.djangoproject.com/en/dev/releases/1.5/

Good things (selected for relevance to our wants/needs):

• User model from django.contrib.auth is more flexible (we might want to use this with KarmaUser)
• requires Python 2.6.5 but recommends Python 2.7.3 (beta and prod both run 2.7.3)
• "The dumpdata management command outputs one row at a time, preventing out-of-memory errors when dumping large datasets." #68
• "In the admin, you can now filter users by groups which they are members of." (looks like Django has built-in RBAC that we should be using, and this version makes it a little easier to manage)
• multi column index support (this might be useful for speeding up some database calls, like adding an index on all unique columns ... some of which are unique over multiple columns)

Bad things (all of them, ! marking indicates we might care):

• ! ALLOWED_HOSTS is a required field in production systems, and we presently do not implement it.
• Managers on abstract classes have changed, but after last night's work, I can confirm we do not have any managers on abstract classes.
• YearArchiveView changed year type, but I found no instances of this in our code.
• YearArchiveView and MonthArchiveView changed sort order, but I found no instances of either in our code.
• ! TemplateView uses URLconf instead of params for context. We use TemplateView for the about page. I don't understand the implications, but it might mean we need to modify this line: https://github.com/FinalsClub/karmaworld/blob/dab4a237910ee521e5889c31925957e5660033f6/karmaworld/apps/courses/views.py#L85
• request_finished signal behavior changed, but I found no instances in our code.
• OPTIONS, PUT and DELETE behavior changed for the "test client." We definitely use Client from django.test, but we only ever POST with it, so this should be fine.
• Django's internals moved from external simplejson to built-in json module. This should only affect Django JSON Serialization, and since we only just started making use of that, should not be an issue.
• Change to custom password hashers, which we don't use.
• Change to object pagination, but I found no instances of Paginator in our code.
• postgresql autocommit fixed. It looks like we are not using autocommit.
• sessions are not saved on error 500, which is probably fine because we aren't doing much with sessions upon internal server error.
• cosmetic modification to admin page authentication which isn't as alarming as it was.
• TransactionTestCase changed a whole bunch, but I didn't find any instances in our code.
• test ordering changed amidst different test flavors, but we only use TestCase so it shouldn't change our tests.
• ! cleaned_data always returns now, but can return partial data. It looks like we never check cleaned_data for validity, rather we do the right thing and check is_valid, so this should be fine. We should double check.
• syncdb changed with multiple databases. We only have one database.
• XML deserializer changed, but we will be using JSON serialization/deserialization.
• formsets default to 1000 max instead of no limit. This is probably fine, it's about managing Django attacks and security, which we haven't messed with.
• ModelMultipleChoiceField changed but I didn't find it in our code.
• int_to_base36 changed but I didn't find it in our code.
• ! slugify is now available through django.utils.text.slugify(). We use this, but it isn't clear whether our current use of defaultfilters.slugify() is no longer allowed. We would probably want to update the code to use the new location.
• default FILE_UPLOAD_PERMISSIONS changed, but we don't use Django for file uploads anymore. Basically it removes the execute bit from default, which won't change anything anyway, since we only uploaded non-executables
• ! F expressions changed, but I couldn't find any. I did a search for " F(" which didn't show up, but maybe we are using them and I didn't search for them well enough.
• CRSF templates changed, although it should only cause problems for certain DTDs (those prior to HTML5 Strict)
• adminmedia removed, but I can't find any instances in our code.
• django.contrib.redirects may not be used without django.contrib.sites, but we are using sites so it doesn't matter.
• BoundField changed, but I don't find any instances in our code.
• select_related() behavior changed on error, but I find no instances in our code.

[btbonval]

Based on my review, I recommend we upgrade. The incompatibilities appear easy to fix and it adds two features we want: improved Serialization support and enhanced User/Group management stuff that we should be using for KarmaUser and so forth.

[AndrewMagliozzi]

I'm game to try.

On Jan 9, 2014, at 1:39 PM, Bryan Bonvallet notifications@github.com wrote:

It looks like Django has moved on to developing 1.6.x presently.

According to reqs/common.txt, we're running the largest 1.4.x that can be found.

Since 1.5 is no longer in development, it is worth attempting to upgrade to 1.5.x from 1.4.x. This upgrade would probably fix #89 although who knows how many problems it would create.

That's what VMs are good for :)

— Reply to this email directly or view it on GitHub.

[btbonval]

Naive attempt to simply run with Django 1.5.5 yields 500 error, and just as I'm about to start debugging, I'm being asked to go outside and play. I'll look into this more later.

[btbonval]

Ran into a problem with one View written for Django 1.2 that just happened to still work in Django 1.3 and 1.4 but broke in Django 1.5. https://github.com/FinalsClub/karmaworld/blob/master/karmaworld/urls.py#L9 https://github.com/FinalsClub/karmaworld/blob/master/karmaworld/urls.py#L62

Need to replace direct_to_template with TemplateView: https://docs.djangoproject.com/en/1.4/topics/generic-views-migration/ https://docs.djangoproject.com/en/1.4/ref/class-based-views/#django.views.generic.base.TemplateView

[btbonval]

SuspiciousOperation: Invalid HTTP_HOST header (you may need to set ALLOWED_HOSTS): 127.0.0.1:8000

Added ALLOWED_HOSTS and now the system is running without 500 errors.

Terms page (modified in the previous comment) displays just as on current Karmanotes.org.

About page failed to load, which is probably related to one of the above bullet points noting this line: https://github.com/FinalsClub/karmaworld/blob/dab4a237910ee521e5889c31925957e5660033f6/karmaworld/apps/courses/views.py#L85

Adding a course worked fine with sluggify, so that bullet point wasn't a problem. We should still update the slugify calls.

[btbonval]

woops the About page 404 was because it forwarded http://localhost/about/ and not http://localhost:6659/about as required. Different problem.

But actually, I modified the AboutTemplate to remove the params attribute (which simply took in kwargs) and instead inserted schools into kwargs and returned that as the context. This is the correct pattern for the new TemplateView, although I think we don't use kwargs so it made no real difference for now.

[btbonval]

Rewrote slugify for the new import location. Added a course and slugify still worked.

Attempted to run unit tests, but I can't parse the results.

Ran 557 tests in 20.370s

FAILED (failures=2, errors=47, skipped=1, expected failures=1)

For now, I'll push these changes to django155 branch, with HEAD at commit 119237d311f0e82900840127cadab52e32929ce0

[btbonval]

Although it's bad form to rebase a pushed commit, fair notice: I'm rebasing the django155 branch. I don't think anyone else has touched it or plans on touching it. I pushed it in case my computer was wiped in a nuclear holocaust.

On my VM, I rebased master into django155 and everything looks good at high level. Could view PDFs, add a course, upload a text file, delete notes and courses from the Moderator panel.

Looks good enough to push to master.

[charlesconnell]

I've committed to that branch! But I don't care, I'll just re-clone.

• Charles

On Tue 14 Jan 2014 04:50:41 PM EST, Bryan Bonvallet wrote:

Although it's bad form to rebase a pushed commit, fair notice: I'm rebasing the django155 branch. I don't think anyone else has touched it or plans on touching it. I pushed it in case my computer was wiped in a nuclear holocaust.

On my VM, I rebased master into django155 and everything looks good at high level. Could view PDFs, add a course, upload a text file, delete notes and courses from the Moderator panel.

Looks good enough to push to master.

— Reply to this email directly or view it on GitHub https://github.com/FinalsClub/karmaworld/issues/263#issuecomment-32311290.

[btbonval]

Wait you commited to django155?

I haven't merged to master or pushed or anything. I guess I'll drop working space changes and switch to merge.

[btbonval]

I tend to delete branches after I merge them, good thing you said something. Let me double check that I even have your commit.

[btbonval]

oh you pushed a commit while I was working on the merge into master. This should be easy to fast forward in.

Yeah I'm going to delete django155 once the feature is completed and fully merged into master, so you won't have to worry about cloning or anything. That branch will be vapor. I don't know why so many people leave branches that are 5 years old and fully merged into master.

[btbonval]

Nuked my local rebase since other people were working with django155 branch. I will do a proper double merge which is ugly as a mofo, but much more proper: git checkout django155; git merge master; git checkout master; git merge django155; git branch -d django155

Trashed my dev VM and starting scratch with master merged into django155. Will run some hand tests and run the auto tests.

[btbonval]

Finally got testing working, but there have been a number of commits and tomfoolery. Nuking VM and starting all over with the current django155.

[btbonval]

Fresh VM. django155 branch. selenium tests passed. created a course, flagged the course, uploaded a PDF to the course, viewed PDF render, deleted one note from moderator page, reset flags on course from moderator page.

PDF was uploaded twice. illustrated two problems:

• we should disable that button prior to doing any additional work to prevent double clicks
• unique on (fp_file, upstream_link) doesn't work when upstream_link is null. fp_file might need its own unique constraint.

[btbonval]

django155 branch merged into master. branch deleted (the name at least, all commits are in the tree by way of master).

[btbonval]

backed up /var/www/karmaworld/venv to /home/djkarma/venv.django149 on beta.

created a new virtualenv. ran fab -H 127.0.0.1 deploy to bring virtualenv and database up to date. ... takes awhile.

[btbonval]

failed.

    "Invalid HTTP_HOST header (you may need to set ALLOWED_HOSTS): %s" % host)
SuspiciousOperation: Invalid HTTP_HOST header (you may need to set ALLOWED_HOSTS): beta.karmanotes.org

welp its right there: https://github.com/FinalsClub/karmaworld/blob/90775067e14ea4017e2702a66b933a1a8365826e/karmaworld/settings/common.py#L27

maybe no period? period is supposed to indicated FQDN, but I don't care. time for guess and check.

[btbonval]

yup. removed period and the site loads. need to fix that for all the domains in that list.

[btbonval]

I did not do exhaustive testing of beta site. Leave that for the interns on Thursday.

Okay, besides production, it looks like Django 1.5 is officially supported and running.

Closing ticket and emitting woopwoops from my meatbag. Sorry internets, you can't hear my meatbag because you're not in meatspace.