Fix the scoreboard on the about page

[AndrewMagliozzi]

https://www.karmanotes.org/about/

[btbonval]

@AndrewMagliozzi what's broken? Looks correct to me.

[AndrewMagliozzi]

It is not tracking notes as they are added.

[AndrewMagliozzi]

search brown on the homepage... there are more than a hundred notes but the scoreboard only shows a count of 24. It's not urgent, but let's fix this in January.

[btbonval]

Looks like this is related to #218.

I had written a dynamic solution which does not cache, but tallies up every page load. The cached solution was accepted instead.

So we should run the cache update management command. python manage.py fix_note_counts, which would need to be done as a one-off Heroku worker just one time. https://devcenter.heroku.com/articles/one-off-dynos https://github.com/FinalsClub/karmaworld/blob/c3bc08a38663ebb246b87f95fad871180f6dc298/karmaworld/apps/courses/management/commands/fix_note_counts.py

For long term support, we might want to build this into beat to run once every 24 hours, then we don't have to worry about one-offs and it'll stay "in sync" with 24 hour granularity.

[btbonval]

To schedule a beat task, we'd want to update prod.py with the relevant task: https://github.com/FinalsClub/karmaworld/blob/3fb9188be79556bd351e3659a341c9aeeb6b4926/karmaworld/settings/prod.py#L84-L93

schedule: timedelta(days=1)

[btbonval]

heroku run manage.py fix_note_counts appeared to work for beta without incident. Running on production to ensure a result before automating.

[btbonval]

Hrm, that did not work. Brown is still listed with only 24 notes throughout all courses after running the fix_note_counts process. The leaderboard did change some numbers around, but not Brown's.

It looks like the about page is detailing the cached file_count for the school. https://github.com/FinalsClub/karmaworld/blob/3fb9188be79556bd351e3659a341c9aeeb6b4926/karmaworld/templates/about.html#L104-L116

fix_note_counts calls school.update_note_count() which might be the problem somehow, as it updates school.file_count: https://github.com/FinalsClub/karmaworld/blob/master/karmaworld/apps/courses/models.py#L91-L96

Perhaps it is worth creating a section for notes and courses.

[btbonval]

Only one course is being returned for Brown in the Django shell on production.

>>> School.objects.get(name='Brown University')
<School: Brown University>
>>> school = School.objects.get(name='Brown University')
>>> courses = [course.file_count for course in school.course_set.all()]
>>> courses
[24]
>>> courses = [course.name for course in school.course_set.all()]
>>> courses
[u'Management of Industrial and Nonprofit Organizations']
>>> len(courses)
1

>>> school2 = School.objects.get(name='Massachusetts Institute of Technology')
>>> school2
<School: Massachusetts Institute of Technology>
courses = [course.name for course in school2.course_set.all()]
>>> len(courses)
212

so course_set is not working for Brown University correctly. I wonder if there are two Brown Universities in the database??

[btbonval]

So here is one course, specifically chosen for its number of files (> 24) which was found to be associated with Brown University backwards (course -> department -> school).

I wonder if the school count does not take department into account properly when searching for courses.

>>> Course.objects.get(name='ECON 110: Introduction to Economics')
<Course: Course ECON 110: Introduction to Economics in Economics>
>>> test_course = Course.objects.get(name='ECON 110: Introduction to Economics')
>>> test_course.file_count
38
>>> test_course.department
<Department: Economics>
>>> test_course.department.school
<School: Brown University>
>>> test_course.department.school.id
2
>>> school
<School: Brown University>
>>> school.id
2

[btbonval]

Appears to only aggregate courses added without departments.

>>> school.course_set.all()[0]
<Course: Course Management of Industrial and Nonprofit Organizations in None>
>>> school.course_set.all()[0].department
>>> for department in school.department_set.all():
...   for course in department.course_set.all():
...       print course.name
... 
Introduction to Cognitive Neuroscience
Perspectives on Social Interaction: Introduction to Social Psychology
Sociology 1010: Classical Sociological Theory
ECON 110: Introduction to Economics
Arch 150: Introduction to Egyptian Archaeology and Art
Math 100: Introductory Calculus Part II
Math 90: Introductory Calculus, Part I
URBN 210 - The City: Introduction to Urban Studies
Chem 0330
Introduction to Cultural Anthropology
Anthro 110: Anthropology and Global Social Problems
History 1900: American Empire Since 1890
Pols1550 : War and Politics
Pols 400: Intro to International Relations

[btbonval]

Possible fix:

# find all courses without a department
course_list = list(self.course_set.all())
# find all courses with a department
for department in self.department_set.all():
  for course in department.course_set.all():
    course_list.append(course)
self.file_count = sum([course.file_count for course in course_list])
self.save()

[btbonval]

Got that commit pushed to beta (finally). Nothing appears to have broken, and the score board did get updated. Harvard got an additional 15 notes. I surmise that Harvard has 15 notes across courses in departments on Beta.

Confirmed.

>>> i = 0
>>> for department in school.department_set.all():
...   for course in department.course_set.all():
...     i = i + course.file_count
... 
>>> print i
15

This looks like a solution worth pushing to master!

[AndrewMagliozzi]

Thanks Bryan!

On Jan 9, 2015, at 7:10 PM, Bryan Bonvallet notifications@github.com wrote:

Got that commit pushed to beta (finally). Nothing appears to have broken, and the score board did get updated. Harvard got an additional 15 notes. I surmise that Harvard has 15 notes across courses in departments on Beta.

Confirmed.

i = 0 for department in school.department_set.all(): ... for course in department.course_set.all(): ... i = i + course.file_count ... print i 15 This looks like a solution worth pushing to master!

— Reply to this email directly or view it on GitHub.

[btbonval]

Needs to be tested on Beta as a beat task, repeating periodically. Beta is not playing nicely for now.

[btbonval]

Looks like beta is playing more nicely. I uploaded a branch which schedules board updates every 5 minutes. After about 10 minutes, an update occurred and a new school was added to the scoreboard.

I have to wait on the Beta system to catch up with its backlog before I can adequately test that these numbers are updating in a sensible way: add a note, see the school's count go up.

[btbonval]

I set the scoreboard update to 5 minutes, and like clockwork, every five minutes that scoreboard has updated a new school. I assume as the backlog is chugging through, new notes are being generated and added. If I see an old school's note count get updated, I'll consider that good enough. So far, it seems to be entirely new schools.

[btbonval]

Sweet! Backlog finished. I uploaded a brand new note and saw the note count increase by one for the respective school. Pushing fix to master.

[AndrewMagliozzi]

Sweet!

On Jan 24, 2015, at 3:24 AM, Bryan Bonvallet notifications@github.com wrote:

Closed #377.

— Reply to this email directly or view it on GitHub.