search

tbpmig / mig-website

Apache License 2.0
8 stars 5 forks source link

Website periodically borkes all over the place #206

Open mikehand opened 8 years ago

mikehand commented 8 years ago

Seems to be (possibly) related to the event_list page

Internal Server Error: /calendar/event_list/
Traceback (most recent call last):
  File "/home/webdev/.virtualenvs/migweb/lib/python2.7/site-packages/django/core/handlers/base.py", line 111, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/srv/www/event_cal/views.py", line 1227, in list
    for event in events.order_by('earliest_start'):
  File "/home/webdev/.virtualenvs/migweb/lib/python2.7/site-packages/django/db/models/query.py", line 141, in __iter__
    self._fetch_all()
  File "/home/webdev/.virtualenvs/migweb/lib/python2.7/site-packages/django/db/models/query.py", line 966, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/home/webdev/.virtualenvs/migweb/lib/python2.7/site-packages/django/db/models/query.py", line 265, in iterator
    for row in compiler.results_iter():
  File "/home/webdev/.virtualenvs/migweb/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 700, in results_iter
    for rows in self.execute_sql(MULTI):
  File "/home/webdev/.virtualenvs/migweb/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 786, in execute_sql
    cursor.execute(sql, params)
  File "/home/webdev/.virtualenvs/migweb/lib/python2.7/site-packages/django/db/backends/utils.py", line 65, in execute
    return self.cursor.execute(sql, params)
  File "/home/webdev/.virtualenvs/migweb/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/home/webdev/.virtualenvs/migweb/lib/python2.7/site-packages/django/db/backends/utils.py", line 65, in execute
    return self.cursor.execute(sql, params)
OperationalError: stack depth limit exceeded
HINT:  Increase the configuration parameter "max_stack_depth" (currently 2048kB), after ensuring the platform's stack depth limit is adequate.
mikehand commented 8 years ago

cc: @mrealm, @dadrian

dadrian commented 8 years ago

It looks like a SQL query is so complex / accidentally recursive, that the query compilation process recursively calls itself enough that the program crashes. Do you run some sort of scheduled calendar update, or do you have some sort of infinite retry process if something go wrong?

kylelady commented 8 years ago

ASSIGN #206 TO MY BUTT

On Sunday, February 7, 2016, David Adrian notifications@github.com wrote:

It looks like a SQL query is so complex / accidentally recursive, that the query compilation process recursively calls itself enough that the program crashes. Do you run some sort of scheduled calendar update, or do you have some sort of infinite retry process if something go wrong?

— Reply to this email directly or view it on GitHub https://github.com/tbpmig/mig-website/issues/206#issuecomment-180946909.

mikehand commented 8 years ago

So it's not a scheduled thing (AFAICT) because the source of the error is from a view function...

I'm not sure about the retry process, is that a db config thing?

dadrian commented 8 years ago

I mean, do you hit Google Calendar when you load this page?

mikehand commented 8 years ago

No that's just the main cal page and event (un)sign up logic

On Sunday, February 7, 2016, David Adrian notifications@github.com wrote:

I mean, do you hit Google Calendar when you load this page?

— Reply to this email directly or view it on GitHub https://github.com/tbpmig/mig-website/issues/206#issuecomment-181056220.

Michael Hand Electrical Engineering/Control Systems M.S.E. The University of Michigan mikehand@umich.edu

dadrian commented 8 years ago

Do you have any DB access/query logs?

dadrian commented 8 years ago

Also, which file is this source in?

mikehand commented 8 years ago

Source is in event_cal/views.py function name event_list, not sure that will be much help as the offending call is likely buried in django. Kyle would need to pull logs

Michael Hand Electrical Engineering/Control Systems M.S.E. The University of Michigan mikehand@umich.edu

On Mon, Feb 8, 2016 at 8:01 PM, David Adrian notifications@github.com wrote:

Also, which file is this source in?

— Reply to this email directly or view it on GitHub https://github.com/tbpmig/mig-website/issues/206#issuecomment-181650333.

dadrian commented 8 years ago

Well, if you post to the event_list page, you execute no less than 11 joins. I wouldn't be surprised if this page always crashes when you POST to it in production, and it's just that hardly anyone ever causes a POST. See https://github.com/tbpmig/mig-website/blob/master/event_cal/views.py#L1169

dadrian commented 8 years ago

Essentially, everytime you access one of those subfields, you're incurring at least one JOIN.

dadrian commented 8 years ago

Yup. This filter will crash it reliably.

screen shot 2016-02-08 at 10 32 18 pm

mikehand commented 8 years ago

Ah ok, I had to date not been able to get a query to crash it. Any suggestions on how to go about doing that kind of complex querying in a manner that doesn't borke the whole thing?

Michael Hand Electrical Engineering/Control Systems M.S.E. The University of Michigan mikehand@umich.edu

On Mon, Feb 8, 2016 at 10:33 PM, David Adrian notifications@github.com wrote:

Yup. This filter will crash it reliably.

[image: screen shot 2016-02-08 at 10 32 18 pm] https://cloud.githubusercontent.com/assets/1158264/12906977/f4f96d9e-ceb3-11e5-9f71-c1afaaa7099a.png

— Reply to this email directly or view it on GitHub https://github.com/tbpmig/mig-website/issues/206#issuecomment-181690666.

dadrian commented 8 years ago

Filter by shifts, then group by events, then join. Essentially, do what you're doing but in reverse.

Effectively, this page is also using up all the memory on the server as well (the database may also be contributing while processing this query). See the latest error message.

Internal Server Error: /calendar/event_list/
Traceback (most recent call last):
  File "/home/webdev/.virtualenvs/migweb/lib/python2.7/site-packages/django/core/handlers/base.py", line 111, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/srv/www/event_cal/views.py", line 1209, in list
    query_event_type = category.get_children(query_event_type)
  File "/srv/www/requirements/models.py", line 202, in get_children
    query |= child.get_children(query)
  File "/home/webdev/.virtualenvs/migweb/lib/python2.7/site-packages/django/db/models/query_utils.py", line 58, in __or__
    return self._combine(other, self.OR)
  File "/home/webdev/.virtualenvs/migweb/lib/python2.7/site-packages/django/db/models/query_utils.py", line 53, in _combine
    obj.add(self, conn)
  File "/home/webdev/.virtualenvs/migweb/lib/python2.7/site-packages/django/utils/tree.py", line 119, in add
    self.children.extend(data.children)
MemoryError
dadrian commented 8 years ago

Also, FWIW, the UI on the event_list page is awful.

mikehand commented 8 years ago

So @dadrian assuming the queries are rewritten as appropriate. This should fix it?

shifts = EventShift.objects.filter(query_date & query_location & q_can_attend)  
events = CalendarEvent.objects.filter(  
                    query_members &  
                    query_event_type  
)  
events = events.filter(eventshift__in=shifts).distinct()
mikehand commented 8 years ago

Oh wait, it's actually the query_event_typethat's causing it to crash, even if the other sorting has happened. Assumedly because of how it checks A | B | C | D | etc.

dadrian commented 8 years ago

All of the OR statements require a JOIN, because you're looking into the eventshift itself, rather than the event. Is there someway you could select all eventshifts, and then filter those?

mikehand commented 8 years ago

So the view actually looks through events rather than shifts. The event_categoryattribute is on the CalendarEvent objects. The problematic part is that I'm trying to find events that will satisfy any one of those event types. Doing all the advance filtering on shifts and other things doesn't cause issues.

There must be a clean, efficient way of doing this. It would seem that simply looping through all the candidate pre-filtered events and checking if it satisfies any of those would actually work without issue

dadrian commented 8 years ago

Yes, the view looks through events, but every time you filter on eventshift__some_property you're doing a join of events against event shifts.

mikehand commented 8 years ago

I removed the filtering across shift relationships (not yet committed) and it still is crashing

The issue seems to be the events.filter(event_type=(A OR B OR C OR ...)) part.

kylelady commented 8 years ago

Send me a link to the relevant code? I'll try to DBA this shit.

mikehand commented 8 years ago

It's in event_cal/views.py

The list function

On Monday, February 15, 2016, Kyle Lady notifications@github.com wrote:

Send me a link to the relevant code? I'll try to DBA this shit.

— Reply to this email directly or view it on GitHub https://github.com/tbpmig/mig-website/issues/206#issuecomment-184217706.

Michael Hand Electrical Engineering/Control Systems M.S.E. The University of Michigan mikehand@umich.edu

dadrian commented 8 years ago

@kylelady https://github.com/tbpmig/mig-website/blob/master/event_cal/views.py#L1169

kylelady commented 8 years ago

https://github.com/tbpmig/mig-website/blob/da6da9fbb2d89e58d51bfc5898df7129d890325c/event_cal/views.py#L518-L558

mikehand commented 8 years ago

Dude that's not even the right view. Sure it's a tragedy against basic coding, but it works, and it doesn't cause performance issues at the moment.

Michael Hand Electrical Engineering/Control Systems M.S.E. The University of Michigan mikehand@umich.edu

On Wed, Feb 17, 2016 at 7:12 AM, Kyle Lady notifications@github.com wrote:

https://github.com/tbpmig/mig-website/blob/da6da9fbb2d89e58d51bfc5898df7129d890325c/event_cal/views.py#L518-L558

https://camo.githubusercontent.com/34d29b7153667599d1c837189a7aad5dba950698/687474703a2f2f69322e6b796d2d63646e2e636f6d2f70686f746f732f696d616765732f6f726967696e616c2f3030302f3231302f3430332f4f68437261702e706e67

— Reply to this email directly or view it on GitHub https://github.com/tbpmig/mig-website/issues/206#issuecomment-185174059.

kylelady commented 8 years ago

I know.

dadrian commented 8 years ago

Sure it's a tragedy against basic coding

This entire website is a tragedy against basic coding principles

mikehand commented 8 years ago

Yeah but it's my tragedy against basic coding

On Wednesday, February 17, 2016, David Adrian notifications@github.com wrote:

Sure it's a tragedy against basic coding

This entire website is a tragedy against basic coding principles

— Reply to this email directly or view it on GitHub https://github.com/tbpmig/mig-website/issues/206#issuecomment-185524584.

Michael Hand Electrical Engineering/Control Systems M.S.E. The University of Michigan mikehand@umich.edu