jonkiparsky
commented
8 years ago @bethlakshmi @BenignCremator If either of you have a chance to pick this up, please do so. I won't be able to look at it until ~7:00 tonight.
Closed jonkiparsky closed 8 years ago
jonkiparsky
commented
8 years ago @bethlakshmi @BenignCremator If either of you have a chance to pick this up, please do so. I won't be able to look at it until ~7:00 tonight.
BenignCremator
commented
8 years ago On 12/02/2015 02:16 PM, Jon Kiparsky wrote:
@bethlakshmi https://github.com/bethlakshmi @BenignCremator https://github.com/BenignCremator If either of you have a chance to pick this up, please do so. I won't be able to look at it until ~7:00 tonight.
All of the errors we have gotten since Jon assisted Shimmy LaRoux this morning were requests for event details from the scheduler where the scheduler could not find the eventItem by its id. Quick scan of the access logs and the error messages shows that all of the requests that have generated errors occur in four distinct batches, corresponding to specific users. The first occurred at approx 9:55 to 10:30 this morning, from an IP near Brattleboro VT, using Windows NT 6.1 and Chrome, for event Ids 369, 303, 316, 370, 371, and 380. The second batch occurred during approx 1:55 pm to 2:05 pm from an IP in Roxbury or Dorchester, using Mac OS X 10.9.5 and Safari, and was for event Ids 377, 378, 369, 370, and 3621. Access logs for this IP shows that it also searched for adjacent event Ids in some semblance of order, with event id 3621 being bracketed by 361 and 362 (neither of which generated errors), along with requests for a number of event ids in that numerical range. This is a strong indicator that this batch was someone hand scanning for events in the region of ids that were problematic, and 3621 was a typo. This would suggest that it was one of us researching the problem, but checking the IP addresses in use by Scratch and Jon resolved that it was not them. @bethlakshmi, can you check your IP address is not the one in these error messages? That last two blocks are overlapping in time, from about 2:15pm to now (3:20 pm), and come from a different IP address in Roxbury or Dorchester, and from my personal IP address. The one from my IP address was me trying to research the problem, and can be discounted. The last set was for event Ids 377, 3621, and 378, using Mac OS X 10.11.1 and Safari. This also appears to be another one of us researching the problem.
So if we discount the local ones, which could be one of us trying to recreate the problem, that leaves me focusing on Event Ids 369, 303, 316, 370, 371, and 380, of which, 369, 370, and 371 CURRENTLY WORK. They are all classes being taught by Sailor St. Claire. The website does not return the class logistics info with the details, so I am not able to tell if this is from GBE 9 or for GBE 10, and a reason why these work now is that there were scheduled since 10:30 this morning. This is not supported by the access logs, as only 371 has a change stage record associated with it, at about 9:40am EST (14:40 ZULU). So this means that at least some of these errors are intermittent, while the others could be for events that have been deleted and a user was scanning for events. This is supported for event 303 (requesting IP had preceding requests for 300, 301, 302, and followed by 304, all tech related), 316 (with events 311, 312, 313, 314, 318, 319, and 320, all movement classes), and a single block of requests for 369, 370, and 371 (block begins at 366, and goes straight through to 371), with each block having the requests within a min or two window. Using the admin interface shows that event ids 303, 316, and 380 simply do not exist, so I suspect that these are deleted or not yet created events, as event 371 is the highest numbered event the admin interface reports about.
So this leaves us with why are event ids 369, 370, and 371 periodically disappearing and then coming back later? Digging deeper....
burlexpo
commented
8 years ago The Dorchester IP is me -- at least, that was my activity (searching things in order). The reason I searched for event 3621 is because there is one -- at least, there's an error message for one:
Traceback (most recent call last):
File "/home/gbelive/webapps/gbelive/lib/python2.7/django/core/handlers/base.py", line 112, in get_response response = wrapped_callback(request, _callback_args, *_callback_kwargs)
File "/home/gbelive/webapps/gbelive/expo/scheduler/views.py", line 217, in detail_view eventitem_view = get_event_display_info(eventitem_id)
File "/home/gbelive/webapps/gbelive/expo/scheduler/views.py", line 158, in get_event_display_info item = EventItem.objects.get_subclass(eventitem_id=eventitem_id)
File "/home/gbelive/lib/python2.7/model_utils/managers.py", line 185, in get_subclass return self.get_queryset().get_subclass(_args, *_kwargs)
File "/home/gbelive/lib/python2.7/model_utils/managers.py", line 160, in get_subclass return self.select_subclasses().get(_args, *_kwargs)
File "/home/gbelive/webapps/gbelive/lib/python2.7/django/db/models/query.py", line 310, in get self.model._meta.object_name)
DoesNotExist: EventItem matching query does not exist.
<WSGIRequest path:/scheduler/details/3621, GET:<QueryDict: {}>, POST:<QueryDict: {}>, COOKIES:{'csrftoken': '9brmP0moceAl7jRpH91RfsINp0tShYvT', 'django_language': 'en-us', 'sessionid': '4aioz4b1d18wdegml1cjjgft6hn1ug6c'}, META:{u'CSRF_COOKIE': u'9brmP0moceAl7jRpH91RfsINp0tShYvT', 'DOCUMENT_ROOT': '/usr/local/apache2/htdocs', 'GATEWAY_INTERFACE': 'CGI/1.1', 'HTTPACCEPT': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/_;q=0.8', 'HTTP_ACCEPT_ENCODING': 'gzip, deflate, sdch', 'HTTP_ACCEPT_LANGUAGE': 'en-US,en;q=0.8', 'HTTP_CONNECTION': 'close', 'HTTP_COOKIE': 'django_language=en-us; sessionid=4aioz4b1d18wdegml1cjjgft6hn1ug6c; csrftoken=9brmP0moceAl7jRpH91RfsINp0tShYvT', 'HTTP_FORWARDED_REQUEST_URI': '/scheduler/details/3621', 'HTTP_HOST': 'burlesque-expo.com', 'HTTP_HTTPS': 'off', 'HTTP_HTTP_X_FORWARDED_PROTO': 'http', 'HTTP_UPGRADE_INSECURE_REQUESTS': '1', 'HTTP_USER_AGENT': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36', 'HTTP_X_FORWARDED_FOR': '73.16.153.57', 'HTTP_X_FORWARDED_HOST': 'burlesque-expo.com', 'HTTP_X_FORWARDED_PROTO': 'http', 'HTTP_X_FORWARDED_SERVER': 'burlesque-expo.com', 'HTTP_X_FORWARDED_SSL': 'off', 'PATH_INFO': u'/scheduler/details/3621', 'PATH_TRANSLATED': '/home/gbelive/webapps/gbelive/expo/expo/wsgi.py/scheduler/details/3621', 'QUERY_STRING': '', 'REMOTE_ADDR': '127.0.0.1', 'REMOTE_PORT': '36710', 'REQUEST_METHOD': 'GET', 'REQUEST_URI': '/scheduler/details/3621', 'SCRIPT_FILENAME': '/home/gbelive/webapps/gbelive/expo/expo/wsgi.py', 'SCRIPT_NAME': u'', 'SERVER_ADDR': '127.0.0.1', 'SERVER_ADMIN': '[no address given]', 'SERVER_NAME': 'burlesque-expo.com', 'SERVER_PORT': '80', 'SERVER_PROTOCOL': 'HTTP/1.0', 'SERVER_SIGNATURE': '', 'SERVER_SOFTWARE': 'Apache/2.2.25 (Unix) mod_wsgi/3.5 Python/2.7.9', 'mod_wsgi.application_group': 'web489.webfaction.com|', 'mod_wsgi.callable_object': 'application', 'mod_wsgi.enable_sendfile': '0', 'mod_wsgi.handler_script': '', 'mod_wsgi.input_chunked': '0', 'mod_wsgi.listener_host': '', 'mod_wsgi.listener_port': '14983', 'mod_wsgi.process_group': 'gbelive', 'mod_wsgi.queue_start': '1449086152115884', 'mod_wsgi.request_handler': 'wsgi-script', 'mod_wsgi.script_reloading': '1', 'mod_wsgi.version': (3, 5), 'wsgi.errors': <mod_wsgi.Log object at 0x7f1cdbd2ba70>, 'wsgi.file_wrapper': <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f1cdb7d64e0>, 'wsgi.input': <mod_wsgi.Input object at 0x7f1cdbd2bcf0>, 'wsgi.multiprocess': True, 'wsgi.multithread': True, 'wsgi.run_once': False, 'wsgi.url_scheme': 'http', 'wsgi.version': (1, 0)}>
On 12/2/15 5:37 PM, Hunter Heinlen wrote:
On 12/02/2015 02:16 PM, Jon Kiparsky wrote:
@bethlakshmi https://github.com/bethlakshmi @BenignCremator https://github.com/BenignCremator If either of you have a chance to pick this up, please do so. I won't be able to look at it until ~7:00 tonight.
All of the errors we have gotten since Jon assisted Shimmy LaRoux this morning were requests for event details from the scheduler where the scheduler could not find the eventItem by its id. Quick scan of the access logs and the error messages shows that all of the requests that have generated errors occur in four distinct batches, corresponding to specific users. The first occurred at approx 9:55 to 10:30 this morning, from an IP near Brattleboro VT, using Windows NT 6.1 and Chrome, for event Ids 369, 303, 316, 370, 371, and 380. The second batch occurred during approx 1:55 pm to 2:05 pm from an IP in Roxbury or Dorchester, using Mac OS X 10.9.5 and Safari, and was for event Ids 377, 378, 369, 370, and 3621. Access logs for this IP shows that it also searched for adjacent event Ids in some semblance of order, with event id 3621 being bracketed by 361 and 362 (neither of which generated errors), along with requests for a number of event ids in that numerical range. This is a strong indicator that this batch was someone hand scanning for events in the region of ids that were problematic, and 3621 was a typo. This would suggest that it was one of us researching the problem, but checking the IP addresses in use by Scratch and Jon resolved that it was not them. @bethlakshmi, can you check your IP address is not the one in these error messages? That last two blocks are overlapping in time, from about 2:15pm to now (3:20 pm), and come from a different IP address in Roxbury or Dorchester, and from my personal IP address. The one from my IP address was me trying to research the problem, and can be discounted. The last set was for event Ids 377, 3621, and 378, using Mac OS X 10.11.1 and Safari. This also appears to be another one of us researching the problem.
So if we discount the local ones, which could be one of us trying to recreate the problem, that leaves me focusing on Event Ids 369, 303, 316, 370, 371, and 380, of which, 369, 370, and 371 CURRENTLY WORK. They are all classes being taught by Sailor St. Claire. The website does not return the class logistics info with the details, so I am not able to tell if this is from GBE 9 or for GBE 10, and a reason why these work now is that there were scheduled since 10:30 this morning. This is not supported by the access logs, as only 371 has a change stage record associated with it, at about 9:40am EST (14:40 ZULU). So this means that at least some of these errors are intermittent, while the others could be for events that have been deleted and a user was scanning for events. This is supported for event 303 (requesting IP had preceding requests for 300, 301, 302, and followed by 304, all tech related), 316 (with events 311, 312, 313, 314, 318, 319, and 320, all movement classes), and a single block of requests for 369, 370, and 371 (block begins at 366, and goes straight through to 371), with each block having the requests within a min or two window. Using the admin interface shows that event ids 303, 316, and 380 simply do not exist, so I suspect that these are deleted or not yet created events, as event 371 is the highest numbered event the admin interface reports about.
So this leaves us with why are event ids 369, 370, and 371 periodically disappearing and then coming back later? Digging deeper....
— Reply to this email directly or view it on GitHub https://github.com/bethlakshmi/GBE2/issues/588#issuecomment-161456117.
The Great Burlesque Exposition of 2016 Feb. 5-7, 2016 Cambridge, MA www.Burlesque-Expo.com
jonkiparsky
commented
8 years ago A couple of notes: 1) There were at least some of these yesterday and last night. (see the ticket logged for this issue last night, which was before the Shimmy LaRoux issue) 2) The scraping hypothesis is interesting. 3) Since I see session ids, I'll be able to recover user ids for them later on this evening. 4) The hypothesis that is most interesting to me at present is "This is the result of some error in the codebase". If that can be confirmed or falsified, that would be a great help.
On Wed, Dec 2, 2015 at 5:37 PM, Hunter Heinlen notifications@github.com wrote:
On 12/02/2015 02:16 PM, Jon Kiparsky wrote:
@bethlakshmi https://github.com/bethlakshmi @BenignCremator https://github.com/BenignCremator If either of you have a chance to pick this up, please do so. I won't be able to look at it until ~7:00 tonight.
All of the errors we have gotten since Jon assisted Shimmy LaRoux this morning were requests for event details from the scheduler where the scheduler could not find the eventItem by its id. Quick scan of the access logs and the error messages shows that all of the requests that have generated errors occur in four distinct batches, corresponding to specific users. The first occurred at approx 9:55 to 10:30 this morning, from an IP near Brattleboro VT, using Windows NT 6.1 and Chrome, for event Ids 369, 303, 316, 370, 371, and 380. The second batch occurred during approx 1:55 pm to 2:05 pm from an IP in Roxbury or Dorchester, using Mac OS X 10.9.5 and Safari, and was for event Ids 377, 378, 369, 370, and 3621. Access logs for this IP shows that it also searched for adjacent event Ids in some semblance of order, with event id 3621 being bracketed by 361 and 362 (neither of which generated errors), along with requests for a number of event ids in that numerical range. This is a strong indicator that this batch was someone hand scanning for events in the region of ids that were problematic, and 3621 was a typo. This would suggest that it was one of us researching the problem, but checking the IP addresses in use by Scratch and Jon resolved that it was not them. @bethlakshmi, can you check your IP address is not the one in these error messages? That last two blocks are overlapping in time, from about 2:15pm to now (3:20 pm), and come from a different IP address in Roxbury or Dorchester, and from my personal IP address. The one from my IP address was me trying to research the problem, and can be discounted. The last set was for event Ids 377, 3621, and 378, using Mac OS X 10.11.1 and Safari. This also appears to be another one of us researching the problem.
So if we discount the local ones, which could be one of us trying to recreate the problem, that leaves me focusing on Event Ids 369, 303, 316, 370, 371, and 380, of which, 369, 370, and 371 CURRENTLY WORK. They are all classes being taught by Sailor St. Claire. The website does not return the class logistics info with the details, so I am not able to tell if this is from GBE 9 or for GBE 10, and a reason why these work now is that there were scheduled since 10:30 this morning. This is not supported by the access logs, as only 371 has a change stage record associated with it, at about 9:40am EST (14:40 ZULU). So this means that at least some of these errors are intermittent, while the others could be for events that have been deleted and a user was scanning for events. This is supported for event 303 (requesting IP had preceding requests for 300, 301, 302, and followed by 304, all tech related), 316 (with events 311, 312, 313, 314, 318, 319, and 320, all movement classes), and a single block of requests for 369, 370, and 371 (block begins at 366, and goes straight through to 371), with each block having the requests within a min or two window. Using the admin interface shows that event ids 303, 316, and 380 simply do not exist, so I suspect that these are deleted or not yet created events, as event 371 is the highest numbered event the admin interface reports about.
So this leaves us with why are event ids 369, 370, and 371 periodically disappearing and then coming back later? Digging deeper....
— Reply to this email directly or view it on GitHub https://github.com/bethlakshmi/GBE2/issues/588#issuecomment-161456117.
bethlakshmi
commented
8 years ago I'm am just getting into this now. I have been trying to finish the costume bid stuff in every spare moment, since my goal was to at least get 1 pull request on it in by Dec. 5.
I'm going to dig in and if there's any thing I can contribute to the knowledgebase before babbling along half-assed.
---Betty
On Wed, Dec 2, 2015 at 6:02 PM, Jon Kiparsky notifications@github.com wrote:
A couple of notes: 1) There were at least some of these yesterday and last night. (see the ticket logged for this issue last night, which was before the Shimmy LaRoux issue) 2) The scraping hypothesis is interesting. 3) Since I see session ids, I'll be able to recover user ids for them later on this evening. 4) The hypothesis that is most interesting to me at present is "This is the result of some error in the codebase". If that can be confirmed or falsified, that would be a great help.
On Wed, Dec 2, 2015 at 5:37 PM, Hunter Heinlen notifications@github.com
wrote:
On 12/02/2015 02:16 PM, Jon Kiparsky wrote:
@bethlakshmi https://github.com/bethlakshmi @BenignCremator https://github.com/BenignCremator If either of you have a chance to pick this up, please do so. I won't be able to look at it until ~7:00 tonight.
All of the errors we have gotten since Jon assisted Shimmy LaRoux this morning were requests for event details from the scheduler where the scheduler could not find the eventItem by its id. Quick scan of the access logs and the error messages shows that all of the requests that have generated errors occur in four distinct batches, corresponding to specific users. The first occurred at approx 9:55 to 10:30 this morning, from an IP near Brattleboro VT, using Windows NT 6.1 and Chrome, for event Ids 369, 303, 316, 370, 371, and 380. The second batch occurred during approx 1:55 pm to 2:05 pm from an IP in Roxbury or Dorchester, using Mac OS X 10.9.5 and Safari, and was for event Ids 377, 378, 369, 370, and 3621. Access logs for this IP shows that it also searched for adjacent event Ids in some semblance of order, with event id 3621 being bracketed by 361 and 362 (neither of which generated errors), along with requests for a number of event ids in that numerical range. This is a strong indicator that this batch was someone hand scanning for events in the region of ids that were problematic, and 3621 was a typo. This would suggest that it was one of us researching the problem, but checking the IP addresses in use by Scratch and Jon resolved that it was not them. @bethlakshmi, can you check your IP address is not the one in these error messages? That last two blocks are overlapping in time, from about 2:15pm to now (3:20 pm), and come from a different IP address in Roxbury or Dorchester, and from my personal IP address. The one from my IP address was me trying to research the problem, and can be discounted. The last set was for event Ids 377, 3621, and 378, using Mac OS X 10.11.1 and Safari. This also appears to be another one of us researching the problem.
So if we discount the local ones, which could be one of us trying to recreate the problem, that leaves me focusing on Event Ids 369, 303, 316, 370, 371, and 380, of which, 369, 370, and 371 CURRENTLY WORK. They are all classes being taught by Sailor St. Claire. The website does not return the class logistics info with the details, so I am not able to tell if this is from GBE 9 or for GBE 10, and a reason why these work now is that there were scheduled since 10:30 this morning. This is not supported by the access logs, as only 371 has a change stage record associated with it, at about 9:40am EST (14:40 ZULU). So this means that at least some of these errors are intermittent, while the others could be for events that have been deleted and a user was scanning for events. This is supported for event 303 (requesting IP had preceding requests for 300, 301, 302, and followed by 304, all tech related), 316 (with events 311, 312, 313, 314, 318, 319, and 320, all movement classes), and a single block of requests for 369, 370, and 371 (block begins at 366, and goes straight through to 371), with each block having the requests within a min or two window. Using the admin interface shows that event ids 303, 316, and 380 simply do not exist, so I suspect that these are deleted or not yet created events, as event 371 is the highest numbered event the admin interface reports about.
So this leaves us with why are event ids 369, 370, and 371 periodically disappearing and then coming back later? Digging deeper....
— Reply to this email directly or view it on GitHub https://github.com/bethlakshmi/GBE2/issues/588#issuecomment-161456117.
— Reply to this email directly or view it on GitHub https://github.com/bethlakshmi/GBE2/issues/588#issuecomment-161461417.
jonkiparsky
commented
8 years ago @bethlakshmi Please be careful about posting server passwords to the issues tracker.
bethlakshmi
commented
8 years ago Fuck. Sorry.
On Wed, Dec 2, 2015 at 11:16 PM, Jon Kiparsky notifications@github.com wrote:
@bethlakshmi https://github.com/bethlakshmi Please be careful about posting server passwords to the issues tracker.
— Reply to this email directly or view it on GitHub https://github.com/bethlakshmi/GBE2/issues/588#issuecomment-161510282.
bethlakshmi
commented
8 years ago So... some interesting research:
Now... our mystery cases that actually DO exist;
I'm getting this info from the Classes object, which as a Biddable has a created at and updated at field that are done automatically on the model level. These are definitely the last three ids in the system.
Summary:
Thoughts:
BenignCremator
commented
8 years ago While waiting on 'other things' to occur (coverage tests take time), I scanned the site for URLs using the bogus event ids that have been generating the errors, and found none. While this is not exhaustive, I am confident that the errors are not generated by people following a link on the site. Recommended course of action is to (eventually) create a better error page for that particular error.
bethlakshmi
commented
8 years ago Read the code and verified on live site - this specific error was fixed. We are still stamping out others.
User-facing error. Can determine user's identity if I can get on to the live server, but need connection info. (I believe I have the correct password, but not username/server)