search

Rjaylyn / search-reporting

Automatically exported from code.google.com/p/search-reporting
0 stars 0 forks source link

Please verify if my environment was set up correctly - asr.py crashes after 1 hour #1

Open GoogleCodeExporter opened 8 years ago

GoogleCodeExporter commented 8 years ago 
What steps will reproduce the problem?
1. I don't have steps to reproduce the error at this time
2.
3.

What is the expected output? What do you see instead?
We put the ASR tool into our production site and the asr.py process crashes
after about 1 hour

What version of the product are you using? On what operating system?
1.1 on SunOS 5.10 Generic_118833-24 sun4u sparc SUNW,Sun-Fire-280R

Please provide any additional information below.

 Here's some of the log entries right before it crashes:

----------------------------------------
Exception happened during processing of request from ('193.148.29.85', 49992)
----------------------------------------
----------------------------------------
Exception happened during processing of request from ('193.148.29.85', 50015)
----------------------------------------
ct=load ip=10.155.14.28 q=allie young r= start=1 time=1204044889 
ct=c ip=129.212.11.23 q=disaster recovery r=5 start=1 time=1204044889
url=javascript:openDoc('/DisplayDocument?ref=g_search&id=552812&subref=simplesea
rch','_blank')

----------------------------------------
Exception happened during processing of request from ('199.67.203.141', 24754)
----------------------------------------
ct=load ip=194.51.20.124 q=MASTER DATA MANAGEMENT r= start=1 time=1204044889 
----------------------------------------
Exception happened during processing of request from ('10.1.9.33', 4347)
----------------------------------------
----------------------------------------
Exception happened during processing of request from ('82.44.82.96', 3768)
----------------------------------------
----------------------------------------
Exception happened during processing of request from ('157.130.225.194', 53634)
----------------------------------------
ct=load ip=72.112.80.55 r= start=1 time=1204044890 
ct=c ip=170.201.180.137 q="Computer Associates" r=1 start=1
time=1204044890
url=javascript:openDoc('/DisplayDocument?ref=g_search&id=487415&subref=simplesea
rch','_blank')

ct=c ip=68.193.143.165 q=speech engines r=1 start=1 time=1204044890
url=javascript:openDoc('/DisplayDocument?ref=g_search&id=482054&subref=simplesea
rch','_blank')

Original issue reported on code.google.com by bke...@gmail.com on 26 Feb 2008 at 11:30

GoogleCodeExporter commented 8 years ago 
That definitely shouldn't happen.  There are quite a few large companies using 
this
with no trouble, so I'd like to ask a few things about your environment:

1. What is your approx. query volume & click volume?
2. How large/powerful is the server where your python script resides?
3. Do you have enough hard disk space for the log files?
4. Are you able to re-produce this?  If so, do you clear log files between 
tries?
5. Did it only crash once after 1 hour, or always crashes after 1 hour?

If it's reliably crashing after 1 hour, it makes me think the log file is 
filling up,
but let's get some more data to help you out.  Thanks!

Original comment by cyr...@gmail.com on 27 Feb 2008 at 1:35

GoogleCodeExporter commented 8 years ago

Original comment by cyr...@gmail.com on 6 Mar 2008 at 7:46

GoogleCodeExporter commented 8 years ago 
Hi, I'm one of the developers of this and I have to ask a few dumb questions.

[1] When you show text like this above, where is it appearing (stdout/stderr? 
log
file?) and is it associated with a (python) stack trace?

----------------------------------------
Exception happened during processing of request from ('82.44.82.96', 3768)
----------------------------------------

[2] When this happens does the server exit or does it continue to work?

[3] Related - but just to clarify - when you say the server crashes, do you 
mean it
exits, or does it stay up and become nonoperational?

===
My current guess is that under some conditions the browser cancels or 
interrupts the
request it sends to the python logging server, and this causes problems. 
The process we're using of having a request being sent in an onmousedown 
handler in a
<a href> has the advantage of reducing client latency vs. other approaches, and 
with
the disadvantage that under some conditions a click my not be logged (the 
browser
canceling the onmousedown logging req), however this crashing shouldn't happen 
of course.

thx,
 Dave Spencer

Original comment by david.spencerian on 27 Mar 2008 at 5:49

GoogleCodeExporter commented 8 years ago 
[1]
Yes, its in the stdout/stderr log. Here it is with some surrounding text

ct=c ip=137.215.6.53 r=9 start=1 time=1206977862
url=javascript:openDoc('/DisplayDocument?ref=g_search&id=309024&subref=browse','
_blank')
----------------------------------------
Exception happened during processing of request from ('163.185.210.89', 2141)
----------------------------------------
ct=nav.srch ip=12.22.249.3 q=Scenarios for Avian Influenza AND How IT Can 
Mitigation
Risk r=null start=1 time=1206977863 url=javascript:submitSearch('Top','Bottom');
ct=nav.next ipTraceback (most recent call last):
  File "/usr/sfw/lib/python2.3/SocketServer.py", line 222, in handle_request
    self.process_request(request, client_address)
  File "/usr/sfw/lib/python2.3/SocketServer.py", line 241, in process_request
    self.finish_request(request, client_address)
  File "/usr/sfw/lib/python2.3/SocketServer.py", line 254, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/sfw/lib/python2.3/SocketServer.py", line 521, in __init__
    self.handle()
  File "/usr/sfw/lib/python2.3/BaseHTTPServer.py", line 324, in handle
    self.handle_one_request()
  File "/usr/sfw/lib/python2.3/BaseHTTPServer.py", line 318, in handle_one_request
    method()
  File "./asr.py", line 135, in do_GET
    self.send_response(204)
  File "/usr/sfw/lib/python2.3/BaseHTTPServer.py", line 374, in send_response
    self.wfile.write("%s %d %s\r\n" %
  File "/usr/sfw/lib/python2.3/socket.py", line 254, in write
    self.flush()
  File "/usr/sfw/lib/python2.3/socket.py", line 241, in flush
    self._sock.sendall(buffer)
error: (32, 'Broken pipe')
=137.201.242.130 q=hype cycle, 2008 r=null start=2 time=1206977863
url=javascript:changePage('52','0');
ct=load ip=12.22.249.3 q=Key Steps to Prepare for a Possible Avian Influenza 
Pandemic
r= start=1 time=1206977864

[2]
Sometimes it exits but sometimes the process remains but stops logging to the 
csv file. 

[3]
We have seen cases where it exits as well as where it remains but becomes
non-operational.

Original comment by bke...@gmail.com on 8 Apr 2008 at 1:51

GoogleCodeExporter commented 8 years ago 
Interesting - it appears that the python server can't send the "dummy" response
back to the browser, and the python code is then wrongly affected by this.

To fix, go to approx line 135 in asr.py and replace

    self.send_response(204)

with

    try:
      self.send_response(204)
    except:
      pass # client went away?

and note the indentation - the "try" should be indented as much as the
"self.send_resopnse" was in the original version of the code, and now
the "self.send..." line is indented...

I'm pretty sure I know what's going on - the browser sends a request to the 
python 
server from an onmousedown handler, and then it goes on to the hyperlink 
(href=..).
Under some circumstances (browser version, timing..) it can cancel the 
onmousedown
request, and it's pretty clearly doing so here.

The code change makes the server more resilient to "spurious" errors, as the 
server
shouldn't care if the client still exists.

thx,
 Dave

Original comment by david.spencerian on 21 Apr 2008 at 8:38