Server timeout in rapid annotation mode

[TomokoOhta]

Sometimes it happens with an error message. Error: ActionsuggestSpanTypes failed on error error

Note: It happens inconsistently, and not yet sure when/why it happens. Will keep reporting when I meet this.

[spyysalo]

Note: I've added extra debugging code to the CG task production server on weaver, so we should have timing info next time this happens.

@TomokoOhta : please note the exact time you see this next time!

[TomokoOhta]

Met this at 23.09 on Thu. 29 Nov. on Win7 at home. Reloading helped this.

[spyysalo]

@TomokoOhta : thank you! The relevant part of the log is

2012-11-29 23:02:49.467430 suggest_span_types: start, text: treatment
2012-11-29 23:02:49.467430 suggest_span_types: got 52 predictions after 0.7 seconds
2012-11-29 23:03:06.216061 suggest_span_types: start, text: up-regulated
2012-11-29 23:03:06.216061 suggest_span_types: got 52 predictions after 0.7 seconds
2012-11-29 23:03:19.519696 suggest_span_types: start, text: inducing
2012-11-29 23:03:19.519696 suggest_span_types: got 52 predictions after 0.7 seconds
2012-11-29 23:11:23.572578 suggest_span_types: start, text: inducing
2012-11-29 23:11:23.572578 suggest_span_types: got 52 predictions after 0.7 seconds
2012-11-29 23:11:27.185964 suggest_span_types: start, text: apoptosis
2012-11-29 23:11:27.185964 suggest_span_types: got 52 predictions after 0.7 seconds
2012-11-29 23:12:18.796903 suggest_span_types: start, text: treating
2012-11-29 23:12:18.796903 suggest_span_types: got 52 predictions after 0.10 seconds

There's a gap in the log between 23:03 and 23:11. The interesting bit is, the server logs the start of the request, and there's no record of that (nor of a failed request to the simsem server).

At least in this case, I think the server never received the client request at all, which suggests the issue is not between the brat server and the simsem server, as previously.

To make sure this is the only issue, please keep tracking when this happens. It would be helpful if you could log also which document you were working on, and which word(s) you were annotating when the rapid annotation mode failed.

[TomokoOhta]

I've been working on http://[REDACTED]/Apoptosis-set-2/12432255 when I met this.

It repeated at 23.15 (in the same document). Although I forgot to leave a memo on which word when it happened, I'm afraid, and not entirely sure anymore (because the information at the bottom was removed after reloading), but as far as I remember, it could be Pos_reg annotation (E6) on the word "induce" in Sentence 3.

Will log more detail when I have this timeout next time.

[spyysalo]

Log around 23:15

2012-11-29 23:14:22.424927 suggest_span_types: start, text: expression
2012-11-29 23:14:22.424927 suggest_span_types: got 52 predictions after 0.8 seconds
2012-11-29 23:14:29.132970 suggest_span_types: start, text: unchanged
2012-11-29 23:14:29.132970 suggest_span_types: got 52 predictions after 0.7 seconds
2012-11-29 23:15:55.574627 suggest_span_types: start, text: apoptosis
2012-11-29 23:15:55.574627 suggest_span_types: got 52 predictions after 0.7 seconds
2012-11-29 23:16:37.693036 suggest_span_types: start, text: underwent
2012-11-29 23:16:37.693036 suggest_span_types: got 52 predictions after 0.9 seconds

(also no indication of issues in server-server communication.)

[TomokoOhta]

I experienced another at 18.52 on Fri. 7 Dec.

Server Python crash, stack-trace is: 

Traceback (most recent call last): 
File "/home/brat/public_html/projects/cg/ohta-master/server/src/server.py", line 322, in serve 
return _safe_serve(params, client_ip, client_hostname, cookie_data) 
File "/home/brat/public_html/projects/cg/ohta-master/server/src/server.py", line 197, in _safe_serve 
json_dic = dispatch(http_args, client_ip, client_hostname) 
File "/home/brat/public_html/projects/cg/ohta-master/server/src/dispatch.py", line 313, in dispatch 
json_dic = action_function(*action_args) 
File "/home/brat/public_html/projects/cg/ohta-master/server/src/predict.py", line 78, in suggest_span_types 
resp = urlopen(model_url % quoted_text, None, QUERY_TIMEOUT) 
File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen 
return _opener.open(url, data, timeout) 
File "/usr/lib/python2.7/urllib2.py", line 400, in open 
response = self._open(req, data) 
File "/usr/lib/python2.7/urllib2.py", line 418, in _open 
'_open', req) 
File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain 
result = func(*args) 
File "/usr/lib/python2.7/urllib2.py", line 1207, in http_open 
return self.do_open(httplib.HTTPConnection, req) 
File "/usr/lib/python2.7/urllib2.py", line 1180, in do_open 
r = h.getresponse(buffering=True) 
File "/usr/lib/python2.7/httplib.py", line 1030, in getresponse 
response.begin() 
File "/usr/lib/python2.7/httplib.py", line 407, in begin 
version, status, reason = self._read_status() 
File "/usr/lib/python2.7/httplib.py", line 365, in _read_status 
line = self.fp.readline() 
File "/usr/lib/python2.7/socket.py", line 447, in readline 
data = self._sock.recv(self._rbufsize) 
timeout: timed out 

[spyysalo]

OK, that's pretty clearly a real timeout in the communication between the brat server and the simsem server, a separate issue from the one discussed above. If this repeats, please talk to @ninjin .