Socket timeout happened, returning None

[ayrtonmassey]

I'm trying to use this software to parse files continuously, but after about 300 or so I got this error:

INFO:StanfordSocketWrap:Socket timeout happened, returning None: <class 'socket.timeout'> timed out

Is there a fix for this? Some sort of flag I can set to restart the server when a timeout occurs?

[brendano]

If right after that hapens, if you call it again with a short document that's easy to parse, does it work?

Also you could switch to the "namedpipe" branch, which doesn't have any timeout mechanism at all.

[ayrtonmassey]

It keeps timing out. I've only let it run for about 4 or 5 documents after the first timeout but every single one gives me the same error. It ran out of memory trying to parse a huge sentence, but it processed another 30 documents before requests started timing out.

I found the code that's generating the exception in sockwrap.py (there's a comment that explicitly says the code does not restart the server in the event of a timeout) so I suppose changing the exception handling to restart the server would fix the issue (although it'd be very time consuming)?

I'll try the other branch first and post an update.

[brendano]

yeah, changing the code there to restart the server sounds like a reasonable option. i always find it difficult to determine the best approach for this stuff. going down this road, here are two things to be concerned about:

• if the restarter code is unsuccessful in killing the current process, you could have a situation where two or more server processes start piling up in the background. i'm not sure how robust the restarter code is. if kill -9 isn't in there, it should be added.
  • the server timing out might be symptomatic of a larger problem. the memory limit might be too low (it's hardcoded as a python constant; might need to be increased), or the documents might be too long and need to be shortened somehow (because corenlp, like many other academic nlp software systems, very heavily assumes an entire document can be loaded into memory at once and memory usage linear in doc length is ok). another is that maybe everything is working find and it just needs a really long time to process the document?

[ayrtonmassey]

I suppose part of my issue is that I'm scraping documents and then passing them to the server directly, which is fine if the document is a regular passage of text, but right now I'm not doing any sanity checks on the input (my bad). The document that failed due to running out of memory was a set of lines with no full stops, so the parser interpreted the entire document as one sentence. The majority of documents should be sensible input, so I think it's more likely that the server crashed shortly after the memory error rather than the input taking too long to process.

Either way I could do with setting the server to auto-restart so that I don't have to constantly monitor the process (it's going to be on 24/7), so I'll make the changes you suggested and get back to you. Cheers!

[brendano]

Hope it goes well. FYI I just merged a fairly large change, where I switched to a new implementation mode of using named pipes for the server communication. it doesn't have timeouts. but i kept the older socket communication as an option, comm_mode='SOCKET'. some notes on this near the bottom of the README. Only socket mode has timeouts, so if you work from the newest master version, you'll have to make sure to use socket mode. the code has changed too so you might have to look at it again to figure out where to put in the restarts. sorry for the trouble! (for some use cases where the timeouts don't matter, the pipe mode is easier to use esp in parallel and a new implementation in R. all tradeoffs.)

[ayrtonmassey]

I'm using the newest master (wasn't difficult to port to the named pipe system) and the repeated timeout issue seems to be resolved - the process has ran out of memory a couple of times due to bad input but it's managed to stay running all evening, so things are looking good.

Thanks for your all your help, the new version seems to be a huge improvement!

[brendano]

ok i guess i'll close this though it may have returned through #28