Closed ayrtonmassey closed 9 years ago
If right after that hapens, if you call it again with a short document that's easy to parse, does it work?
Also you could switch to the "namedpipe" branch, which doesn't have any timeout mechanism at all.
It keeps timing out. I've only let it run for about 4 or 5 documents after the first timeout but every single one gives me the same error. It ran out of memory trying to parse a huge sentence, but it processed another 30 documents before requests started timing out.
I found the code that's generating the exception in sockwrap.py (there's a comment that explicitly says the code does not restart the server in the event of a timeout) so I suppose changing the exception handling to restart the server would fix the issue (although it'd be very time consuming)?
I'll try the other branch first and post an update.
yeah, changing the code there to restart the server sounds like a reasonable option. i always find it difficult to determine the best approach for this stuff. going down this road, here are two things to be concerned about:
I suppose part of my issue is that I'm scraping documents and then passing them to the server directly, which is fine if the document is a regular passage of text, but right now I'm not doing any sanity checks on the input (my bad). The document that failed due to running out of memory was a set of lines with no full stops, so the parser interpreted the entire document as one sentence. The majority of documents should be sensible input, so I think it's more likely that the server crashed shortly after the memory error rather than the input taking too long to process.
Either way I could do with setting the server to auto-restart so that I don't have to constantly monitor the process (it's going to be on 24/7), so I'll make the changes you suggested and get back to you. Cheers!
Hope it goes well. FYI I just merged a fairly large change, where I switched to a new implementation mode of using named pipes for the server communication. it doesn't have timeouts. but i kept the older socket communication as an option, comm_mode='SOCKET'
. some notes on this near the bottom of the README. Only socket mode has timeouts, so if you work from the newest master
version, you'll have to make sure to use socket mode. the code has changed too so you might have to look at it again to figure out where to put in the restarts. sorry for the trouble! (for some use cases where the timeouts don't matter, the pipe mode is easier to use esp in parallel and a new implementation in R. all tradeoffs.)
I'm using the newest master (wasn't difficult to port to the named pipe system) and the repeated timeout issue seems to be resolved - the process has ran out of memory a couple of times due to bad input but it's managed to stay running all evening, so things are looking good.
Thanks for your all your help, the new version seems to be a huge improvement!
ok i guess i'll close this though it may have returned through #28
I'm trying to use this software to parse files continuously, but after about 300 or so I got this error:
Is there a fix for this? Some sort of flag I can set to restart the server when a timeout occurs?