non-deterministic output on perWord endpoint

[jonorthwash]

With a single request, we're getting up to three different responses.

This is the request we're using: http://beta.apertium.org/apy/perWord?lang=eng-fra&modes=biltrans&q=hi

Depending on APy's mood, these are the outputs we get:

Correct output:

[{"biltrans": ["salut<ij>"], "input": "hi"}]

Empty output:

[{"biltrans": [], "input": "hi"}]

Error output:

{"message": "Internal Server Error", "code": 500, "status": "error", "explanation": "Internal Server Error"}

Noticed by @avyayv.

[sushain97]

Server logs:

apertium@oqaa:~/beta/apertium-html-tools$ docker-compose logs -f --tail 0 apy
Attaching to apertiumhtmltools_apy_1
apy_1         | [I 180104 04:01:15 util:82] util.apertium('hi', '/source/apertium-fra-eng', 'eng-fra-morph', 'txt')
apy_1         | [I 180104 04:01:16 web:2063] 200 GET /perWord?lang=eng-fra&modes=biltrans&q=hi (172.18.0.1) 1360.03ms
apy_1         | [W 180104 04:01:16 servlet:100] Caught signal: 15
apy_1         | [E 180104 04:01:23 ioloop:638] Exception in callback None
apy_1         |     Traceback (most recent call last):
apy_1         |       File "/usr/local/lib/python3.5/dist-packages/tornado/ioloop.py", line 887, in start
apy_1         |         fd_obj, handler_func = self._handlers[fd]
apy_1         |     KeyError: 89
apy_1         | [E 180104 04:01:23 ioloop:638] Exception in callback None
apy_1         |     Traceback (most recent call last):
apy_1         |       File "/usr/local/lib/python3.5/dist-packages/tornado/ioloop.py", line 887, in start
apy_1         |         fd_obj, handler_func = self._handlers[fd]
apy_1         |     KeyError: 89
apy_1         | [E 180104 04:01:23 ioloop:638] Exception in callback None
apy_1         |     Traceback (most recent call last):
apy_1         |       File "/usr/local/lib/python3.5/dist-packages/tornado/ioloop.py", line 887, in start
apy_1         |         fd_obj, handler_func = self._handlers[fd]
apy_1         |     KeyError: 89
apy_1         | [E 180104 04:01:26 web:1590] Uncaught exception GET /perWord?lang=eng-fra&modes=biltrans&q=hi (172.18.0.1)
apy_1         |     HTTPServerRequest(protocol='http', host='beta.apertium.org', method='GET', uri='/perWord?lang=eng-fra&modes=biltrans&q=hi', version='HTTP/1.1', remote_ip='172.18.0.1', headers={'X-Forwarded-Server': 'beta.apertium.org', 'Accept-Encoding': 'gzip, deflate', 'X-Forwarded-Host': 'beta.apertium.org', 'X-Forwarded-For': '76.30.93.54', 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.108 Safari/537.36', 'Host': 'beta.apertium.org', 'Accept-Language': 'en-US,en;q=0.9', 'Upgrade-Insecure-Requests': '1', 'Cache-Control': 'max-age=0', 'Connection': 'Keep-Alive', 'Dnt': '1', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8', 'If-None-Match': '"745e2304f744fb68bb1667fe7626f8fe37e4bc40"'})
apy_1         |     Traceback (most recent call last):
apy_1         |       File "/usr/local/lib/python3.5/dist-packages/tornado/web.py", line 1467, in _stack_context_handle_exception
apy_1         |         raise_exc_info((type, value, traceback))
apy_1         |       File "<string>", line 4, in raise_exc_info
apy_1         |       File "/usr/local/lib/python3.5/dist-packages/tornado/stack_context.py", line 316, in wrapped
apy_1         |         ret = fn(*args, **kwargs)
apy_1         |       File "/usr/local/lib/python3.5/dist-packages/tornado/web.py", line 1680, in future_complete
apy_1         |         f.result()
apy_1         |       File "/usr/local/lib/python3.5/dist-packages/tornado/concurrent.py", line 238, in result
apy_1         |         raise_exc_info(self._exc_info)
apy_1         |       File "<string>", line 4, in raise_exc_info
apy_1         |       File "/usr/local/lib/python3.5/dist-packages/tornado/gen.py", line 307, in wrapper
apy_1         |         yielded = next(result)
apy_1         |       File "/root/apertium-apy/servlet.py", line 1032, in get
apy_1         |         pool = Pool(processes=1)
apy_1         |       File "/usr/lib/python3.5/multiprocessing/context.py", line 118, in Pool
apy_1         |         context=self.get_context())
apy_1         |       File "/usr/lib/python3.5/multiprocessing/pool.py", line 168, in __init__
apy_1         |         self._repopulate_pool()
apy_1         |       File "/usr/lib/python3.5/multiprocessing/pool.py", line 233, in _repopulate_pool
apy_1         |         w.start()
apy_1         |       File "/usr/lib/python3.5/multiprocessing/process.py", line 103, in start
apy_1         |         'daemonic processes are not allowed to have children'
apy_1         |     AssertionError: daemonic processes are not allowed to have children
apy_1         | [E 180104 04:01:26 web:2063] 500 GET /perWord?lang=eng-fra&modes=biltrans&q=hi (172.18.0.1) 1.44ms
apy_1         | [E 180104 04:01:26 ioloop:638] Exception in callback None
apy_1         |     Traceback (most recent call last):
apy_1         |       File "/usr/local/lib/python3.5/dist-packages/tornado/ioloop.py", line 887, in start
apy_1         |         fd_obj, handler_func = self._handlers[fd]
apy_1         |     KeyError: 8
apy_1         | [E 180104 04:01:26 ioloop:638] Exception in callback None
apy_1         |     Traceback (most recent call last):
apy_1         |       File "/usr/local/lib/python3.5/dist-packages/tornado/ioloop.py", line 887, in start
apy_1         |         fd_obj, handler_func = self._handlers[fd]
apy_1         |     KeyError: 8
apy_1         | [E 180104 04:01:26 ioloop:638] Exception in callback None
apy_1         |     Traceback (most recent call last):
apy_1         |       File "/usr/local/lib/python3.5/dist-packages/tornado/ioloop.py", line 887, in start
apy_1         |         fd_obj, handler_func = self._handlers[fd]
apy_1         |     KeyError: 8

cc @Androbin this is a bug you can look at if you'd like

[Androbin]

What is Exception in callback None even supposed to mean? There is no callback, but it threw an exception?

[sushain97]

Honestly, I have no idea. It needs some debugging. @unhammer ?

[unhammer]

I don't understand what PerWordHandler does there with the pool and everything – anyone know the intention behind that code? It's not written in the same style as the translation stuff at least (and processPerWord is not a @gen.coroutine)

[Androbin]

@unhammer The Pool used reminds me of what we had in #76.

[avyayv]

Now all I seem to get is: {"explanation": "Internal Server Error", "message": "Internal Server Error", "code": 500, "status": "error"} I don't get [{"biltrans": [], "input": "hi"}] or [{"biltrans": ["salut<ij>"], "input": "hi"}] anymore

[jonorthwash]

Yeah, it only seems to return the other outputs off and on for a while after restart?