Error with API connecting to the backend

[dale-wahl]

Hello and thanks for building and maintaining Hyphe! I have a Docker version built from commit ea22f0532dbf419d4670d8633c6e14a42799518c set up the week of January 10th and am running into an issue with the API. It starts to occur on certain calls, but disappears after stopping and then restarting the backend container.

I noticed it with a python call for store.get_webentities such as: response = requests.post('http://127.0.0.1:6978', json={'method':"store.get_webentities",'params':{'corpus':'ic-1-356283'},'jsonrpc':'2.0','id':'bff612dc-349b-4096-b511-97ce1c62a39b'})

The request times out or never completes without a timeout parameter though after restarting the backend it only takes a couple minutes. The backend presents this error:

2022-02-02 11:38:07+0000 [-] Unhandled error in Deferred:
2022-02-02 11:38:07+0000 [-] Unhandled Error
        Traceback (most recent call last):
          File "/usr/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1475, in gotResult
            _inlineCallbacks(r, g, status)
          File "/usr/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1461, in _inlineCallbacks
            status.deferred.callback(e.value)
          File "/usr/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 460, in callback
            self._startRunCallbacks(result)
          File "/usr/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 568, in _startRunCallbacks
            self._runCallbacks()
        --- <exception caught here> ---
          File "/usr/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
            current.result = callback(current.result, *args, **kw)
          File "/app/hyphe_backend/lib/jsonrpc_custom.py", line 125, in _cbRender
            return JSONRPC._cbRender(self, result, request, id, version)
          File "/usr/local/lib/python2.7/site-packages/txjsonrpc/web/jsonrpc.py", line 194, in _cbRender
            request.write(six.b(s))
          File "/usr/local/lib/python2.7/site-packages/twisted/web/server.py", line 238, in write
            http.Request.write(self, data)
          File "/usr/local/lib/python2.7/site-packages/twisted/web/http.py", line 1122, in write
            self.channel.writeHeaders(version, code, reason, headers)
        exceptions.AttributeError: 'NoneType' object has no attribute 'writeHeaders'

[boogheta]

Hello Dave and thanks for your feedback. Did you encounter an actual error while using Hyphe, or is this just a stack that you found repeatedly in the backend's logs without knowing exactly what it might have caused? Because it looks to me indeed like a common timeout error on some queries, that might be due to a user having closed his hyphe tab in the browser before an API query was completed, in which case I agree we should try and avoid displaying such stacks in the logs, but you shouldn't worry about it I think

[dale-wahl]

The initial error was non-response from Hyphe's API. That stack repeated every time I sent the requests.post() from above to the API. After restarting the backend Docker container, I saw the same error stack from the request, but also a response from Hyphe's API with the final line "'NoneType' object has no attribute 'writeHeaders'". Clearly some other error was responsible for Hyphe's API no longer responding.

I have since deleted that particular corpus and that error stack no longer appears so something was likely wrong with that corpus itself. After deleting the corpus and restarting the backend again, I have not had the API hang again. It was interesting because it was clearly related to the store.get_webentities endpoint. I could not make any store.get_webentities requests regardless of corpus and receive responses, but I could still make other requests of Hyphe API and receive responses. If the API hangs up again, I can further investigate. Sorry that I do not have more details; for the moment things are running normally again!

[dale-wahl]

I should probably note that I am using Hyphe to collect data and then extracting it via the API. We're piping it to another tool for different sorts of network analysis. We do have some users utilizing Hyphe's native UI, but pretty rarely at the moment.

[boogheta]

OK thanks for the details. It looks to me like a timeout due to some connection closure somewhere indeed, maybe from another stack above indeed? Could you restart the backend after having setup HYPHE_DEBUG=2 in config-backend.env, retry and paste more lines of log from above the stack when it happens again?

[dale-wahl]

Hello boogheta,

I just wanted to update you and let you know that I have not run into the issue again. I think that I may have been misusing the pagination for these calls causing something to hang on large corpora, but after making changes there everything seems fine.

[boogheta]

Well, good news then! I'm closing the issue, but feel free to reopen it if the problem raises again!