Bot stops working after exception in handling thread

[laanwj]

A few days ago my instance of GHI got an invalid request, which did not come from github, for all intents or purposes it looks like a random scan for a XMLRPC vulnerability, but it managed to make the bot stop working

Jun 27 17:38:37 galatea python3[458]: POST /ui/h5-vsan/rest/proxy/service/invalidArgument/invalidArgument (XX.XX.XX.XX)
Jun 27 17:38:37 galatea python3[458]: Found configuration file at '/home/ghi/.ghi.yml'
Jun 27 17:38:38 galatea python3[458]:   config = yaml.load(readFile(configFilePath))
Jun 27 17:38:38 galatea python3[458]: Exception in thread Thread-1:
Jun 27 17:38:38 galatea python3[458]: Traceback (most recent call last):
Jun 27 17:38:38 galatea python3[458]:   File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
Jun 27 17:38:38 galatea python3[458]:     self.run()
Jun 27 17:38:38 galatea python3[458]:   File "/usr/lib/python3.8/threading.py", line 870, in run
Jun 27 17:38:38 galatea python3[458]:     self._target(*self._args, **self._kwargs)
Jun 27 17:38:38 galatea python3[458]:   File "ghi/ghi/server.py", line 86, in worker
Jun 27 17:38:38 galatea python3[458]:     response = item(*args, **kwargs)
Jun 27 17:38:38 galatea python3[458]:   File "ghi/ghi/server.py", line 44, in InvokeFunction
Jun 27 17:38:38 galatea python3[458]:     return handler(event=payload, sysd=GetArgs()['log_mode'])
Jun 27 17:38:38 galatea python3[458]:   File "ghi/index.py", line 54, in handler
Jun 27 17:38:38 galatea python3[458]:     pool = getPool(githubPayload, configuration["pools"])
Jun 27 17:38:38 galatea python3[458]:   File "ghi/github.py", line 23, in getPool
Jun 27 17:38:38 galatea python3[458]:     repo = payload["repository"]["full_name"]
Jun 27 17:38:38 galatea python3[458]: KeyError: 'repository'

After this are no more reports of notifications at all, nor errors, although I'm sure github kept sending them. It's as if it became deaf. Likely, the exception brought it into an invalid state.

After restarting everything worked again.

[laanwj]

Today we had a similar issue where notification to mastodon failed, and the bot stopped responding to new events afterwards:

Dec 03 12:13:26 galatea python3[3647184]: Exception in thread Thread-1:
Dec 03 12:13:26 galatea python3[3647184]: Traceback (most recent call last):
Dec 03 12:13:26 galatea python3[3647184]:   File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
Dec 03 12:13:26 galatea python3[3647184]:     self.run()
Dec 03 12:13:26 galatea python3[3647184]:   File "/usr/lib/python3.8/threading.py", line 870, in run
Dec 03 12:13:26 galatea python3[3647184]:     self._target(*self._args, **self._kwargs)
Dec 03 12:13:26 galatea python3[3647184]:   File "ghi/ghi/server.py", line 86, in worker
Dec 03 12:13:26 galatea python3[3647184]:     response = item(*args, **kwargs)
Dec 03 12:13:26 galatea python3[3647184]:   File "ghi/ghi/server.py", line 44, in InvokeFunction
Dec 03 12:13:26 galatea python3[3647184]:     return handler(event=payload, sysd=GetArgs()['log_mode'])
Dec 03 12:13:26 galatea python3[3647184]:   File "/home/ghi/ghi/ghi/index.py", line 133, in handler
Dec 03 12:13:26 galatea python3[3647184]:     sendToMastodon = sendToots(pool["pool"], getMessages["mastMessages"])
Dec 03 12:13:26 galatea python3[3647184]:   File "/home/ghi/ghi/ghi/ghimastodon.py", line 73, in sendToots
Dec 03 12:13:26 galatea python3[3647184]:     mastodon = login(pool)
Dec 03 12:13:26 galatea python3[3647184]:   File "/home/ghi/ghi/ghi/ghimastodon.py", line 52, in login
Dec 03 12:13:26 galatea python3[3647184]:     mastodon.account_verify_credentials()
Dec 03 12:13:26 galatea python3[3647184]:   File "<decorator-gen-24>", line 2, in account_verify_credentials
Dec 03 12:13:26 galatea python3[3647184]:   File "/home/ghi/.local/lib/python3.8/site-packages/mastodon/Mastodon.py", line 102, in wrapper
Dec 03 12:13:26 galatea python3[3647184]:     return function(self, *args, **kwargs)
Dec 03 12:13:26 galatea python3[3647184]:   File "/home/ghi/.local/lib/python3.8/site-packages/mastodon/Mastodon.py", line 1020, in account_verify_credentials
Dec 03 12:13:26 galatea python3[3647184]:     return self.__api_request('GET', '/api/v1/accounts/verify_credentials')
Dec 03 12:13:26 galatea python3[3647184]:   File "/home/ghi/.local/lib/python3.8/site-packages/mastodon/Mastodon.py", line 3425, in __api_request
Dec 03 12:13:26 galatea python3[3647184]:     raise ex_type(
Dec 03 12:13:26 galatea python3[3647184]: mastodon.Mastodon.MastodonGatewayTimeoutError: ('Mastodon API returned error', 504, 'Gateway Time-out', None)

[laanwj]

It looks like all in all, mastodon fails to post the request pretty often. It's a bit weird as it runs on the same server. Maybe the server is just slow.

In any case it would be better if ghi would keep responding to events after an exception (just missing that particular event), instead of going into incommunicado mode. Even straight-up crashing would be preferable.

[gkrizek]

Thanks for the issue @laanwj ! I agree with your suggestion, ghi should continue to process events.

I've been super busy lately and haven't had time to give Ghi much love lately but I'll try to carve out some time soon.

[m3dwards]

@laanwj can I confirm that you are running the server.py version of this and not via AWS lambda?

I can replicate the issue when an unexpected exception occurs when ghi uses it's own server.

[gkrizek]

I'm almost positive he is not using Lambda.

[laanwj]

@laanwj can I confirm that you are running the server.py version of this and not via AWS lambda?

Yes, I'm using my own server.