search

tripleee / sloshy

Sloshy the Thawman, a simple chatbot to keep rooms alive on Stack Overflow / Stack Exchange
6 stars 11 forks source link

Nightly timed out on Github Actions, too #103

Open tripleee opened 3 weeks ago

tripleee commented 3 weeks ago

Nightly #933 died with 429 errors just after I switched back to Github Actions from CircleCI (closed #87) 

Exception in thread ChatExchange: message_sender for chat.stackoverflow.com:
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/home/runner/.local/lib/python3.10/site-packages/chatexchange/client.py", line 229, in _worker
    self._do_action_despite_throttling(next_action)
  File "/home/runner/.local/lib/python3.10/site-packages/chatexchange/client.py", line 265, in _do_action_despite_throttling
    response = self._br.send_message(room_id, text)
  File "/home/runner/.local/lib/python3.10/site-packages/chatexchange/browser.py", line 318, in send_message
    return self.post_fkeyed(
  File "/home/runner/.local/lib/python3.10/site-packages/chatexchange/browser.py", line 136, in post_fkeyed
    return self.post(url, data, headers)
  File "/home/runner/.local/lib/python3.10/site-packages/chatexchange/browser.py", line 116, in post
    return self._request('post', url, data, headers, with_chat_root)
  File "/home/runner/.local/lib/python3.10/site-packages/chatexchange/browser.py", line 105, in _request
    response.raise_for_status()
  File "/usr/lib/python3/dist-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://chat.stackoverflow.com/chats/233626/messages/new
ERROR:root:** Error: could not fetch transcript for [SOBotics](https://chat.stackoverflow.com/rooms/11[134](https://github.com/tripleee/sloshy/actions/runs/9425364237/job/25966939141#step:3:135)7)
ERROR:root:RetryError(MaxRetryError("HTTPSConnectionPool(host='chat.stackoverflow.com', port=443): Max retries exceeded with url: /rooms/info/111347/?tab=feeds (Caused by ResponseError('too many 429 error responses'))"))
ERROR:root:** Error: could not fetch transcript for [friendly bin](https://chat.stackoverflow.com/rooms/109494)
ERROR:root:RetryError(MaxRetryError("HTTPSConnectionPool(host='chat.stackoverflow.com', port=443): Max retries exceeded with url: /rooms/info/109494/?tab=feeds (Caused by ResponseError('too many 429 error responses'))"))
ERROR:root:** Error: could not fetch transcript for [Git](https://chat.stackoverflow.com/rooms/228186)
ERROR:root:RetryError(MaxRetryError("HTTPSConnectionPool(host='chat.stackoverflow.com', port=443): Max retries exceeded with url: /rooms/info/228186/?tab=feeds (Caused by ResponseError('too many 429 error responses'))"))
ERROR:root:** Error: could not fetch transcript for [Duck Overflow](https://chat.stackoverflow.com/rooms/193903)
ERROR:root:RetryError(MaxRetryError("HTTPSConnectionPool(host='chat.stackoverflow.com', port=443): Max retries exceeded with url: /rooms/info/193903/?tab=feeds (Caused by ResponseError('too many 429 error responses'))"))
ERROR:root:** Error: could not fetch transcript for [shree's room](https://chat.stackoverflow.com/rooms/232981)
ERROR:root:RetryError(MaxRetryError("HTTPSConnectionPool(host='chat.stackoverflow.com', port=443): Max retries exceeded with url: /rooms/info/232981/?tab=feeds (Caused by ResponseError('too many 429 error responses'))"))
ERROR:root:** Error: could not fetch transcript for [Google Apps Script chat community](https://chat.stackoverflow.com/rooms/217630)
ERROR:root:RetryError(MaxRetryError("HTTPSConnectionPool(host='chat.stackoverflow.com', port=443): Max retries exceeded with url: /rooms/info/217630/?tab=feeds (Caused by ResponseError('too many 429 error responses'))"))
ERROR:root:** Error: could not fetch transcript for [V-eye](https://chat.stackoverflow.com/rooms/240030)
ERROR:root:RetryError(MaxRetryError("HTTPSConnectionPool(host='chat.stackoverflow.com', port=443): Max retries exceeded with url: /rooms/info/240030/?tab=feeds (Caused by ResponseError('too many 429 error responses'))"))
ERROR:root:** Error: could not fetch transcript for [\[dev\] SpotDetector](https://chat.stackoverflow.com/rooms/244392)
ERROR:root:RetryError(MaxRetryError("HTTPSConnectionPool(host='chat.stackoverflow.com', port=443): Max retries exceeded with url: /rooms/info/244392/?tab=feeds (Caused by ResponseError('too many 429 error responses'))"))
ERROR:root:** Error: could not fetch transcript for [Userscript newbies and friends](https://chat.stackoverflow.com/rooms/2[143](https://github.com/tripleee/sloshy/actions/runs/9425364237/job/25966939141#step:3:144)45)
ERROR:root:RetryError(MaxRetryError("HTTPSConnectionPool(host='chat.stackoverflow.com', port=443): Max retries exceeded with url: /rooms/info/214345/?tab=feeds (Caused by ResponseError('too many 429 error responses'))"))
ERROR:root:** Error: could not fetch transcript for [ElectionBot Development (SO)](https://chat.stackoverflow.com/rooms/190503)
ERROR:root:RetryError(MaxRetryError("HTTPSConnectionPool(host='chat.stackoverflow.com', port=443): Max retries exceeded with url: /rooms/info/190503/?tab=feeds (Caused by ResponseError('too many 429 error responses'))"))
ERROR:root:** Error: could not fetch transcript for [tkinter](https://chat.stackoverflow.com/rooms/249491)
ERROR:root:RetryError(MaxRetryError("HTTPSConnectionPool(host='chat.stackoverflow.com', port=443): Max retries exceeded with url: /rooms/info/249491/?tab=feeds (Caused by ResponseError('too many 429 error responses'))"))
tripleee commented 3 weeks ago

The last two nightlies both failed with 429 errors.

Probably not related to https://meta.stackexchange.com/questions/400700/chat-search-does-not-bring-up-new-results (since it is reported to have started only today) but still mentioning, just in case.

tripleee commented 3 weeks ago

Dumbfoundingly, adding more debug logging seems to have made the problem go away, at least for now. I'll leave the setting hardcoded for the time being.

tripleee commented 2 weeks ago

Nightly #1006 failed with a 429 error, apparently killing the chat client.

tripleee commented 3 hours ago

This happened again, and I expect it to continue to happen again from time to time.

My current thinking is that ChatExchange needs to be updated to cope with 429 errors correctly. There is some logic in there which attempts to handle them, but clearly it is not working in this case. The dreaded threaded design exacerbates this by hiding the error, so the bot continues to try even though the delivery thread is dead because of an unhandled exception.