Nightly timed out after 1h

tripleee commented 4 months ago

https://app.circleci.com/pipelines/github/tripleee/sloshy/138/workflows/3fc44a90-0d62-4d23-bc26-e1bf833b534f/jobs/135 failed at https://chat.stackoverflow.com/transcript/message/57381231#57381231 after 18 rooms, apparently because it exceeded the 1h timeout deadline on CircleCI.

tripleee commented 4 months ago

There was a 429 error earlier in the failed transcript; maybe that's the root cause?

2024-05-24 03:30:34.794825 [Matter Modeling: Questions related to Matter Modeling, but posted in other places (e.g. other SE sites, or Reddit)](https://chat.stackexchange.com/transcript/110113/2024/5/17): latest activity 2024-05-17 14:13:00 (6 days, 13:05:03 hours ago)
Exception in thread ChatExchange: message_sender for chat.stackoverflow.com:
Traceback (most recent call last):
  File "/home/circleci/.pyenv/versions/3.12.3/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
    self.run()
  File "/home/circleci/.pyenv/versions/3.12.3/lib/python3.12/threading.py", line 1010, in run
    self._target(*self._args, **self._kwargs)
  File "/home/circleci/.pyenv/versions/3.12.3/lib/python3.12/site-packages/chatexchange/client.py", line 229, in _worker
    self._do_action_despite_throttling(next_action)
  File "/home/circleci/.pyenv/versions/3.12.3/lib/python3.12/site-packages/chatexchange/client.py", line 265, in _do_action_despite_throttling
    response = self._br.send_message(room_id, text)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/circleci/.pyenv/versions/3.12.3/lib/python3.12/site-packages/chatexchange/browser.py", line 318, in send_message
    return self.post_fkeyed(
           ^^^^^^^^^^^^^^^^^
  File "/home/circleci/.pyenv/versions/3.12.3/lib/python3.12/site-packages/chatexchange/browser.py", line 136, in post_fkeyed
    return self.post(url, data, headers)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/circleci/.pyenv/versions/3.12.3/lib/python3.12/site-packages/chatexchange/browser.py", line 116, in post
    return self._request('post', url, data, headers, with_chat_root)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/circleci/.pyenv/versions/3.12.3/lib/python3.12/site-packages/chatexchange/browser.py", line 105, in _request
    response.raise_for_status()
  File "/home/circleci/.pyenv/versions/3.12.3/lib/python3.12/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://chat.stackoverflow.com/chats/111347/messages/new
2024-05-24 03:33:22.208002 [Matter Modeling: Ask a Moderator](https://chat.stackexchange.com/transcript/110431): latest activity 2024-05-20 03:31:00 (3 days, 23:47:03 hours ago)
2024-05-24 03:33:25.438350 [Matter Modeling: Announcements of user or site milestones achieved](https://chat.stackexchange.com/transcript/109902): latest activity 2024-05-14 03:31:00 (9 days, 23:47:03 hours ago)

tripleee commented 4 months ago

Maybe fallout from #78; some rooms took a really long time.

14:03 Physics questions at MMSE started 03:37:06.100515 finished 03:51:09.160783
3:17 Matter Modeling: LAMMPS software started 03:19:44.673711 finished 03:23:11.320331
2:52 HNQ (Operations Research) started 03:52:13.234857 finished 03:55:05.044884
2:19 PLDI Hot Network Questions started 03:56:51.981890 finished 03:59:10.019396
1:43 Matter Modeling: Q-Chem softwar started 03:35:23.132817 finished 03:37:06.100515
1:15 Matter Modeling: Phonopy software started 03:18:34.089748 finished 03:19:44.673711

tripleee commented 4 months ago

For now, rerunning the job manually worked.

tripleee commented 4 months ago

Happened again on Saturday (nightly #145; chat link) but Sunday's job worked normally again.

tripleee commented 4 months ago

Again again nightly #147, though this seems to have been purely a chat glitch.

tripleee commented 4 months ago

Again again nightly #165

tripleee commented 4 months ago

Still timing out with 429 errors, though the transcript logic should now be sounder. Do I need to increase the sleep between traversals? It works on my laptop, though.

tripleee commented 4 months ago

https://github.com/tripleee/sloshy/commit/b34429f739866a62240724410ab2a6f6dd6ff011 adds a proper 429 retry handler

tripleee commented 4 months ago

PR:s #98 and #99 (silly typo) add a sleep when there's an error.

tripleee commented 4 months ago

Switching back to Github Actions seems to trivially fix this. Will proceed to disable CircleCI again.

For the future, my best vague guess is that it might need a bigger machine. https://circleci.com/docs/configuration-reference/#resourceclass

tripleee commented 4 months ago

Clicked "Stop Building" on https://app.circleci.com/settings/project/github/tripleee/sloshy

tripleee commented 4 months ago

It's not entirely obvious how to enable CircleCI again. https://circleci.com/docs/stop-building-a-project-on-circleci/ says to go through the steps to create a new project.

tripleee / sloshy

Nightly timed out after 1h #87