In the logs, this error message is popping up. It's due to sqlitedict trying to start a thread to store deployment details on a failed run (in this case due to git timeout due to waiting on the SSH key to be unlocked via user input), but the main thread has exited?
2024-04-09 23:55:16,111 - deployments.kill_deployment - ERROR - Failure! `shield-management/golf-rocket` deployment failed!
Exception in thread shield-management/golf-rocket:
Traceback (most recent call last):
File "/Users/admin/git/personal/harvey/harvey/git.py", line 61, in pull_repo
command_output = run_subprocess_command(['git', '-C', project_path, 'pull', '--rebase'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/admin/git/personal/harvey/harvey/utils/utils.py", line 35, in run_subprocess_command
command_output = subprocess.check_output( # nosec
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/python@3.12/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/subprocess.py", line 466, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/python@3.12/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/subprocess.py", line 550, in run
stdout, stderr = process.communicate(input, timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/admin/git/personal/harvey/venv/lib/python3.12/site-packages/sentry_sdk/integrations/stdlib.py", line 250, in sentry_patched_popen_communicate
return old_popen_communicate(self, *a, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/python@3.12/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/subprocess.py", line 1209, in communicate
stdout, stderr = self._communicate(input, endtime, timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/python@3.12/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/subprocess.py", line 2114, in _communicate
self._check_timeout(endtime, orig_timeout, stdout, stderr)
File "/opt/homebrew/Cellar/python@3.12/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/subprocess.py", line 1253, in _check_timeout
raise TimeoutExpired(
subprocess.TimeoutExpired: Command '['git', '-C', '/Users/admin/harvey/projects/shield-management/golf-rocket', 'pull', '--rebase']' timed out after 300 seconds
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/admin/git/personal/harvey/harvey/deployments.py", line 114, in run_deployment
webhook_config, webhook_output, start_time = Deployment.initialize_deployment(webhook)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/admin/git/personal/harvey/harvey/deployments.py", line 62, in initialize_deployment
git = Git.update_git_repo(webhook)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/admin/git/personal/harvey/harvey/git.py", line 23, in update_git_repo
output = Git.pull_repo(project_path, webhook)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/admin/git/personal/harvey/harvey/git.py", line 64, in pull_repo
kill_deployment(
File "/Users/admin/git/personal/harvey/harvey/utils/deployments.py", line 27, in kill_deployment
store_deployment_details(webhook, _strip_emojis_from_logs(deployment_logs))
File "/Users/admin/git/personal/harvey/harvey/repos/deployments.py", line 30, in store_deployment_details
with SqliteDict(filename=Config.database_file, tablename=DATABASE_TABLE_NAME) as database_table:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/admin/git/personal/harvey/venv/lib/python3.12/site-packages/sqlitedict.py", line 222, in __init__
self.conn = self._new_conn()
^^^^^^^^^^^^^^^^
File "/Users/admin/git/personal/harvey/venv/lib/python3.12/site-packages/sqlitedict.py", line 235, in _new_conn
return SqliteMultithread(
^^^^^^^^^^^^^^^^^^
File "/Users/admin/git/personal/harvey/venv/lib/python3.12/site-packages/sqlitedict.py", line 463, in __init__
self.start()
File "/Users/admin/git/personal/harvey/venv/lib/python3.12/site-packages/sentry_sdk/integrations/threading.py", line 56, in sentry_start
return old_start(self, *a, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/python@3.12/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/threading.py", line 992, in start
_start_new_thread(self._bootstrap, ())
RuntimeError: can't create new thread at interpreter shutdown
We should look into proper cleanup of threads on exit
Justintime50
commented
4 months ago
In the logs, this error message is popping up. It's due to sqlitedict trying to start a thread to store deployment details on a failed run (in this case due to git timeout due to waiting on the SSH key to be unlocked via user input), but the main thread has exited?
We should look into proper cleanup of threads on exit