search

openzim / zimfarm

Farm operated by bots to grow and harvest new zim files
https://farm.openzim.org
GNU General Public License v3.0
81 stars 24 forks source link

StackExchanger watcher regularly restarts after connection reset by peer errors #980

Open benoit74 opened 1 month ago

benoit74 commented 1 month ago

StackExchanger watcher regularly restarts after connection reset by peer errors

It looks like intermittent issue on an upstream (archive.org? wasabi?)

Sample occurence:

Common labels: {"cluster":"kiwix-prod","container":"zimfarm-watcher","flags":"F","job":"zimfarm/zimfarm-watcher","namespace":"zimfarm","pod":"watcher-deployment-54cf77666f-k79zm","service_name":"zimfarm-watcher"}
Line limit: "1000 (379 displayed)"
Total bytes processed: "33.4  kB"

2024-06-03 08:56:03.893 [2024-06-03 06:56:03,762::INFO] Getting list of SE domains
2024-06-03 08:56:07.699 [2024-06-03 06:56:07,699::INFO] Grabbing online versions and comparing with S3…
2024-06-03 09:01:18.439 [2024-06-03 07:01:18,439::ERROR] FAILED. An error occurred: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2024-06-03 09:01:19.269 [2024-06-03 07:01:18,439::ERROR] ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2024-06-03 09:01:19.269 Traceback (most recent call last):
2024-06-03 09:01:19.269   File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 793, in urlopen
2024-06-03 09:01:19.269     response = self._make_request(
2024-06-03 09:01:19.269                ^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.269   File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 491, in _make_request
2024-06-03 09:01:19.269     raise new_e
2024-06-03 09:01:19.269   File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 467, in _make_request
2024-06-03 09:01:19.269     self._validate_conn(conn)
2024-06-03 09:01:19.269   File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
2024-06-03 09:01:19.269     conn.connect()
2024-06-03 09:01:19.269   File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 653, in connect
2024-06-03 09:01:19.270     sock_and_verified = _ssl_wrap_socket_and_match_hostname(
2024-06-03 09:01:19.270                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270   File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 806, in _ssl_wrap_socket_and_match_hostname
2024-06-03 09:01:19.270     ssl_sock = ssl_wrap_socket(
2024-06-03 09:01:19.270                ^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270   File "/usr/local/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 465, in ssl_wrap_socket
2024-06-03 09:01:19.270     ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
2024-06-03 09:01:19.270                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270   File "/usr/local/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 509, in _ssl_wrap_socket_impl
2024-06-03 09:01:19.270     return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
2024-06-03 09:01:19.270            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270   File "/usr/local/lib/python3.12/ssl.py", line 455, in wrap_socket
2024-06-03 09:01:19.270     return self.sslsocket_class._create(
2024-06-03 09:01:19.270            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270   File "/usr/local/lib/python3.12/ssl.py", line 1042, in _create
2024-06-03 09:01:19.270     self.do_handshake()
2024-06-03 09:01:19.270   File "/usr/local/lib/python3.12/ssl.py", line 1320, in do_handshake
2024-06-03 09:01:19.270     self._sslobj.do_handshake()
2024-06-03 09:01:19.270 ConnectionResetError: [Errno 104] Connection reset by peer
2024-06-03 09:01:19.270 
2024-06-03 09:01:19.270 During handling of the above exception, another exception occurred:
2024-06-03 09:01:19.270 
2024-06-03 09:01:19.270 Traceback (most recent call last):
2024-06-03 09:01:19.270   File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 564, in send
2024-06-03 09:01:19.270     resp = conn.urlopen(
2024-06-03 09:01:19.270            ^^^^^^^^^^^^^
2024-06-03 09:01:19.270   File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 847, in urlopen
2024-06-03 09:01:19.270     retries = retries.increment(
2024-06-03 09:01:19.270               ^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270   File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 470, in increment
2024-06-03 09:01:19.270     raise reraise(type(error), error, _stacktrace)
2024-06-03 09:01:19.270           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270   File "/usr/local/lib/python3.12/site-packages/urllib3/util/util.py", line 38, in reraise
2024-06-03 09:01:19.270     raise value.with_traceback(tb)
2024-06-03 09:01:19.270   File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 793, in urlopen
2024-06-03 09:01:19.270     response = self._make_request(
2024-06-03 09:01:19.270                ^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270   File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 491, in _make_request
2024-06-03 09:01:19.270     raise new_e
2024-06-03 09:01:19.270   File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 467, in _make_request
2024-06-03 09:01:19.270     self._validate_conn(conn)
2024-06-03 09:01:19.270   File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
2024-06-03 09:01:19.270     conn.connect()
2024-06-03 09:01:19.270   File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 653, in connect
2024-06-03 09:01:19.270     sock_and_verified = _ssl_wrap_socket_and_match_hostname(
2024-06-03 09:01:19.270                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270   File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 806, in _ssl_wrap_socket_and_match_hostname
2024-06-03 09:01:19.270     ssl_sock = ssl_wrap_socket(
2024-06-03 09:01:19.270                ^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270   File "/usr/local/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 465, in ssl_wrap_socket
2024-06-03 09:01:19.270     ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
2024-06-03 09:01:19.270                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270   File "/usr/local/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 509, in _ssl_wrap_socket_impl
2024-06-03 09:01:19.270     return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
2024-06-03 09:01:19.270            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270   File "/usr/local/lib/python3.12/ssl.py", line 455, in wrap_socket
2024-06-03 09:01:19.270     return self.sslsocket_class._create(
2024-06-03 09:01:19.270            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270   File "/usr/local/lib/python3.12/ssl.py", line 1042, in _create
2024-06-03 09:01:19.270     self.do_handshake()
2024-06-03 09:01:19.270   File "/usr/local/lib/python3.12/ssl.py", line 1320, in do_handshake
2024-06-03 09:01:19.270     self._sslobj.do_handshake()
2024-06-03 09:01:19.270 urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2024-06-03 09:01:19.270 
2024-06-03 09:01:19.270 During handling of the above exception, another exception occurred:
2024-06-03 09:01:19.270 
2024-06-03 09:01:19.270 Traceback (most recent call last):
2024-06-03 09:01:19.270   File "/usr/local/bin/watcher", line 583, in entrypoint
2024-06-03 09:01:19.270     sys.exit(runner.run())
2024-06-03 09:01:19.270              ^^^^^^^^^^^^
2024-06-03 09:01:19.270   File "/usr/local/bin/watcher", line 494, in run
2024-06-03 09:01:19.270     self.check_and_go()
2024-06-03 09:01:19.270   File "/usr/local/bin/watcher", line 427, in check_and_go
2024-06-03 09:01:19.270     last_modified = get_last_modified_for(url)
2024-06-03 09:01:19.270                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270   File "/usr/local/bin/watcher", line 68, in get_last_modified_for
2024-06-03 09:01:19.270     with requests.head(url, allow_redirects=True) as resp:
2024-06-03 09:01:19.270          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270   File "/usr/local/lib/python3.12/site-packages/requests/api.py", line 100, in head
2024-06-03 09:01:19.270     return request("head", url, **kwargs)
2024-06-03 09:01:19.270            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270   File "/usr/local/lib/python3.12/site-packages/requests/api.py", line 59, in request
2024-06-03 09:01:19.270     return session.request(method=method, url=url, **kwargs)
2024-06-03 09:01:19.270            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270   File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
2024-06-03 09:01:19.270     resp = self.send(prep, **send_kwargs)
2024-06-03 09:01:19.270            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270   File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
2024-06-03 09:01:19.270     r = adapter.send(request, **kwargs)
2024-06-03 09:01:19.270         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270   File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 579, in send
2024-06-03 09:01:19.270     raise ConnectionError(err, request=request)
2024-06-03 09:01:19.270 requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2024-06-03 09:01:55.223 [entrypoint] exposing s3_url secret as S3_URL
2024-06-03 09:01:55.356 [entrypoint] exposing zimfarm_username secret as ZIMFARM_USERNAME
2024-06-03 09:01:55.357 [entrypoint] exposing zimfarm_password secret as ZIMFARM_PASSWORD
2024-06-03 09:02:16.063 [2024-06-03 07:02:16,063::INFO] 
2024-06-03 09:02:16.063   __                                         _       _
2024-06-03 09:02:16.063  / _| __ _ _ __ _ __ ___      __      ____ _| |_ ___| |__   ___ _ __
2024-06-03 09:02:16.063 | |_ / _` | '__| '_ ` _ \ ____\ \ /\ / / _` | __/ __| '_ \ / _ \ '__|
2024-06-03 09:02:16.063 |  _| (_| | |  | | | | | |_____\ V  V / (_| | || (__| | | |  __/ |
2024-06-03 09:02:16.063 |_|  \__,_|_|  |_| |_| |_|      \_/\_/ \__,_|\__\___|_| |_|\___|_|
2024-06-03 09:02:16.063 
2024-06-03 09:02:16.063 
2024-06-03 09:02:16.063 [2024-06-03 07:02:16,063::INFO] Testing S3 credentials
2024-06-03 09:02:40.652 [2024-06-03 07:02:40,652::INFO] Testing Zimfarm credentials with https://api.farm.openzim.org/v1…
2024-06-03 09:02:42.048 [2024-06-03 07:02:42,048::INFO] Starting watcher:
2024-06-03 09:02:42.048   with zimfarm username: stackwatcher
2024-06-03 09:02:42.048   using cache: s3.us-west-1.wasabisys.com
2024-06-03 09:02:42.048   with bucket: org-kiwix-stackexchange
2024-06-03 09:02:42.048 [2024-06-03 07:02:42,048::INFO] Getting list of SE domains
2024-06-03 09:02:44.595 [2024-06-03 07:02:44,594::INFO] Grabbing online versions and comparing with S3…
2024-06-03 09:03:30.530 [2024-06-03 07:03:30,530::ERROR] FAILED. An error occurred: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2024-06-03 09:03:30.549 [2024-06-03 07:03:30,530::ERROR] ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2024-06-03 09:03:30.549 Traceback (most recent call last):
2024-06-03 09:03:30.549   File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 793, in urlopen
2024-06-03 09:03:30.549     response = self._make_request(
2024-06-03 09:03:30.549                ^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.549   File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 491, in _make_request
2024-06-03 09:03:30.549     raise new_e
2024-06-03 09:03:30.549   File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 467, in _make_request
2024-06-03 09:03:30.549     self._validate_conn(conn)
2024-06-03 09:03:30.549   File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
2024-06-03 09:03:30.549     conn.connect()
2024-06-03 09:03:30.549   File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 653, in connect
2024-06-03 09:03:30.549     sock_and_verified = _ssl_wrap_socket_and_match_hostname(
2024-06-03 09:03:30.549                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.549   File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 806, in _ssl_wrap_socket_and_match_hostname
2024-06-03 09:03:30.549     ssl_sock = ssl_wrap_socket(
2024-06-03 09:03:30.549                ^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.549   File "/usr/local/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 465, in ssl_wrap_socket
2024-06-03 09:03:30.549     ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
2024-06-03 09:03:30.550                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550   File "/usr/local/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 509, in _ssl_wrap_socket_impl
2024-06-03 09:03:30.550     return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
2024-06-03 09:03:30.550            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550   File "/usr/local/lib/python3.12/ssl.py", line 455, in wrap_socket
2024-06-03 09:03:30.550     return self.sslsocket_class._create(
2024-06-03 09:03:30.550            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550   File "/usr/local/lib/python3.12/ssl.py", line 1042, in _create
2024-06-03 09:03:30.550     self.do_handshake()
2024-06-03 09:03:30.550   File "/usr/local/lib/python3.12/ssl.py", line 1320, in do_handshake
2024-06-03 09:03:30.550     self._sslobj.do_handshake()
2024-06-03 09:03:30.550 ConnectionResetError: [Errno 104] Connection reset by peer
2024-06-03 09:03:30.550 
2024-06-03 09:03:30.550 During handling of the above exception, another exception occurred:
2024-06-03 09:03:30.550 
2024-06-03 09:03:30.550 Traceback (most recent call last):
2024-06-03 09:03:30.550   File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 564, in send
2024-06-03 09:03:30.550     resp = conn.urlopen(
2024-06-03 09:03:30.550            ^^^^^^^^^^^^^
2024-06-03 09:03:30.550   File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 847, in urlopen
2024-06-03 09:03:30.550     retries = retries.increment(
2024-06-03 09:03:30.550               ^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550   File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 470, in increment
2024-06-03 09:03:30.550     raise reraise(type(error), error, _stacktrace)
2024-06-03 09:03:30.550           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550   File "/usr/local/lib/python3.12/site-packages/urllib3/util/util.py", line 38, in reraise
2024-06-03 09:03:30.550     raise value.with_traceback(tb)
2024-06-03 09:03:30.550   File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 793, in urlopen
2024-06-03 09:03:30.550     response = self._make_request(
2024-06-03 09:03:30.550                ^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550   File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 491, in _make_request
2024-06-03 09:03:30.550     raise new_e
2024-06-03 09:03:30.550   File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 467, in _make_request
2024-06-03 09:03:30.550     self._validate_conn(conn)
2024-06-03 09:03:30.550   File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
2024-06-03 09:03:30.550     conn.connect()
2024-06-03 09:03:30.550   File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 653, in connect
2024-06-03 09:03:30.550     sock_and_verified = _ssl_wrap_socket_and_match_hostname(
2024-06-03 09:03:30.550                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550   File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 806, in _ssl_wrap_socket_and_match_hostname
2024-06-03 09:03:30.550     ssl_sock = ssl_wrap_socket(
2024-06-03 09:03:30.550                ^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550   File "/usr/local/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 465, in ssl_wrap_socket
2024-06-03 09:03:30.550     ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
2024-06-03 09:03:30.550                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550   File "/usr/local/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 509, in _ssl_wrap_socket_impl
2024-06-03 09:03:30.550     return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
2024-06-03 09:03:30.550            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550   File "/usr/local/lib/python3.12/ssl.py", line 455, in wrap_socket
2024-06-03 09:03:30.550     return self.sslsocket_class._create(
2024-06-03 09:03:30.550            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550   File "/usr/local/lib/python3.12/ssl.py", line 1042, in _create
2024-06-03 09:03:30.550     self.do_handshake()
2024-06-03 09:03:30.550   File "/usr/local/lib/python3.12/ssl.py", line 1320, in do_handshake
2024-06-03 09:03:30.550     self._sslobj.do_handshake()
2024-06-03 09:03:30.550 urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2024-06-03 09:03:30.550 
2024-06-03 09:03:30.550 During handling of the above exception, another exception occurred:
2024-06-03 09:03:30.550 
2024-06-03 09:03:30.550 Traceback (most recent call last):
2024-06-03 09:03:30.550   File "/usr/local/bin/watcher", line 583, in entrypoint
2024-06-03 09:03:30.550     sys.exit(runner.run())
2024-06-03 09:03:30.550              ^^^^^^^^^^^^
2024-06-03 09:03:30.550   File "/usr/local/bin/watcher", line 490, in run
2024-06-03 09:03:30.550     self.check_and_go()
2024-06-03 09:03:30.550   File "/usr/local/bin/watcher", line 427, in check_and_go
2024-06-03 09:03:30.550     last_modified = get_last_modified_for(url)
2024-06-03 09:03:30.550                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550   File "/usr/local/bin/watcher", line 68, in get_last_modified_for
2024-06-03 09:03:30.550     with requests.head(url, allow_redirects=True) as resp:
2024-06-03 09:03:30.550          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550   File "/usr/local/lib/python3.12/site-packages/requests/api.py", line 100, in head
2024-06-03 09:03:30.550     return request("head", url, **kwargs)
2024-06-03 09:03:30.550            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550   File "/usr/local/lib/python3.12/site-packages/requests/api.py", line 59, in request
2024-06-03 09:03:30.550     return session.request(method=method, url=url, **kwargs)
2024-06-03 09:03:30.550            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550   File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
2024-06-03 09:03:30.550     resp = self.send(prep, **send_kwargs)
2024-06-03 09:03:30.550            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550   File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
2024-06-03 09:03:30.551     r = adapter.send(request, **kwargs)
2024-06-03 09:03:30.551         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.551   File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 579, in send
2024-06-03 09:03:30.551     raise ConnectionError(err, request=request)
2024-06-03 09:03:30.551 requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2024-06-03 09:04:19.656 [entrypoint] exposing s3_url secret as S3_URL
2024-06-03 09:04:19.657 [entrypoint] exposing zimfarm_username secret as ZIMFARM_USERNAME
2024-06-03 09:04:19.658 [entrypoint] exposing zimfarm_password secret as ZIMFARM_PASSWORD
2024-06-03 09:04:22.099 [2024-06-03 07:04:22,098::INFO] 
2024-06-03 09:04:22.298   __                                         _       _
2024-06-03 09:04:22.298  / _| __ _ _ __ _ __ ___      __      ____ _| |_ ___| |__   ___ _ __
2024-06-03 09:04:22.298 | |_ / _` | '__| '_ ` _ \ ____\ \ /\ / / _` | __/ __| '_ \ / _ \ '__|
2024-06-03 09:04:22.298 |  _| (_| | |  | | | | | |_____\ V  V / (_| | || (__| | | |  __/ |
2024-06-03 09:04:22.298 |_|  \__,_|_|  |_| |_| |_|      \_/\_/ \__,_|\__\___|_| |_|\___|_|
2024-06-03 09:04:22.298 
2024-06-03 09:04:22.298 
2024-06-03 09:04:22.298 [2024-06-03 07:04:22,099::INFO] Testing S3 credentials
2024-06-03 09:04:27.539 [2024-06-03 07:04:27,539::INFO] Testing Zimfarm credentials with https://api.farm.openzim.org/v1…
2024-06-03 09:04:27.651 [2024-06-03 07:04:27,651::INFO] Starting watcher:
2024-06-03 09:04:27.651   with zimfarm username: stackwatcher
2024-06-03 09:04:27.651   using cache: s3.us-west-1.wasabisys.com
2024-06-03 09:04:27.651   with bucket: org-kiwix-stackexchange
2024-06-03 09:04:27.651 [2024-06-03 07:04:27,651::INFO] Getting list of SE domains
2024-06-03 09:04:28.826 [2024-06-03 07:04:28,826::INFO] Grabbing online versions and comparing with S3…
2024-06-03 09:10:17.017 [2024-06-03 07:10:17,017::ERROR] FAILED. An error occurred: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2024-06-03 09:10:17.534 [2024-06-03 07:10:17,017::ERROR] ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2024-06-03 09:10:17.534 Traceback (most recent call last):
2024-06-03 09:10:17.534   File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 793, in urlopen
2024-06-03 09:10:17.534     response = self._make_request(
2024-06-03 09:10:17.534                ^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534   File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 491, in _make_request
2024-06-03 09:10:17.534     raise new_e
2024-06-03 09:10:17.534   File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 467, in _make_request
2024-06-03 09:10:17.534     self._validate_conn(conn)
2024-06-03 09:10:17.534   File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
2024-06-03 09:10:17.534     conn.connect()
2024-06-03 09:10:17.534   File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 653, in connect
2024-06-03 09:10:17.534     sock_and_verified = _ssl_wrap_socket_and_match_hostname(
2024-06-03 09:10:17.534                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534   File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 806, in _ssl_wrap_socket_and_match_hostname
2024-06-03 09:10:17.534     ssl_sock = ssl_wrap_socket(
2024-06-03 09:10:17.534                ^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534   File "/usr/local/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 465, in ssl_wrap_socket
2024-06-03 09:10:17.534     ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
2024-06-03 09:10:17.534                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534   File "/usr/local/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 509, in _ssl_wrap_socket_impl
2024-06-03 09:10:17.534     return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
2024-06-03 09:10:17.534            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534   File "/usr/local/lib/python3.12/ssl.py", line 455, in wrap_socket
2024-06-03 09:10:17.534     return self.sslsocket_class._create(
2024-06-03 09:10:17.534            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534   File "/usr/local/lib/python3.12/ssl.py", line 1042, in _create
2024-06-03 09:10:17.534     self.do_handshake()
2024-06-03 09:10:17.534   File "/usr/local/lib/python3.12/ssl.py", line 1320, in do_handshake
2024-06-03 09:10:17.534     self._sslobj.do_handshake()
2024-06-03 09:10:17.534 ConnectionResetError: [Errno 104] Connection reset by peer
2024-06-03 09:10:17.534 
2024-06-03 09:10:17.534 During handling of the above exception, another exception occurred:
2024-06-03 09:10:17.534 
2024-06-03 09:10:17.534 Traceback (most recent call last):
2024-06-03 09:10:17.534   File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 564, in send
2024-06-03 09:10:17.534     resp = conn.urlopen(
2024-06-03 09:10:17.534            ^^^^^^^^^^^^^
2024-06-03 09:10:17.534   File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 847, in urlopen
2024-06-03 09:10:17.534     retries = retries.increment(
2024-06-03 09:10:17.534               ^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534   File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 470, in increment
2024-06-03 09:10:17.534     raise reraise(type(error), error, _stacktrace)
2024-06-03 09:10:17.534           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534   File "/usr/local/lib/python3.12/site-packages/urllib3/util/util.py", line 38, in reraise
2024-06-03 09:10:17.534     raise value.with_traceback(tb)
2024-06-03 09:10:17.534   File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 793, in urlopen
2024-06-03 09:10:17.534     response = self._make_request(
2024-06-03 09:10:17.534                ^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534   File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 491, in _make_request
2024-06-03 09:10:17.534     raise new_e
2024-06-03 09:10:17.534   File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 467, in _make_request
2024-06-03 09:10:17.534     self._validate_conn(conn)
2024-06-03 09:10:17.534   File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
2024-06-03 09:10:17.534     conn.connect()
2024-06-03 09:10:17.534   File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 653, in connect
2024-06-03 09:10:17.534     sock_and_verified = _ssl_wrap_socket_and_match_hostname(
2024-06-03 09:10:17.534                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534   File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 806, in _ssl_wrap_socket_and_match_hostname
2024-06-03 09:10:17.534     ssl_sock = ssl_wrap_socket(
2024-06-03 09:10:17.534                ^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534   File "/usr/local/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 465, in ssl_wrap_socket
2024-06-03 09:10:17.534     ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
2024-06-03 09:10:17.534                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534   File "/usr/local/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 509, in _ssl_wrap_socket_impl
2024-06-03 09:10:17.534     return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
2024-06-03 09:10:17.534            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534   File "/usr/local/lib/python3.12/ssl.py", line 455, in wrap_socket
2024-06-03 09:10:17.534     return self.sslsocket_class._create(
2024-06-03 09:10:17.534            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534   File "/usr/local/lib/python3.12/ssl.py", line 1042, in _create
2024-06-03 09:10:17.534     self.do_handshake()
2024-06-03 09:10:17.534   File "/usr/local/lib/python3.12/ssl.py", line 1320, in do_handshake
2024-06-03 09:10:17.534     self._sslobj.do_handshake()
2024-06-03 09:10:17.534 urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2024-06-03 09:10:17.534 
2024-06-03 09:10:17.534 During handling of the above exception, another exception occurred:
2024-06-03 09:10:17.534 
2024-06-03 09:10:17.534 Traceback (most recent call last):
2024-06-03 09:10:17.534   File "/usr/local/bin/watcher", line 583, in entrypoint
2024-06-03 09:10:17.534     sys.exit(runner.run())
2024-06-03 09:10:17.534              ^^^^^^^^^^^^
2024-06-03 09:10:17.534   File "/usr/local/bin/watcher", line 490, in run
2024-06-03 09:10:17.534     self.check_and_go()
2024-06-03 09:10:17.534   File "/usr/local/bin/watcher", line 427, in check_and_go
2024-06-03 09:10:17.534     last_modified = get_last_modified_for(url)
2024-06-03 09:10:17.534                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534   File "/usr/local/bin/watcher", line 68, in get_last_modified_for
2024-06-03 09:10:17.534     with requests.head(url, allow_redirects=True) as resp:
2024-06-03 09:10:17.534          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534   File "/usr/local/lib/python3.12/site-packages/requests/api.py", line 100, in head
2024-06-03 09:10:17.534     return request("head", url, **kwargs)
2024-06-03 09:10:17.534            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534   File "/usr/local/lib/python3.12/site-packages/requests/api.py", line 59, in request
2024-06-03 09:10:17.534     return session.request(method=method, url=url, **kwargs)
2024-06-03 09:10:17.534            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534   File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
2024-06-03 09:10:17.534     resp = self.send(prep, **send_kwargs)
2024-06-03 09:10:17.534            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534   File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
2024-06-03 09:10:17.534     r = adapter.send(request, **kwargs)
2024-06-03 09:10:17.534         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.535   File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 579, in send
2024-06-03 09:10:17.535     raise ConnectionError(err, request=request)
2024-06-03 09:10:17.535 requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2024-06-03 09:11:02.072 [entrypoint] exposing s3_url secret as S3_URL
2024-06-03 09:11:02.074 [entrypoint] exposing zimfarm_username secret as ZIMFARM_USERNAME
2024-06-03 09:11:02.075 [entrypoint] exposing zimfarm_password secret as ZIMFARM_PASSWORD
2024-06-03 09:11:19.567 [2024-06-03 07:11:19,567::INFO] 
2024-06-03 09:11:19.567   __                                         _       _
2024-06-03 09:11:19.567  / _| __ _ _ __ _ __ ___      __      ____ _| |_ ___| |__   ___ _ __
2024-06-03 09:11:19.567 | |_ / _` | '__| '_ ` _ \ ____\ \ /\ / / _` | __/ __| '_ \ / _ \ '__|
2024-06-03 09:11:19.567 |  _| (_| | |  | | | | | |_____\ V  V / (_| | || (__| | | |  __/ |
2024-06-03 09:11:19.567 |_|  \__,_|_|  |_| |_| |_|      \_/\_/ \__,_|\__\___|_| |_|\___|_|
2024-06-03 09:11:19.567 
2024-06-03 09:11:19.567 
2024-06-03 09:11:19.567 [2024-06-03 07:11:19,567::INFO] Testing S3 credentials
2024-06-03 09:11:35.117 [2024-06-03 07:11:35,116::INFO] Testing Zimfarm credentials with https://api.farm.openzim.org/v1…
2024-06-03 09:11:35.275 [2024-06-03 07:11:35,275::INFO] Starting watcher:
2024-06-03 09:11:35.275   with zimfarm username: stackwatcher
2024-06-03 09:11:35.275   using cache: s3.us-west-1.wasabisys.com
2024-06-03 09:11:35.275   with bucket: org-kiwix-stackexchange
2024-06-03 09:11:35.275 [2024-06-03 07:11:35,275::INFO] Getting list of SE domains
2024-06-03 09:11:37.900 [2024-06-03 07:11:37,899::INFO] Grabbing online versions and comparing with S3…
2024-06-03 09:21:05.560 [2024-06-03 07:21:05,463::INFO] All synced up.
2024-06-03 09:21:05.560 [2024-06-03 07:21:05,560::INFO] Done.

Not sure something is needed to be done since the system recover nicely ; maybe just avoid to restart completely the process.

benoit74 commented 1 month ago

List of all restarts in the last 30 days (not filtered by cause, some of them are expected due to updates typically)

2024-05-07 17:29:05
2024-05-13 10:38:41
2024-05-14 09:56:57
2024-05-14 11:17:08
2024-05-14 12:42:25
2024-05-14 13:52:54
2024-05-14 14:08:04
2024-05-15 04:08:13
2024-05-27 04:30:34
2024-05-28 08:12:46
2024-05-29 08:14:52
2024-05-30 08:16:38
2024-05-31 08:19:12
2024-06-01 08:20:42
2024-06-01 08:22:34
2024-06-01 08:24:48
2024-06-01 08:34:15
2024-06-01 08:44:18
2024-06-01 08:45:28
2024-06-01 08:55:54
2024-06-03 09:02:16
2024-06-03 09:04:22
2024-06-03 09:11:19