search

jooola / earhorn

Listen, monitor and archive your Icecast streams!
GNU General Public License v3.0
10 stars 4 forks source link

ReadTimeout stopps stream listener without restart #143

Closed paddatrapper closed 1 year ago

paddatrapper commented 2 years ago

It looks like there was an interruption in the stream that Earhorn was listening to, resulting in the following error. It did not attempt to restart listening.

2022-10-31 08:24:03.540 | DEBUG    | earhorn.stream_archive_s3:ingest_segment:33 - uploading segment incoming/segment.2022-10-31-08-22-33.mp3 to s3://uct-radio-archive-test                                                                   
2022-10-31 08:24:03.796 | DEBUG    | earhorn.stream:listen:112 - command exited with 0    
2022-10-31 08:24:04.684 | INFO     | earhorn.stream:listen:117 - stream listener stopped
Traceback (most recent call last):                                                                                     
  File "/opt/venv/lib/python3.10/site-packages/httpcore/_exceptions.py", line 8, in map_exceptions
    yield                                                                                                              
  File "/opt/venv/lib/python3.10/site-packages/httpcore/backends/sync.py", line 26, in read
    return self._sock.recv(max_bytes)                                                                                  
  File "/usr/local/lib/python3.10/ssl.py", line 1259, in recv
    return self.read(buflen)                                                                                           
  File "/usr/local/lib/python3.10/ssl.py", line 1132, in read
    return self._sslobj.read(len)                                                                                      
TimeoutError: The read operation timed out   

During handling of the above exception, another exception occurred:

Traceback (most recent call last):              
  File "/opt/venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield                          
  File "/opt/venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 218, in handle_request
    resp = self._pool.handle_request(req)
  File "/opt/venv/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 253, in handle_request   
    raise exc                         
  File "/opt/venv/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 237, in handle_request
    response = connection.handle_request(request)          
  File "/opt/venv/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 90, in handle_request                                                                                                                                       
    return self._connection.handle_request(request)
  File "/opt/venv/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 105, in handle_request                                                                                                                                          
    raise exc                                                                                                                                                                                                                                  
  File "/opt/venv/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 84, in handle_request
    ) = self._receive_response_headers(**kwargs)
  File "/opt/venv/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 148, in _receive_response_headers
    event = self._receive_event(timeout=timeout)
  File "/opt/venv/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 177, in _receive_event
    data = self._network_stream.read(
  File "/opt/venv/lib/python3.10/site-packages/httpcore/backends/sync.py", line 24, in read
    with map_exceptions(exc_map):
  File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/opt/venv/lib/python3.10/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions
    raise to_exc(exc)
httpcore.ReadTimeout: The read operation timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/venv/bin/earhorn", line 8, in <module>
    sys.exit(cli())
  File "/opt/venv/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/opt/venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/earhorn/main.py", line 251, in cli
    stream_listener.run_forever()
  File "/opt/venv/lib/python3.10/site-packages/earhorn/stream.py", line 83, in run_forever
    self.check_stream()
  File "/opt/venv/lib/python3.10/site-packages/earhorn/stream.py", line 69, in check_stream
    with self._client.stream("GET", self.stream_url) as response:
  File "/usr/local/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/opt/venv/lib/python3.10/site-packages/httpx/_client.py", line 858, in stream
    response = self.send(
  File "/opt/venv/lib/python3.10/site-packages/httpx/_client.py", line 902, in send
    response = self._send_handling_auth(
  File "/opt/venv/lib/python3.10/site-packages/httpx/_client.py", line 930, in _send_handling_auth
    response = self._send_handling_redirects(
  File "/opt/venv/lib/python3.10/site-packages/httpx/_client.py", line 967, in _send_handling_redirects
    response = self._send_single_request(request)
  File "/opt/venv/lib/python3.10/site-packages/httpx/_client.py", line 1003, in _send_single_request
    response = transport.handle_request(request)
  File "/opt/venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 217, in handle_request
    with map_httpcore_exceptions():
  File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/opt/venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadTimeout: The read operation timed out

This is running v0.13.0 via Docker

jooola commented 2 years ago

I guess we are missing a timeout exception catching here https://github.com/jooola/earhorn/blob/10977511df40f028fcce3e35a235057ba158be13/earhorn/stream.py#L75

I will add one of the parent exception to make sure we catch most them https://www.python-httpx.org/exceptions/#the-exception-hierarchy