cabernetwork / cabernet

Cabernet allows control of IPTV streams. Plugins supports DaddyLive, Pluto TV, XUMO, M3U/XMLTV.XML files (SamsungTV, STIRR, DistroTV, Plex TV)
https://cabernetwork.github.io
MIT License
200 stars 28 forks source link

[Bug] Python threading error when trying to set scheduled task intervals #118

Closed successtheman closed 11 months ago

successtheman commented 11 months ago

I get an error when trying to set intervals for the EPG updates in the webui, and I am not sure what is causing it so I thought I would share the log output here. It happens sometimes when trying to create a new task interval or delete an old one. It also seems to happen periodically when the task is supposed to actually run if I do somehow manage to get it to add a new task interval. I've attached the log output below and also indicated how to get the error to show in the screenshot.

I should note that it occurs with both the m3u plugin and plutotv as well

python_threadingerror_12-18-23.log

I am also pasting the log output here for convenience (Expand details to view it)

```python Exception in thread Thread-26: Traceback (most recent call last): File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner self.run() File "/app/lib/streams/m3u8_queue.py", line 81, in run m3u8_data = self.process_m3u8_item(self.queue_item) File "/app/lib/streams/m3u8_queue.py", line 201, in process_m3u8_item self.video.data = self.get_uri_data(data['uri'], HTTP_RETRIES) File "/app/lib/common/decorators.py", line 63, in wrapper_func x = f(self, *args, **kwargs) File "/app/lib/streams/m3u8_queue.py", line 100, in get_uri_data resp = M3U8Queue.http_session.get(_uri, headers=M3U8Queue.http_header, timeout=HTTP_TIMEOUT, follow_redirects=True) File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1041, in get return self.request( File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 814, in request return self.send(request, auth=auth, follow_redirects=follow_redirects) File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 901, in send response = self._send_handling_auth( File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 929, in _send_handling_auth response = self._send_handling_redirects( File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 966, in _send_handling_redirects response = self._send_single_request(request) File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1002, in _send_single_request response = transport.handle_request(request) File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 228, in handle_request resp = self._pool.handle_request(req) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection_pool.py", line 268, in handle_request raise exc File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection_pool.py", line 251, in handle_request response = connection.handle_request(request) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection.py", line 103, in handle_request return self._connection.handle_request(request) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 185, in handle_request raise exc File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 142, in handle_request self._send_request_headers(request=request, stream_id=stream_id) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 247, in _send_request_headers self._h2_state.send_headers(stream_id, headers, end_stream=end_stream) File "/usr/local/lib/python3.8/site-packages/h2/connection.py", line 770, in send_headers frames = stream.send_headers( File "/usr/local/lib/python3.8/site-packages/h2/stream.py", line 867, in send_headers frames = self._build_headers_frames( File "/usr/local/lib/python3.8/site-packages/h2/stream.py", line 1254, in _build_headers_frames encoded_headers = encoder.encode(headers) File "/usr/local/lib/python3.8/site-packages/hpack/hpack.py", line 255, in encode header_block.append(self.add(header, sensitive, huffman)) File "/usr/local/lib/python3.8/site-packages/hpack/hpack.py", line 280, in add match = self.header_table.search(name, value) File "/usr/local/lib/python3.8/site-packages/hpack/table.py", line 184, in search for (i, (n, v)) in enumerate(self.dynamic_entries): RuntimeError: deque mutated during iteration Exception in thread Thread-29: Traceback (most recent call last): File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner self.run() File "/app/lib/streams/m3u8_queue.py", line 81, in run m3u8_data = self.process_m3u8_item(self.queue_item) File "/app/lib/streams/m3u8_queue.py", line 201, in process_m3u8_item self.video.data = self.get_uri_data(data['uri'], HTTP_RETRIES) File "/app/lib/common/decorators.py", line 63, in wrapper_func x = f(self, *args, **kwargs) File "/app/lib/streams/m3u8_queue.py", line 100, in get_uri_data resp = M3U8Queue.http_session.get(_uri, headers=M3U8Queue.http_header, timeout=HTTP_TIMEOUT, follow_redirects=True) File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1041, in get return self.request( File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 814, in request return self.send(request, auth=auth, follow_redirects=follow_redirects) File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 901, in send response = self._send_handling_auth( File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 929, in _send_handling_auth response = self._send_handling_redirects( File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 966, in _send_handling_redirects response = self._send_single_request(request) File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1002, in _send_single_request response = transport.handle_request(request) File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 228, in handle_request resp = self._pool.handle_request(req) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection_pool.py", line 268, in handle_request raise exc File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection_pool.py", line 251, in handle_request response = connection.handle_request(request) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection.py", line 103, in handle_request return self._connection.handle_request(request) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 185, in handle_request raise exc File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 142, in handle_request self._send_request_headers(request=request, stream_id=stream_id) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 247, in _send_request_headers self._h2_state.send_headers(stream_id, headers, end_stream=end_stream) File "/usr/local/lib/python3.8/site-packages/h2/connection.py", line 770, in send_headers frames = stream.send_headers( File "/usr/local/lib/python3.8/site-packages/h2/stream.py", line 867, in send_headers frames = self._build_headers_frames( File "/usr/local/lib/python3.8/site-packages/h2/stream.py", line 1254, in _build_headers_frames encoded_headers = encoder.encode(headers) File "/usr/local/lib/python3.8/site-packages/hpack/hpack.py", line 255, in encode header_block.append(self.add(header, sensitive, huffman)) File "/usr/local/lib/python3.8/site-packages/hpack/hpack.py", line 280, in add match = self.header_table.search(name, value) File "/usr/local/lib/python3.8/site-packages/hpack/table.py", line 184, in search for (i, (n, v)) in enumerate(self.dynamic_entries): RuntimeError: deque mutated during iteration Exception in thread Thread-35: Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 148, in handle_request status, headers = self._receive_response( File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 292, in _receive_response event = self._receive_stream_event(request, stream_id) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 333, in _receive_stream_event self._receive_events(request, stream_id) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 352, in _receive_events raise RemoteProtocolError(self._connection_terminated) httpcore.RemoteProtocolError: During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner self.run() File "/app/lib/streams/m3u8_queue.py", line 81, in run m3u8_data = self.process_m3u8_item(self.queue_item) File "/app/lib/streams/m3u8_queue.py", line 201, in process_m3u8_item self.video.data = self.get_uri_data(data['uri'], HTTP_RETRIES) File "/app/lib/common/decorators.py", line 63, in wrapper_func x = f(self, *args, **kwargs) File "/app/lib/streams/m3u8_queue.py", line 100, in get_uri_data resp = M3U8Queue.http_session.get(_uri, headers=M3U8Queue.http_header, timeout=HTTP_TIMEOUT, follow_redirects=True) File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1041, in get return self.request( File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 814, in request return self.send(request, auth=auth, follow_redirects=follow_redirects) File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 901, in send response = self._send_handling_auth( File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 929, in _send_handling_auth response = self._send_handling_redirects( File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 966, in _send_handling_redirects response = self._send_single_request(request) File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1002, in _send_single_request response = transport.handle_request(request) File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 228, in handle_request resp = self._pool.handle_request(req) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection_pool.py", line 268, in handle_request raise exc File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection_pool.py", line 251, in handle_request response = connection.handle_request(request) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection.py", line 103, in handle_request return self._connection.handle_request(request) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 167, in handle_request self._response_closed(stream_id=stream_id) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 406, in _response_closed del self._events[stream_id] KeyError: 11 Exception in thread Thread-36: Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions yield File "/usr/local/lib/python3.8/site-packages/httpcore/_backends/sync.py", line 126, in read return self._sock.recv(max_bytes) File "/usr/local/lib/python3.8/ssl.py", line 1259, in recv return self.read(buflen) File "/usr/local/lib/python3.8/ssl.py", line 1134, in read return self._sslobj.read(len) socket.timeout: The read operation timed out The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 148, in handle_request status, headers = self._receive_response( File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 292, in _receive_response event = self._receive_stream_event(request, stream_id) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 333, in _receive_stream_event self._receive_events(request, stream_id) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 361, in _receive_events events = self._read_incoming_data(request) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 452, in _read_incoming_data raise exc File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 438, in _read_incoming_data data = self._network_stream.read(self.READ_NUM_BYTES, timeout) File "/usr/local/lib/python3.8/site-packages/httpcore/_backends/sync.py", line 126, in read return self._sock.recv(max_bytes) File "/usr/local/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/usr/local/lib/python3.8/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions raise to_exc(exc) from exc httpcore.ReadTimeout: The read operation timed out During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner self.run() File "/app/lib/streams/m3u8_queue.py", line 81, in run m3u8_data = self.process_m3u8_item(self.queue_item) File "/app/lib/streams/m3u8_queue.py", line 201, in process_m3u8_item self.video.data = self.get_uri_data(data['uri'], HTTP_RETRIES) File "/app/lib/common/decorators.py", line 63, in wrapper_func x = f(self, *args, **kwargs) File "/app/lib/streams/m3u8_queue.py", line 100, in get_uri_data resp = M3U8Queue.http_session.get(_uri, headers=M3U8Queue.http_header, timeout=HTTP_TIMEOUT, follow_redirects=True) File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1041, in get return self.request( File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 814, in request return self.send(request, auth=auth, follow_redirects=follow_redirects) File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 901, in send response = self._send_handling_auth( File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 929, in _send_handling_auth response = self._send_handling_redirects( File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 966, in _send_handling_redirects response = self._send_single_request(request) File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1002, in _send_single_request response = transport.handle_request(request) File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 228, in handle_request resp = self._pool.handle_request(req) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection_pool.py", line 268, in handle_request raise exc File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection_pool.py", line 251, in handle_request response = connection.handle_request(request) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection.py", line 103, in handle_request return self._connection.handle_request(request) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 167, in handle_request self._response_closed(stream_id=stream_id) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 406, in _response_closed del self._events[stream_id] KeyError: 11 Exception in thread Thread-2: Traceback (most recent call last): File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner self.run() File "/usr/local/lib/python3.8/threading.py", line 870, in run self._target(*self._args, **self._kwargs) File "/app/lib/schedule/scheduler.py", line 78, in _queue_thread self.process_queue(queue_item) File "/app/lib/schedule/scheduler.py", line 254, in process_queue self.add_trigger(_queue_item['trigger']) File "/app/lib/schedule/scheduler.py", line 298, in add_trigger self.create_trigger(trigger['area'], trigger['title'], File "/app/lib/schedule/scheduler.py", line 309, in create_trigger trigger = self.scheduler_db.get_trigger(uuid) File "/app/lib/db/db_scheduler.py", line 357, in get_trigger trigger = self.get_dict(DB_TRIGGER_TABLE + '_by_uuid', (_uuid,)) File "/app/lib/db/db.py", line 195, in get_dict self.sql_exec(sqlcmd, _where, cur) File "/app/lib/db/db.py", line 67, in sql_exec return _cursor.execute(_sqlcmd, _bindings) sqlite3.InterfaceError: Error binding parameter 0 - probably unsupported type. Exception in thread Thread-42: Traceback (most recent call last): File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner self.run() File "/app/lib/streams/m3u8_queue.py", line 81, in run m3u8_data = self.process_m3u8_item(self.queue_item) File "/app/lib/streams/m3u8_queue.py", line 201, in process_m3u8_item self.video.data = self.get_uri_data(data['uri'], HTTP_RETRIES) File "/app/lib/common/decorators.py", line 63, in wrapper_func x = f(self, *args, **kwargs) File "/app/lib/streams/m3u8_queue.py", line 100, in get_uri_data resp = M3U8Queue.http_session.get(_uri, headers=M3U8Queue.http_header, timeout=HTTP_TIMEOUT, follow_redirects=True) File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1041, in get return self.request( File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 814, in request return self.send(request, auth=auth, follow_redirects=follow_redirects) File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 901, in send response = self._send_handling_auth( File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 929, in _send_handling_auth response = self._send_handling_redirects( File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 966, in _send_handling_redirects response = self._send_single_request(request) File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1002, in _send_single_request response = transport.handle_request(request) File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 228, in handle_request resp = self._pool.handle_request(req) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection_pool.py", line 268, in handle_request raise exc File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection_pool.py", line 251, in handle_request response = connection.handle_request(request) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection.py", line 103, in handle_request return self._connection.handle_request(request) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 185, in handle_request raise exc File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 142, in handle_request self._send_request_headers(request=request, stream_id=stream_id) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 247, in _send_request_headers self._h2_state.send_headers(stream_id, headers, end_stream=end_stream) File "/usr/local/lib/python3.8/site-packages/h2/connection.py", line 760, in send_headers if (self.open_outbound_streams + 1) > max_open_streams: File "/usr/local/lib/python3.8/site-packages/h2/connection.py", line 419, in open_outbound_streams return self._open_streams(outbound_numbers) File "/usr/local/lib/python3.8/site-packages/h2/connection.py", line 408, in _open_streams stream = self.streams.pop(stream_id) KeyError: 7 Exception in thread Thread-41: Traceback (most recent call last): File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner self.run() File "/app/lib/streams/m3u8_queue.py", line 81, in run m3u8_data = self.process_m3u8_item(self.queue_item) File "/app/lib/streams/m3u8_queue.py", line 201, in process_m3u8_item self.video.data = self.get_uri_data(data['uri'], HTTP_RETRIES) File "/app/lib/common/decorators.py", line 63, in wrapper_func x = f(self, *args, **kwargs) File "/app/lib/streams/m3u8_queue.py", line 100, in get_uri_data resp = M3U8Queue.http_session.get(_uri, headers=M3U8Queue.http_header, timeout=HTTP_TIMEOUT, follow_redirects=True) File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1041, in get return self.request( File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 814, in request return self.send(request, auth=auth, follow_redirects=follow_redirects) File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 901, in send response = self._send_handling_auth( File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 929, in _send_handling_auth response = self._send_handling_redirects( File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 966, in _send_handling_redirects response = self._send_single_request(request) File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1002, in _send_single_request response = transport.handle_request(request) File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 228, in handle_request resp = self._pool.handle_request(req) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection_pool.py", line 268, in handle_request raise exc File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection_pool.py", line 251, in handle_request response = connection.handle_request(request) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection.py", line 103, in handle_request return self._connection.handle_request(request) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 185, in handle_request raise exc File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 142, in handle_request self._send_request_headers(request=request, stream_id=stream_id) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 247, in _send_request_headers self._h2_state.send_headers(stream_id, headers, end_stream=end_stream) File "/usr/local/lib/python3.8/site-packages/h2/connection.py", line 770, in send_headers frames = stream.send_headers( File "/usr/local/lib/python3.8/site-packages/h2/stream.py", line 867, in send_headers frames = self._build_headers_frames( File "/usr/local/lib/python3.8/site-packages/h2/stream.py", line 1254, in _build_headers_frames encoded_headers = encoder.encode(headers) File "/usr/local/lib/python3.8/site-packages/hpack/hpack.py", line 255, in encode header_block.append(self.add(header, sensitive, huffman)) File "/usr/local/lib/python3.8/site-packages/hpack/hpack.py", line 280, in add match = self.header_table.search(name, value) File "/usr/local/lib/python3.8/site-packages/hpack/table.py", line 184, in search for (i, (n, v)) in enumerate(self.dynamic_entries): RuntimeError: deque mutated during iteration Exception in thread Thread-40: Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions yield File "/usr/local/lib/python3.8/site-packages/httpcore/_backends/sync.py", line 126, in read return self._sock.recv(max_bytes) File "/usr/local/lib/python3.8/ssl.py", line 1259, in recv return self.read(buflen) File "/usr/local/lib/python3.8/ssl.py", line 1134, in read return self._sslobj.read(len) socket.timeout: The read operation timed out The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 148, in handle_request status, headers = self._receive_response( File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 292, in _receive_response event = self._receive_stream_event(request, stream_id) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 333, in _receive_stream_event self._receive_events(request, stream_id) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 361, in _receive_events events = self._read_incoming_data(request) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 452, in _read_incoming_data raise exc File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 438, in _read_incoming_data data = self._network_stream.read(self.READ_NUM_BYTES, timeout) File "/usr/local/lib/python3.8/site-packages/httpcore/_backends/sync.py", line 126, in read return self._sock.recv(max_bytes) File "/usr/local/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/usr/local/lib/python3.8/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions raise to_exc(exc) from exc httpcore.ReadTimeout: The read operation timed out During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner self.run() File "/app/lib/streams/m3u8_queue.py", line 81, in run m3u8_data = self.process_m3u8_item(self.queue_item) File "/app/lib/streams/m3u8_queue.py", line 201, in process_m3u8_item self.video.data = self.get_uri_data(data['uri'], HTTP_RETRIES) File "/app/lib/common/decorators.py", line 63, in wrapper_func x = f(self, *args, **kwargs) File "/app/lib/streams/m3u8_queue.py", line 100, in get_uri_data resp = M3U8Queue.http_session.get(_uri, headers=M3U8Queue.http_header, timeout=HTTP_TIMEOUT, follow_redirects=True) File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1041, in get return self.request( File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 814, in request return self.send(request, auth=auth, follow_redirects=follow_redirects) File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 901, in send response = self._send_handling_auth( File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 929, in _send_handling_auth response = self._send_handling_redirects( File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 966, in _send_handling_redirects response = self._send_single_request(request) File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1002, in _send_single_request response = transport.handle_request(request) File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 228, in handle_request resp = self._pool.handle_request(req) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection_pool.py", line 268, in handle_request raise exc File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection_pool.py", line 251, in handle_request response = connection.handle_request(request) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection.py", line 103, in handle_request return self._connection.handle_request(request) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 167, in handle_request self._response_closed(stream_id=stream_id) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 406, in _response_closed del self._events[stream_id] KeyError: 11 ^AException in thread Thread-58: Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 66, in map_httpcore_exceptions yield File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 228, in handle_request resp = self._pool.handle_request(req) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection_pool.py", line 268, in handle_request raise exc File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection_pool.py", line 251, in handle_request response = connection.handle_request(request) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection.py", line 103, in handle_request return self._connection.handle_request(request) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 185, in handle_request raise exc File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 148, in handle_request status, headers = self._receive_response( File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 292, in _receive_response event = self._receive_stream_event(request, stream_id) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 333, in _receive_stream_event self._receive_events(request, stream_id) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 361, in _receive_events events = self._read_incoming_data(request) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 435, in _read_incoming_data raise self._read_exception # pragma: nocover File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 66, in map_httpcore_exceptions yield File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 228, in handle_request resp = self._pool.handle_request(req) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection_pool.py", line 268, in handle_request raise exc File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection_pool.py", line 251, in handle_request response = connection.handle_request(request) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection.py", line 103, in handle_request return self._connection.handle_request(request) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 185, in handle_request raise exc File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 148, in handle_request status, headers = self._receive_response( File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 292, in _receive_response event = self._receive_stream_event(request, stream_id) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 333, in _receive_stream_event self._receive_events(request, stream_id) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 361, in _receive_events events = self._read_incoming_data(request) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 435, in _read_incoming_data raise self._read_exception # pragma: nocover File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 148, in handle_request status, headers = self._receive_response( File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 292, in _receive_response event = self._receive_stream_event(request, stream_id) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 333, in _receive_stream_event self._receive_events(request, stream_id) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 361, in _receive_events events = self._read_incoming_data(request) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 435, in _read_incoming_data raise self._read_exception # pragma: nocover File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 66, in map_httpcore_exceptions yield File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 228, in handle_request resp = self._pool.handle_request(req) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection_pool.py", line 268, in handle_request raise exc File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection_pool.py", line 251, in handle_request response = connection.handle_request(request) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection.py", line 103, in handle_request return self._connection.handle_request(request) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 185, in handle_request raise exc File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 148, in handle_request status, headers = self._receive_response( File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 292, in _receive_response event = self._receive_stream_event(request, stream_id) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 333, in _receive_stream_event self._receive_events(request, stream_id) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 361, in _receive_events events = self._read_incoming_data(request) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 452, in _read_incoming_data raise exc File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 440, in _read_incoming_data raise RemoteProtocolError("Server disconnected") httpcore.RemoteProtocolError: Server disconnected During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner self.run() File "/app/lib/streams/m3u8_queue.py", line 81, in run m3u8_data = self.process_m3u8_item(self.queue_item) File "/app/lib/streams/m3u8_queue.py", line 201, in process_m3u8_item self.video.data = self.get_uri_data(data['uri'], HTTP_RETRIES) File "/app/lib/common/decorators.py", line 63, in wrapper_func x = f(self, *args, **kwargs) File "/app/lib/streams/m3u8_queue.py", line 100, in get_uri_data resp = M3U8Queue.http_session.get(_uri, headers=M3U8Queue.http_header, timeout=HTTP_TIMEOUT, follow_redirects=True) File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1041, in get return self.request( File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 814, in request return self.send(request, auth=auth, follow_redirects=follow_redirects) File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 901, in send response = self._send_handling_auth( File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 929, in _send_handling_auth response = self._send_handling_redirects( File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 966, in _send_handling_redirects response = self._send_single_request(request) File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1002, in _send_single_request response = transport.handle_request(request) File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 228, in handle_request resp = self._pool.handle_request(req) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection_pool.py", line 268, in handle_request raise exc File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection_pool.py", line 251, in handle_request response = connection.handle_request(request) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection.py", line 103, in handle_request return self._connection.handle_request(request) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 167, in handle_request self._response_closed(stream_id=stream_id) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 406, in _response_closed del self._events[stream_id] KeyError: 5 Exception in thread Thread-59: Traceback (most recent call last): File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner self.run() File "/app/lib/streams/m3u8_queue.py", line 81, in run m3u8_data = self.process_m3u8_item(self.queue_item) File "/app/lib/streams/m3u8_queue.py", line 201, in process_m3u8_item self.video.data = self.get_uri_data(data['uri'], HTTP_RETRIES) File "/app/lib/common/decorators.py", line 63, in wrapper_func x = f(self, *args, **kwargs) File "/app/lib/streams/m3u8_queue.py", line 100, in get_uri_data resp = M3U8Queue.http_session.get(_uri, headers=M3U8Queue.http_header, timeout=HTTP_TIMEOUT, follow_redirects=True) File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1041, in get return self.request( File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 814, in request return self.send(request, auth=auth, follow_redirects=follow_redirects) File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 901, in send response = self._send_handling_auth( File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 929, in _send_handling_auth response = self._send_handling_redirects( File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 966, in _send_handling_redirects response = self._send_single_request(request) File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1002, in _send_single_request response = transport.handle_request(request) File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 228, in handle_request resp = self._pool.handle_request(req) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection_pool.py", line 268, in handle_request raise exc File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection_pool.py", line 251, in handle_request response = connection.handle_request(request) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/connection.py", line 103, in handle_request return self._connection.handle_request(request) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 185, in handle_request raise exc File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 142, in handle_request self._send_request_headers(request=request, stream_id=stream_id) File "/usr/local/lib/python3.8/site-packages/httpcore/_sync/http2.py", line 247, in _send_request_headers self._h2_state.send_headers(stream_id, headers, end_stream=end_stream) File "/usr/local/lib/python3.8/site-packages/h2/connection.py", line 770, in send_headers frames = stream.send_headers( File "/usr/local/lib/python3.8/site-packages/h2/stream.py", line 867, in send_headers frames = self._build_headers_frames( File "/usr/local/lib/python3.8/site-packages/h2/stream.py", line 1254, in _build_headers_frames encoded_headers = encoder.encode(headers) File "/usr/local/lib/python3.8/site-packages/hpack/hpack.py", line 255, in encode header_block.append(self.add(header, sensitive, huffman)) File "/usr/local/lib/python3.8/site-packages/hpack/hpack.py", line 280, in add match = self.header_table.search(name, value) File "/usr/local/lib/python3.8/site-packages/hpack/table.py", line 184, in search for (i, (n, v)) in enumerate(self.dynamic_entries): RuntimeError: deque mutated during iteration ```

Screenshot_20231218_122053

successtheman commented 11 months ago

I updated the above post adding the log contents in a collapsible details section for convenience

rocky4546 commented 11 months ago

The new HTTPX library is throwing an exception caused by the provider not responding to the request. I'll check and probably update the exceptions list of things to catch to include this exception. Can you provide the exact channel you were using?

successtheman commented 11 months ago

Well it's odd because it's not an exact channel, it happens when editing the task and adding an interval or attempting to delete said interval. However, I will try a bunch of channels and see if I can reproduce it from starting streams as well and report back

rocky4546 commented 11 months ago

The issue has nothing to do with the scheduler and everything to do with the streaming service. It might be a coincidence that it happened while you were doing something in the scheduler, but the log shows the errors were associated with streams. In all cases, the exception starts with /app/lib/streams/m3u8_queue.py file. This file is started in its own process and is totally separate from the scheduler. It is created when a channel is requested from internalproxy. The exceptions show that the trace does go through the decorator, which catches all bad exceptions, but with a new HTTPX lib, not all exceptions have been listed (especially those indirectly associated with HTTPX). Anyway, I will go over the exceptions found and add the ones needed. This will still cause the stream to not run, since the provider is unable to respond to requests, but it will not have the trace dump.

successtheman commented 11 months ago

The issue has nothing to do with the scheduler and everything to do with the streaming service. It might be a coincidence that it happened while you were doing something in the scheduler, but the log shows the errors were associated with streams. In all cases, the exception starts with /app/lib/streams/m3u8_queue.py file. This file is started in its own process and is totally separate from the scheduler. It is created when a channel is requested from internalproxy. The exceptions show that the trace does go through the decorator, which catches all bad exceptions, but with a new HTTPX lib, not all exceptions have been listed (especially those indirectly associated with HTTPX). Anyway, I will go over the exceptions found and add the ones needed. This will still cause the stream to not run, since the provider is unable to respond to requests, but it will not have the trace dump.

I was mistaken and posted the wrong error originally, the error when creating data update intervals is this one (happened as soon as I clicked create and tried to add a 3 hour update interval for Pluto TV while testing)

Exception in thread Thread-7:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/app/lib/schedule/scheduler.py", line 78, in _queue_thread
    self.process_queue(queue_item)
  File "/app/lib/schedule/scheduler.py", line 254, in process_queue
    self.add_trigger(_queue_item['trigger'])
  File "/app/lib/schedule/scheduler.py", line 298, in add_trigger
    self.create_trigger(trigger['area'], trigger['title'],
  File "/app/lib/schedule/scheduler.py", line 309, in create_trigger
    trigger = self.scheduler_db.get_trigger(uuid)
  File "/app/lib/db/db_scheduler.py", line 357, in get_trigger
    trigger = self.get_dict(DB_TRIGGER_TABLE + '_by_uuid', (_uuid,))
  File "/app/lib/db/db.py", line 195, in get_dict
    self.sql_exec(sqlcmd, _where, cur)
  File "/app/lib/db/db.py", line 67, in sql_exec
    return _cursor.execute(_sqlcmd, _bindings)
sqlite3.InterfaceError: Error binding parameter 0 - probably unsupported type.
rocky4546 commented 11 months ago

Fixed a number of the httpcore exceptions, but can't test it, so it should work, just not sure. As for creating trigger exception. I tracked it down to a change made 10 months ago. it has been fixed in RC08.