search

beeldengeluid / dane-video-segmentation-worker

0 stars 0 forks source link

various fixes #66

Closed jblom closed 4 months ago

jblom commented 5 months ago

should fix:

2024-07-05 08:58:25,604|INFO|7|io_util|delete_input_file|191|Verifying deletion of input file: /data/input-files/2101608170158176431__NOS_JOURNAAL_-WON01513227.mp4
2024-07-05 08:58:25,620|INFO|7|io_util|delete_input_file|199|Deleted VisXP input file: /data/input-files/2101608170158176431__NOS_JOURNAAL_-WON01513227.mp4
2024-07-05 08:58:25,621|ERROR|7|io_util|delete_input_file|212|OSError while removing empty input file dirs
Traceback (most recent call last):
File "/src/io_util.py", line 207, in delete_input_file
os.removedirs(
File "<frozen os>", line 243, in removedirs
OSError: [Errno 22] Invalid argument: '.'
2024-07-05 08:58:25,622|INFO|7|worker|callback|79|applying IO on output went well, now finally saving to DANE index
2024-07-05 08:58:25,622|INFO|7|worker|save_to_dane_index|97|saving results to DANE, task id=21caef8d2ffae61c100f0090da592d526674c0f9
jblom commented 5 months ago

not sure yet why this timeout occurred (maybe it's not every time):

Traceback (most recent call last):
File "/src/.venv/lib/python3.11/site-packages/elasticsearch7/connection/http_urllib3.py", line 255, in perform_request
response = self.pool.urlopen(
^^^^^^^^^^^^^^^^^^
File "/src/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 799, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File "/src/.venv/lib/python3.11/site-packages/urllib3/util/retry.py", line 525, in increment
raise six.reraise(type(error), error, _stacktrace)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/src/.venv/lib/python3.11/site-packages/urllib3/packages/six.py", line 770, in reraise
raise value
File "/src/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 715, in urlopen
httplib_response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/src/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 469, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/src/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 358, in _raise_timeout
raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='esc-es-http.dane-es.svc.cluster.local', port=9200): Read timed out. (read timeout=30)
2024-07-05 08:59:18,958|INFO|7|base|log_request_success|265|POST http://esc-es-http.dane-es.svc.cluster.local:9200/dane-index-visxp/_doc?refresh=true&routing=21caef8d2ffae61c100f0090da592d526674c0f9 [status:201 request:23.269s]
2024-07-05 08:59:18,958|INFO|7|base_classes|_ack_with_status_msg|297|Send ACK + msg back to queue (async)

@Veldhoen I did add a try, except around the self.save_to_dane_index in the worker.py to at least catch any errors related to saving back to dane

jblom commented 5 months ago

@Veldhoen I also updated to python 3.11 I hope you have it on your system?