int-brain-lab / ibllib

IBL core shared libraries
MIT License
61 stars 36 forks source link

on-call: Fix DLC task #846

Closed mayofaulkner closed 1 month ago

mayofaulkner commented 1 month ago

The DLC tasks have been failing with the following error

warnings.warn('`pipes.training_preprocessing` to be removed in favour of dynamic pipeline', FutureWarning)
/home/ibladmin/Documents/PYTHON/envs/dlcenv/lib/python3.10/site-packages/ibllib/pipes/ephys_preprocessing.py:43: FutureWarning: `pipes.ephys_preprocessing` to be removed in favour of dynamic pipeline
warnings.warn('`pipes.ephys_preprocessing` to be removed in favour of dynamic pipeline', FutureWarning)
HTTP Error 404: Not Found https://ibl.flatironinstitute.org/resources/dlc/weights_v2.2.1.zip
Traceback (most recent call last):
File "/home/ibladmin/Documents/PYTHON/iblscripts/deploy/serverpc/dlc/run_dlc.py", line 13, in <module>
path_dlc = download_weights(version=__version__)
File "/home/ibladmin/Documents/PYTHON/iblvideo/iblvideo/weights.py", line 31, in download_weights
file_name, hash = one.alyx.download_file(url, target_dir=weights_path, return_md5=True, silent=True)
File "/home/ibladmin/Documents/PYTHON/envs/dlcenv/lib/python3.10/site-packages/one/webclient.py", line 776, in download_file
raise ex
File "/home/ibladmin/Documents/PYTHON/envs/dlcenv/lib/python3.10/site-packages/one/webclient.py", line 771, in download_file
files = download_fcn(url, **pars)
File "/home/ibladmin/Documents/PYTHON/envs/dlcenv/lib/python3.10/site-packages/one/webclient.py", line 394, in http_download_file
raise e
File "/home/ibladmin/Documents/PYTHON/envs/dlcenv/lib/python3.10/site-packages/one/webclient.py", line 391, in http_download_file
u = urllib.request.urlopen(req)
File "/usr/local/lib/python3.10/urllib/request.py", line 216, in urlopen
return opener.open(url, data, timeout)
File "/usr/local/lib/python3.10/urllib/request.py", line 525, in open
response = meth(req, response)
File "/usr/local/lib/python3.10/urllib/request.py", line 634, in http_response
response = self.parent.error(
File "/usr/local/lib/python3.10/urllib/request.py", line 557, in error
result = self._call_chain(*args)
File "/usr/local/lib/python3.10/urllib/request.py", line 496, in _call_chain
result = func(*args)
File "/usr/local/lib/python3.10/urllib/request.py", line 1056, in http_error_401
response = self.http_error_auth_reqed('www-authenticate',
File "/usr/local/lib/python3.10/urllib/request.py", line 1005, in http_error_auth_reqed
return self.retry_http_basic_auth(host, req, realm)
File "/usr/local/lib/python3.10/urllib/request.py", line 1020, in retry_http_basic_auth
return self.parent.open(req, timeout=req.timeout)
File "/usr/local/lib/python3.10/urllib/request.py", line 525, in open
response = meth(req, response)
File "/usr/local/lib/python3.10/urllib/request.py", line 634, in http_response
response = self.parent.error(
File "/usr/local/lib/python3.10/urllib/request.py", line 563, in error
return self._call_chain(*args)
File "/usr/local/lib/python3.10/urllib/request.py", line 496, in _call_chain
result = func(*args)
File "/usr/local/lib/python3.10/urllib/request.py", line 643, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found

The weight file that is downloaded is linked to the version of the iblvideo repo

This commit changed the version and since then the tasks have failed.

Need to adapt this file https://github.com/int-brain-lab/iblvideo/blob/master/iblvideo/weights.py to correctly get the weights file that exists

bimac commented 1 month ago

https://github.com/int-brain-lab/iblrig/issues/719