search

Project-MONAI / tutorials

MONAI Tutorials
https://monai.io/started.html
Apache License 2.0
1.85k stars 682 forks source link

Invaild data link for IXI Dataset #1801

Closed KumoLiu closed 2 months ago

KumoLiu commented 2 months ago 
[2024-08-24T23:11:39.897Z] Running ./3d_regression/densenet_training_array.ipynb
[2024-08-24T23:11:39.897Z] Checking PEP8 compliance...
[2024-08-24T23:11:40.457Z] Running notebook...
[2024-08-24T23:11:40.457Z] Before:
[2024-08-24T23:11:40.457Z]     "max_epochs = 5\n",
[2024-08-24T23:11:40.713Z] After:
[2024-08-24T23:11:40.713Z]     "max_epochs = 1\n",
[2024-08-24T23:11:40.713Z] Before:
[2024-08-24T23:11:40.713Z]     "val_interval = 2\n",
[2024-08-24T23:11:40.713Z] After:
[2024-08-24T23:11:40.713Z]     "val_interval = 1\n",
[2024-08-24T23:11:43.978Z] /usr/local/lib/python3.10/dist-packages/ignite/handlers/checkpoint.py:17: DeprecationWarning: `TorchScript` support for functional optimizers is deprecated and will be removed in a future PyTorch release. Consider using the `torch.compile` optimizer instead.
[2024-08-24T23:11:43.978Z]   from torch.distributed.optim import ZeroRedundancyOptimizer
[2024-08-24T23:11:48.132Z] MONAI version: 1.4.0rc2+2.ga5fbe716
[2024-08-24T23:11:48.132Z] Numpy version: 1.24.4
[2024-08-24T23:11:48.132Z] Pytorch version: 2.4.0+cu121
[2024-08-24T23:11:48.132Z] MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
[2024-08-24T23:11:48.132Z] MONAI rev id: a5fbe716378948630783deef8ee435e7e3bdc918
[2024-08-24T23:11:48.132Z] MONAI __file__: /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/__init__.py
[2024-08-24T23:11:48.132Z] 
[2024-08-24T23:11:48.132Z] Optional dependencies:
[2024-08-24T23:11:48.132Z] Pytorch Ignite version: 0.4.11
[2024-08-24T23:11:48.132Z] ITK version: 5.4.0
[2024-08-24T23:11:48.132Z] Nibabel version: 5.2.1
[2024-08-24T23:11:48.132Z] scikit-image version: 0.23.2
[2024-08-24T23:11:48.132Z] scipy version: 1.12.0
[2024-08-24T23:11:48.132Z] Pillow version: 10.2.0
[2024-08-24T23:11:48.132Z] Tensorboard version: 2.17.1
[2024-08-24T23:11:48.132Z] gdown version: 5.2.0
[2024-08-24T23:11:48.132Z] TorchVision version: 0.19.0+cu121
[2024-08-24T23:11:48.132Z] tqdm version: 4.66.2
[2024-08-24T23:11:48.132Z] lmdb version: 1.5.1
[2024-08-24T23:11:48.132Z] psutil version: 5.9.4
[2024-08-24T23:11:48.132Z] pandas version: 1.5.3
[2024-08-24T23:11:48.132Z] einops version: 0.7.0
[2024-08-24T23:11:48.132Z] transformers version: 4.40.2
[2024-08-24T23:11:48.132Z] mlflow version: 2.15.1
[2024-08-24T23:11:48.132Z] pynrrd version: 1.0.0
[2024-08-24T23:11:48.132Z] clearml version: 1.16.3
[2024-08-24T23:11:48.132Z] 
[2024-08-24T23:11:48.132Z] For details about installing the optional dependencies, please visit:
[2024-08-24T23:11:48.132Z]     https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies
[2024-08-24T23:11:48.132Z] 
[2024-08-24T23:11:50.016Z] papermill  --progress-bar --log-output -k python3
[2024-08-24T23:11:50.016Z] /usr/local/lib/python3.10/dist-packages/papermill/iorw.py:149: UserWarning: the file is not specified with any extension : -
[2024-08-24T23:11:50.016Z]   warnings.warn(f"the file is not specified with any extension : {os.path.basename(path)}")
[2024-08-24T23:12:26.667Z] 
Executing:   0%|          | 0/15 [00:00<?, ?cell/s]
Executing:   7%|▋         | 1/15 [00:00<00:12,  1.17cell/s]
Executing:  20%|██        | 3/15 [00:09<00:39,  3.33s/cell]
Executing:  33%|███▎      | 5/15 [00:16<00:33,  3.37s/cell]
Executing:  60%|██████    | 9/15 [00:33<00:24,  4.01s/cell]
Executing:  60%|██████    | 9/15 [00:36<00:24,  4.02s/cell]
[2024-08-24T23:12:26.667Z] /usr/local/lib/python3.10/dist-packages/papermill/iorw.py:149: UserWarning: the file is not specified with any extension : -
[2024-08-24T23:12:26.667Z]   warnings.warn(f"the file is not specified with any extension : {os.path.basename(path)}")
[2024-08-24T23:12:26.667Z] Traceback (most recent call last):
[2024-08-24T23:12:26.667Z]   File "/usr/local/bin/papermill", line 8, in <module>
[2024-08-24T23:12:26.667Z]     sys.exit(papermill())
[2024-08-24T23:12:26.667Z]   File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1157, in __call__
[2024-08-24T23:12:26.667Z]     return self.main(*args, **kwargs)
[2024-08-24T23:12:26.667Z]   File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1078, in main
[2024-08-24T23:12:26.667Z]     rv = self.invoke(ctx)
[2024-08-24T23:12:26.667Z]   File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1434, in invoke
[2024-08-24T23:12:26.667Z]     return ctx.invoke(self.callback, **ctx.params)
[2024-08-24T23:12:26.667Z]   File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 783, in invoke
[2024-08-24T23:12:26.667Z]     return __callback(*args, **kwargs)
[2024-08-24T23:12:26.667Z]   File "/usr/local/lib/python3.10/dist-packages/click/decorators.py", line 33, in new_func
[2024-08-24T23:12:26.667Z]     return f(get_current_context(), *args, **kwargs)
[2024-08-24T23:12:26.667Z]   File "/usr/local/lib/python3.10/dist-packages/papermill/cli.py", line 235, in papermill
[2024-08-24T23:12:26.667Z]     execute_notebook(
[2024-08-24T23:12:26.667Z]   File "/usr/local/lib/python3.10/dist-packages/papermill/execute.py", line 131, in execute_notebook
[2024-08-24T23:12:26.667Z]     raise_for_execution_errors(nb, output_path)
[2024-08-24T23:12:26.667Z]   File "/usr/local/lib/python3.10/dist-packages/papermill/execute.py", line 251, in raise_for_execution_errors
[2024-08-24T23:12:26.667Z]     raise error
[2024-08-24T23:12:26.667Z] papermill.exceptions.PapermillExecutionError: 
[2024-08-24T23:12:26.667Z] ---------------------------------------------------------------------------
[2024-08-24T23:12:26.667Z] Exception encountered at "In [5]":
[2024-08-24T23:12:26.667Z] ---------------------------------------------------------------------------
[2024-08-24T23:12:26.667Z] HTTPError                                 Traceback (most recent call last)
[2024-08-24T23:12:26.667Z] Cell In[5], line 8
[2024-08-24T23:12:26.667Z]       5 dataset_dir = os.path.join(root_dir, "ixi")
[2024-08-24T23:12:26.667Z]       6 tarfile_name = f"{dataset_dir}.tar"
[2024-08-24T23:12:26.667Z] ----> 8 download_and_extract(resource, tarfile_name, dataset_dir, md5)
[2024-08-24T23:12:26.667Z] 
[2024-08-24T23:12:26.667Z] File /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/apps/utils.py:335, in download_and_extract(url, filepath, output_dir, hash_val, hash_type, file_type, has_base, progress)
[2024-08-24T23:12:26.667Z]     333 with tempfile.TemporaryDirectory() as tmp_dir:
[2024-08-24T23:12:26.667Z]     334     filename = filepath or Path(tmp_dir, _basename(url)).resolve()
[2024-08-24T23:12:26.667Z] --> 335     download_url(url=url, filepath=filename, hash_val=hash_val, hash_type=hash_type, progress=progress)
[2024-08-24T23:12:26.667Z]     336     extractall(filepath=filename, output_dir=output_dir, file_type=file_type, has_base=has_base)
[2024-08-24T23:12:26.667Z] 
[2024-08-24T23:12:26.667Z] File /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/apps/utils.py:226, in download_url(url, filepath, hash_val, hash_type, progress, **gdown_kwargs)
[2024-08-24T23:12:26.667Z]     221             raise RuntimeError(
[2024-08-24T23:12:26.667Z]     222                 f"Download of file from {download_url}, received from {url} "
[2024-08-24T23:12:26.667Z]     223                 + f" to {filepath} failed due to network issue or denied permission."
[2024-08-24T23:12:26.667Z]     224             )
[2024-08-24T23:12:26.667Z]     225 else:
[2024-08-24T23:12:26.667Z] --> 226     _download_with_progress(url, tmp_name, progress=progress)
[2024-08-24T23:12:26.667Z]     227 if not tmp_name.exists():
[2024-08-24T23:12:26.667Z]     228     raise RuntimeError(
[2024-08-24T23:12:26.667Z]     229         f"Download of file from {url} to {filepath} failed due to network issue or denied permission."
[2024-08-24T23:12:26.667Z]     230     )
[2024-08-24T23:12:26.667Z] 
[2024-08-24T23:12:26.667Z] File /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/apps/utils.py:119, in _download_with_progress(url, filepath, progress)
[2024-08-24T23:12:26.667Z]     117 except (URLError, HTTPError, ContentTooShortError, OSError) as e:
[2024-08-24T23:12:26.667Z]     118     logger.error(f"Download failed from {url} to {filepath}.")
[2024-08-24T23:12:26.667Z] --> 119     raise e
[2024-08-24T23:12:26.667Z] 
[2024-08-24T23:12:26.667Z] File /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/apps/utils.py:112, in _download_with_progress(url, filepath, progress)
[2024-08-24T23:12:26.667Z]     109             self.update(b * bsize - self.n)  # will also set self.n = b * bsize
[2024-08-24T23:12:26.667Z]     111     with TqdmUpTo(unit="B", unit_scale=True, unit_divisor=1024, miniters=1, desc=_basename(filepath)) as t:
[2024-08-24T23:12:26.667Z] --> 112         urlretrieve(url, filepath, reporthook=t.update_to)
[2024-08-24T23:12:26.667Z]     113 else:
[2024-08-24T23:12:26.667Z]     114     if not has_tqdm and progress:
[2024-08-24T23:12:26.667Z] 
[2024-08-24T23:12:26.667Z] File /usr/lib/python3.10/urllib/request.py:241, in urlretrieve(url, filename, reporthook, data)
[2024-08-24T23:12:26.667Z]     224 """
[2024-08-24T23:12:26.667Z]     225 Retrieve a URL into a temporary location on disk.
[2024-08-24T23:12:26.667Z]     226 
[2024-08-24T23:12:26.667Z]    (...)
[2024-08-24T23:12:26.667Z]     237 data file as well as the resulting HTTPMessage object.
[2024-08-24T23:12:26.667Z]     238 """
[2024-08-24T23:12:26.667Z]     239 url_type, path = _splittype(url)
[2024-08-24T23:12:26.667Z] --> 241 with contextlib.closing(urlopen(url, data)) as fp:
[2024-08-24T23:12:26.667Z]     242     headers = fp.info()
[2024-08-24T23:12:26.667Z]     244     # Just return the local path and the "headers" for file://
[2024-08-24T23:12:26.667Z]     245     # URLs. No sense in performing a copy unless requested.
[2024-08-24T23:12:26.667Z] 
[2024-08-24T23:12:26.667Z] File /usr/lib/python3.10/urllib/request.py:216, in urlopen(url, data, timeout, cafile, capath, cadefault, context)
[2024-08-24T23:12:26.667Z]     214 else:
[2024-08-24T23:12:26.667Z]     215     opener = _opener
[2024-08-24T23:12:26.667Z] --> 216 return opener.open(url, data, timeout)
[2024-08-24T23:12:26.667Z] 
[2024-08-24T23:12:26.667Z] File /usr/lib/python3.10/urllib/request.py:525, in OpenerDirector.open(self, fullurl, data, timeout)
[2024-08-24T23:12:26.667Z]     523 for processor in self.process_response.get(protocol, []):
[2024-08-24T23:12:26.667Z]     524     meth = getattr(processor, meth_name)
[2024-08-24T23:12:26.667Z] --> 525     response = meth(req, response)
[2024-08-24T23:12:26.667Z]     527 return response
[2024-08-24T23:12:26.667Z] 
[2024-08-24T23:12:26.667Z] File /usr/lib/python3.10/urllib/request.py:634, in HTTPErrorProcessor.http_response(self, request, response)
[2024-08-24T23:12:26.667Z]     631 # According to RFC 2616, "2xx" code indicates that the client's
[2024-08-24T23:12:26.667Z]     632 # request was successfully received, understood, and accepted.
[2024-08-24T23:12:26.667Z]     633 if not (200 <= code < 300):
[2024-08-24T23:12:26.667Z] --> 634     response = self.parent.error(
[2024-08-24T23:12:26.667Z]     635         'http', request, response, code, msg, hdrs)
[2024-08-24T23:12:26.667Z]     637 return response
[2024-08-24T23:12:26.667Z] 
[2024-08-24T23:12:26.667Z] File /usr/lib/python3.10/urllib/request.py:557, in OpenerDirector.error(self, proto, *args)
[2024-08-24T23:12:26.667Z]     555     http_err = 0
[2024-08-24T23:12:26.667Z]     556 args = (dict, proto, meth_name) + args
[2024-08-24T23:12:26.667Z] --> 557 result = self._call_chain(*args)
[2024-08-24T23:12:26.667Z]     558 if result:
[2024-08-24T23:12:26.667Z]     559     return result
[2024-08-24T23:12:26.667Z] 
[2024-08-24T23:12:26.667Z] File /usr/lib/python3.10/urllib/request.py:496, in OpenerDirector._call_chain(self, chain, kind, meth_name, *args)
[2024-08-24T23:12:26.667Z]     494 for handler in handlers:
[2024-08-24T23:12:26.667Z]     495     func = getattr(handler, meth_name)
[2024-08-24T23:12:26.667Z] --> 496     result = func(*args)
[2024-08-24T23:12:26.667Z]     497     if result is not None:
[2024-08-24T23:12:26.667Z]     498         return result
[2024-08-24T23:12:26.667Z] 
[2024-08-24T23:12:26.667Z] File /usr/lib/python3.10/urllib/request.py:749, in HTTPRedirectHandler.http_error_302(self, req, fp, code, msg, headers)
[2024-08-24T23:12:26.667Z]     746 fp.read()
[2024-08-24T23:12:26.667Z]     747 fp.close()
[2024-08-24T23:12:26.667Z] --> 749 return self.parent.open(new, timeout=req.timeout)
[2024-08-24T23:12:26.667Z] 
[2024-08-24T23:12:26.667Z] File /usr/lib/python3.10/urllib/request.py:525, in OpenerDirector.open(self, fullurl, data, timeout)
[2024-08-24T23:12:26.667Z]     523 for processor in self.process_response.get(protocol, []):
[2024-08-24T23:12:26.667Z]     524     meth = getattr(processor, meth_name)
[2024-08-24T23:12:26.667Z] --> 525     response = meth(req, response)
[2024-08-24T23:12:26.667Z]     527 return response
[2024-08-24T23:12:26.667Z] 
[2024-08-24T23:12:26.667Z] File /usr/lib/python3.10/urllib/request.py:634, in HTTPErrorProcessor.http_response(self, request, response)
[2024-08-24T23:12:26.667Z]     631 # According to RFC 2616, "2xx" code indicates that the client's
[2024-08-24T23:12:26.667Z]     632 # request was successfully received, understood, and accepted.
[2024-08-24T23:12:26.667Z]     633 if not (200 <= code < 300):
[2024-08-24T23:12:26.667Z] --> 634     response = self.parent.error(
[2024-08-24T23:12:26.667Z]     635         'http', request, response, code, msg, hdrs)
[2024-08-24T23:12:26.667Z]     637 return response
[2024-08-24T23:12:26.667Z] 
[2024-08-24T23:12:26.668Z] File /usr/lib/python3.10/urllib/request.py:563, in OpenerDirector.error(self, proto, *args)
[2024-08-24T23:12:26.668Z]     561 if http_err:
[2024-08-24T23:12:26.668Z]     562     args = (dict, 'default', 'http_error_default') + orig_args
[2024-08-24T23:12:26.668Z] --> 563     return self._call_chain(*args)
[2024-08-24T23:12:26.668Z] 
[2024-08-24T23:12:26.668Z] File /usr/lib/python3.10/urllib/request.py:496, in OpenerDirector._call_chain(self, chain, kind, meth_name, *args)
[2024-08-24T23:12:26.668Z]     494 for handler in handlers:
[2024-08-24T23:12:26.668Z]     495     func = getattr(handler, meth_name)
[2024-08-24T23:12:26.668Z] --> 496     result = func(*args)
[2024-08-24T23:12:26.668Z]     497     if result is not None:
[2024-08-24T23:12:26.668Z]     498         return result
[2024-08-24T23:12:26.668Z] 
[2024-08-24T23:12:26.668Z] File /usr/lib/python3.10/urllib/request.py:643, in HTTPDefaultErrorHandler.http_error_default(self, req, fp, code, msg, hdrs)
[2024-08-24T23:12:26.668Z]     642 def http_error_default(self, req, fp, code, msg, hdrs):
[2024-08-24T23:12:26.668Z] --> 643     raise HTTPError(req.full_url, code, msg, hdrs, fp)
[2024-08-24T23:12:26.668Z] 
[2024-08-24T23:12:26.668Z] HTTPError: HTTP Error 403: Forbidden
KumoLiu commented 2 months ago

Also effect ./3d_regression/densenet_training_array.ipynb ./3d_classification/densenet_training_array.ipynb ./modules/public_datasets.ipynb

KumoLiu commented 2 months ago

Seems the owner changed the permission and we can't access the data anymore although the link is still on their website. Ref: https://brain-development.org/ixi-dataset/ Hi @zephyrie @Nic-Ma, do you think we can make a mirror for this data?