search

Project-MONAI / tutorials

MONAI Tutorials
https://monai.io/started.html
Apache License 2.0
1.85k stars 682 forks source link

pathology/nuclick/nuclick_infer.ipynb #1542

Open wyli opened 1 year ago

wyli commented 1 year ago

https://warwick.ac.uk/fac/cross_fac/tia/data/hovernet/consep_dataset.zip not available

[2023-10-05T01:48:32.818Z] Running ./pathology/nuclick/nuclick_infer.ipynb
[2023-10-05T01:48:32.818Z] Checking PEP8 compliance...
[2023-10-05T01:48:33.381Z] Running notebook...
[2023-10-05T01:48:33.942Z] Before:
[2023-10-05T01:48:33.942Z]     "max_epochs = 1"
[2023-10-05T01:48:34.197Z] After:
[2023-10-05T01:48:34.197Z]     "max_epochs = 1"
[2023-10-05T01:48:44.136Z] MONAI version: 1.3.0rc2+14.g141bcf0a
[2023-10-05T01:48:44.136Z] Numpy version: 1.22.2
[2023-10-05T01:48:44.136Z] Pytorch version: 2.1.0a0+29c30b1
[2023-10-05T01:48:44.136Z] MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
[2023-10-05T01:48:44.136Z] MONAI rev id: 141bcf0a20c46c505cdd1f722bdd5cb5c856033d
[2023-10-05T01:48:44.136Z] MONAI __file__: /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/__init__.py
[2023-10-05T01:48:44.136Z] 
[2023-10-05T01:48:44.136Z] Optional dependencies:
[2023-10-05T01:48:44.136Z] Pytorch Ignite version: 0.4.11
[2023-10-05T01:48:44.136Z] ITK version: 5.3.0
[2023-10-05T01:48:44.136Z] Nibabel version: 5.1.0
[2023-10-05T01:48:44.136Z] scikit-image version: 0.22.0
[2023-10-05T01:48:44.136Z] scipy version: 1.11.1
[2023-10-05T01:48:44.136Z] Pillow version: 9.2.0
[2023-10-05T01:48:44.136Z] Tensorboard version: 2.9.0
[2023-10-05T01:48:44.136Z] gdown version: 4.7.1
[2023-10-05T01:48:44.136Z] TorchVision version: 0.16.0a0
[2023-10-05T01:48:44.136Z] tqdm version: 4.65.0
[2023-10-05T01:48:44.136Z] lmdb version: 1.4.1
[2023-10-05T01:48:44.136Z] psutil version: 5.9.4
[2023-10-05T01:48:44.136Z] pandas version: 1.5.2
[2023-10-05T01:48:44.136Z] einops version: 0.6.1
[2023-10-05T01:48:44.136Z] transformers version: 4.21.3
[2023-10-05T01:48:44.136Z] mlflow version: 2.7.1
[2023-10-05T01:48:44.136Z] pynrrd version: 1.0.0
[2023-10-05T01:48:44.136Z] clearml version: 1.13.1
[2023-10-05T01:48:44.136Z] 
[2023-10-05T01:48:44.136Z] For details about installing the optional dependencies, please visit:
[2023-10-05T01:48:44.136Z]     https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies
[2023-10-05T01:48:44.136Z] 
[2023-10-05T01:48:45.500Z] papermill  --progress-bar -k python3
[2023-10-05T01:48:46.060Z] /usr/local/lib/python3.10/dist-packages/papermill/iorw.py:153: UserWarning: the file is not specified with any extension : -
[2023-10-05T01:48:46.060Z]   warnings.warn(
[2023-10-05T01:49:00.131Z] 
Executing:   0%|          | 0/18 [00:00<?, ?cell/s]
Executing:   6%|▌         | 1/18 [00:01<00:21,  1.25s/cell]
Executing:  17%|█▋        | 3/18 [00:02<00:09,  1.57cell/s]
Executing:  28%|██▊       | 5/18 [00:07<00:22,  1.77s/cell]
Executing:  44%|████▍     | 8/18 [00:10<00:12,  1.25s/cell]
Executing:  50%|█████     | 9/18 [00:10<00:09,  1.03s/cell]
Executing:  67%|██████▋   | 12/18 [00:12<00:05,  1.09cell/s]
Executing:  67%|██████▋   | 12/18 [00:14<00:07,  1.18s/cell]
[2023-10-05T01:49:00.387Z] /usr/local/lib/python3.10/dist-packages/papermill/iorw.py:153: UserWarning: the file is not specified with any extension : -
[2023-10-05T01:49:00.387Z]   warnings.warn(
[2023-10-05T01:49:00.387Z] Traceback (most recent call last):
[2023-10-05T01:49:00.387Z]   File "/usr/local/bin/papermill", line 8, in <module>
[2023-10-05T01:49:00.387Z]     sys.exit(papermill())
[2023-10-05T01:49:00.387Z]   File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1157, in __call__
[2023-10-05T01:49:00.387Z]     return self.main(*args, **kwargs)
[2023-10-05T01:49:00.387Z]   File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1078, in main
[2023-10-05T01:49:00.387Z]     rv = self.invoke(ctx)
[2023-10-05T01:49:00.387Z]   File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1434, in invoke
[2023-10-05T01:49:00.387Z]     return ctx.invoke(self.callback, **ctx.params)
[2023-10-05T01:49:00.387Z]   File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 783, in invoke
[2023-10-05T01:49:00.387Z]     return __callback(*args, **kwargs)
[2023-10-05T01:49:00.387Z]   File "/usr/local/lib/python3.10/dist-packages/click/decorators.py", line 33, in new_func
[2023-10-05T01:49:00.387Z]     return f(get_current_context(), *args, **kwargs)
[2023-10-05T01:49:00.387Z]   File "/usr/local/lib/python3.10/dist-packages/papermill/cli.py", line 250, in papermill
[2023-10-05T01:49:00.387Z]     execute_notebook(
[2023-10-05T01:49:00.387Z]   File "/usr/local/lib/python3.10/dist-packages/papermill/execute.py", line 128, in execute_notebook
[2023-10-05T01:49:00.387Z]     raise_for_execution_errors(nb, output_path)
[2023-10-05T01:49:00.387Z]   File "/usr/local/lib/python3.10/dist-packages/papermill/execute.py", line 232, in raise_for_execution_errors
[2023-10-05T01:49:00.387Z]     raise error
[2023-10-05T01:49:00.387Z] papermill.exceptions.PapermillExecutionError: 
[2023-10-05T01:49:00.387Z] ---------------------------------------------------------------------------
[2023-10-05T01:49:00.387Z] Exception encountered at "In [7]":
[2023-10-05T01:49:00.387Z] ---------------------------------------------------------------------------
[2023-10-05T01:49:00.387Z] BadZipFile                                Traceback (most recent call last)
[2023-10-05T01:49:00.387Z] Cell In[7], line 5
[2023-10-05T01:49:00.387Z]       2 consep_dir = os.path.join(workspace_path, "CoNSeP")
[2023-10-05T01:49:00.387Z]       4 if not os.path.exists(consep_dir):
[2023-10-05T01:49:00.387Z] ----> 5     download_and_extract(consep_zip, output_dir=workspace_path)
[2023-10-05T01:49:00.387Z] 
[2023-10-05T01:49:00.387Z] File /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/apps/utils.py:329, in download_and_extract(url, filepath, output_dir, hash_val, hash_type, file_type, has_base, progress)
[2023-10-05T01:49:00.387Z]     327 filename = filepath or Path(tmp_dir, _basename(url)).resolve()
[2023-10-05T01:49:00.387Z]     328 download_url(url=url, filepath=filename, hash_val=hash_val, hash_type=hash_type, progress=progress)
[2023-10-05T01:49:00.387Z] --> 329 extractall(filepath=filename, output_dir=output_dir, file_type=file_type, has_base=has_base)
[2023-10-05T01:49:00.387Z] 
[2023-10-05T01:49:00.387Z] File /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/apps/utils.py:283, in extractall(filepath, output_dir, hash_val, hash_type, file_type, has_base)
[2023-10-05T01:49:00.387Z]     281 _file_type = file_type.lower().strip()
[2023-10-05T01:49:00.387Z]     282 if filepath.name.endswith("zip") or _file_type == "zip":
[2023-10-05T01:49:00.387Z] --> 283     zip_file = zipfile.ZipFile(filepath)
[2023-10-05T01:49:00.387Z]     284     zip_file.extractall(output_dir)
[2023-10-05T01:49:00.387Z]     285     zip_file.close()
[2023-10-05T01:49:00.387Z] 
[2023-10-05T01:49:00.387Z] File /usr/lib/python3.10/zipfile.py:1269, in ZipFile.__init__(self, file, mode, compression, allowZip64, compresslevel, strict_timestamps)
[2023-10-05T01:49:00.387Z]    1267 try:
[2023-10-05T01:49:00.387Z]    1268     if mode == 'r':
[2023-10-05T01:49:00.387Z] -> 1269         self._RealGetContents()
[2023-10-05T01:49:00.387Z]    1270     elif mode in ('w', 'x'):
[2023-10-05T01:49:00.387Z]    1271         # set the modified flag so central directory gets written
[2023-10-05T01:49:00.387Z]    1272         # even if no files are added to the archive
[2023-10-05T01:49:00.387Z]    1273         self._didModify = True
[2023-10-05T01:49:00.387Z] 
[2023-10-05T01:49:00.387Z] File /usr/lib/python3.10/zipfile.py:1336, in ZipFile._RealGetContents(self)
[2023-10-05T01:49:00.387Z]    1334     raise BadZipFile("File is not a zip file")
[2023-10-05T01:49:00.387Z]    1335 if not endrec:
[2023-10-05T01:49:00.387Z] -> 1336     raise BadZipFile("File is not a zip file")
[2023-10-05T01:49:00.387Z]    1337 if self.debug > 1:
[2023-10-05T01:49:00.387Z]    1338     print(endrec)
[2023-10-05T01:49:00.387Z] 
[2023-10-05T01:49:00.387Z] BadZipFile: File is not a zip file
KumoLiu commented 1 year ago

The data doesn't appear to be accessible to the public. Can we skip the tutorials on this data and just add some comments in the tutorial?

wyli commented 1 year ago

sure, please help skip the tests for now, and if the datasets are permanently removed, we'll need to redesign the tutorials.

KumoLiu commented 1 year ago

I just noticed this issue https://github.com/Project-MONAI/MONAI/issues/5412. Do we need to confirm with Warwick University, Tissue Image Analytics (TIA) Centre again?

cc @drbeh

drbeh commented 1 year ago

I checked their website and it seems that this dataset is temporarily not available! image

I will check with the Warwick university to see what's going on. Meanwhile, we should disable the check. Anyways, we should think of creating a more reliable mirror for this dataset.

KumoLiu commented 1 year ago

Thanks for the confirmation. Sure, I will skip the test for now. If there is no problem with the license, we may be able to put it in extra data. I remember that the zip size is only 154 MB.

KumoLiu commented 1 year ago

Just skip the tests for now at https://github.com/Project-MONAI/tutorials/pull/1544. May need to have another update after checking with Warwick university.

KumoLiu commented 12 months ago

Hi @drbeh, have you had a chance to get in touch with Warwick University and can we move the data? Thanks!

drbeh commented 12 months ago

Hi @KumoLiu, I have talked to them a while ago about the CoNSeP dataset but it was not clear what was the plan for this dataset. @shaneahmed do you have any update here? Is this dataset going to be available on TIA homepage, or should we make a mirror for this?