Closed wyli closed 1 year ago
[2023-09-25T15:14:51.787Z] Running ./modules/inverse_transforms_and_test_time_augmentations.ipynb [2023-09-25T15:14:51.787Z] Checking PEP8 compliance... [2023-09-25T15:14:52.715Z] Running notebook... [2023-09-25T15:14:52.715Z] Before: [2023-09-25T15:14:52.715Z] " max_epochs = 20\n", [2023-09-25T15:14:52.715Z] After: [2023-09-25T15:14:52.715Z] " max_epochs = 1\n", [2023-09-25T15:14:52.715Z] Before: [2023-09-25T15:14:52.715Z] " val_interval = 1\n", [2023-09-25T15:14:52.970Z] After: [2023-09-25T15:14:52.970Z] " val_interval = 1\n", [2023-09-25T15:14:59.502Z] MONAI version: 1.3.0rc1+4.g18a671a7 [2023-09-25T15:14:59.503Z] Numpy version: 1.26.0 [2023-09-25T15:14:59.503Z] Pytorch version: 2.1.0a0+29c30b1 [2023-09-25T15:14:59.503Z] MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False [2023-09-25T15:14:59.503Z] MONAI rev id: 18a671a7f12389dd8263a4c2232196dcca814bb9 [2023-09-25T15:14:59.503Z] MONAI __file__: /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/__init__.py [2023-09-25T15:14:59.503Z] [2023-09-25T15:14:59.503Z] Optional dependencies: [2023-09-25T15:14:59.503Z] Pytorch Ignite version: 0.4.11 [2023-09-25T15:14:59.503Z] ITK version: 5.3.0 [2023-09-25T15:14:59.503Z] Nibabel version: 5.1.0 [2023-09-25T15:14:59.503Z] scikit-image version: 0.21.0 [2023-09-25T15:14:59.503Z] scipy version: 1.11.1 [2023-09-25T15:14:59.503Z] Pillow version: 9.2.0 [2023-09-25T15:14:59.503Z] Tensorboard version: 2.9.0 [2023-09-25T15:14:59.503Z] gdown version: 4.7.1 [2023-09-25T15:14:59.503Z] TorchVision version: 0.16.0a0 [2023-09-25T15:14:59.503Z] tqdm version: 4.65.0 [2023-09-25T15:14:59.503Z] lmdb version: 1.4.1 [2023-09-25T15:14:59.503Z] psutil version: 5.9.4 [2023-09-25T15:14:59.503Z] pandas version: 2.1.1 [2023-09-25T15:14:59.503Z] einops version: 0.6.1 [2023-09-25T15:14:59.503Z] transformers version: 4.21.3 [2023-09-25T15:14:59.503Z] mlflow version: 2.7.1 [2023-09-25T15:14:59.503Z] pynrrd version: 1.0.0 [2023-09-25T15:14:59.503Z] clearml version: 1.13.0 [2023-09-25T15:14:59.503Z] [2023-09-25T15:14:59.503Z] For details about installing the optional dependencies, please visit: [2023-09-25T15:14:59.503Z] https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies [2023-09-25T15:14:59.503Z] [2023-09-25T15:15:02.018Z] papermill --progress-bar -k python3 [2023-09-25T15:15:02.579Z] /usr/local/lib/python3.10/dist-packages/papermill/iorw.py:153: UserWarning: the file is not specified with any extension : - [2023-09-25T15:15:02.579Z] warnings.warn( [2023-09-25T15:50:44.295Z] Executing: 0%| | 0/29 [00:00<?, ?cell/s] Executing: 3%|▎ | 1/29 [00:01<00:41, 1.49s/cell] Executing: 14%|█▍ | 4/29 [00:09<01:02, 2.51s/cell] Executing: 21%|██ | 6/29 [00:15<00:59, 2.58s/cell] Executing: 34%|███▍ | 10/29 [35:36<1:33:29, 295.23s/cell] Executing: 38%|███▊ | 11/29 [35:39<1:14:04, 246.92s/cell] Executing: 38%|███▊ | 11/29 [35:41<58:24, 194.72s/cell] [2023-09-25T15:50:45.223Z] /usr/local/lib/python3.10/dist-packages/papermill/iorw.py:153: UserWarning: the file is not specified with any extension : - [2023-09-25T15:50:45.223Z] warnings.warn( [2023-09-25T15:50:45.479Z] Traceback (most recent call last): [2023-09-25T15:50:45.479Z] File "/usr/local/bin/papermill", line 8, in <module> [2023-09-25T15:50:45.479Z] sys.exit(papermill()) [2023-09-25T15:50:45.479Z] File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1157, in __call__ [2023-09-25T15:50:45.479Z] return self.main(*args, **kwargs) [2023-09-25T15:50:45.479Z] File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1078, in main [2023-09-25T15:50:45.479Z] rv = self.invoke(ctx) [2023-09-25T15:50:45.479Z] File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1434, in invoke [2023-09-25T15:50:45.479Z] return ctx.invoke(self.callback, **ctx.params) [2023-09-25T15:50:45.479Z] File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 783, in invoke [2023-09-25T15:50:45.479Z] return __callback(*args, **kwargs) [2023-09-25T15:50:45.479Z] File "/usr/local/lib/python3.10/dist-packages/click/decorators.py", line 33, in new_func [2023-09-25T15:50:45.479Z] return f(get_current_context(), *args, **kwargs) [2023-09-25T15:50:45.479Z] File "/usr/local/lib/python3.10/dist-packages/papermill/cli.py", line 250, in papermill [2023-09-25T15:50:45.479Z] execute_notebook( [2023-09-25T15:50:45.479Z] File "/usr/local/lib/python3.10/dist-packages/papermill/execute.py", line 128, in execute_notebook [2023-09-25T15:50:45.479Z] raise_for_execution_errors(nb, output_path) [2023-09-25T15:50:45.479Z] File "/usr/local/lib/python3.10/dist-packages/papermill/execute.py", line 232, in raise_for_execution_errors [2023-09-25T15:50:45.479Z] raise error [2023-09-25T15:50:45.479Z] papermill.exceptions.PapermillExecutionError: [2023-09-25T15:50:45.479Z] --------------------------------------------------------------------------- [2023-09-25T15:50:45.479Z] Exception encountered at "In [5]": [2023-09-25T15:50:45.479Z] --------------------------------------------------------------------------- [2023-09-25T15:50:45.479Z] RuntimeError Traceback (most recent call last) [2023-09-25T15:50:45.479Z] Cell In[5], line 46 [2023-09-25T15:50:45.479Z] 44 ds_2d = Dataset(data_dicts, transform_2d_slice) [2023-09-25T15:50:45.479Z] 45 dl_2d = DataLoader(ds_2d, batch_size=1, num_workers=10) [2023-09-25T15:50:45.479Z] ---> 46 data_dicts_single_slice = list(tqdm(dl_2d)) [2023-09-25T15:50:45.479Z] [2023-09-25T15:50:45.479Z] File /usr/local/lib/python3.10/dist-packages/tqdm/std.py:1178, in tqdm.__iter__(self) [2023-09-25T15:50:45.479Z] 1175 time = self._time [2023-09-25T15:50:45.479Z] 1177 try: [2023-09-25T15:50:45.479Z] -> 1178 for obj in iterable: [2023-09-25T15:50:45.479Z] 1179 yield obj [2023-09-25T15:50:45.479Z] 1180 # Update and possibly print the progressbar. [2023-09-25T15:50:45.479Z] 1181 # Note: does not call self.update(1) for speed optimisation. [2023-09-25T15:50:45.479Z] [2023-09-25T15:50:45.479Z] File /usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py:633, in _BaseDataLoaderIter.__next__(self) [2023-09-25T15:50:45.479Z] 630 if self._sampler_iter is None: [2023-09-25T15:50:45.479Z] 631 # TODO(https://github.com/pytorch/pytorch/issues/76750) [2023-09-25T15:50:45.479Z] 632 self._reset() # type: ignore[call-arg] [2023-09-25T15:50:45.479Z] --> 633 data = self._next_data() [2023-09-25T15:50:45.479Z] 634 self._num_yielded += 1 [2023-09-25T15:50:45.479Z] 635 if self._dataset_kind == _DatasetKind.Iterable and \ [2023-09-25T15:50:45.479Z] 636 self._IterableDataset_len_called is not None and \ [2023-09-25T15:50:45.479Z] 637 self._num_yielded > self._IterableDataset_len_called: [2023-09-25T15:50:45.479Z] [2023-09-25T15:50:45.479Z] File /usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py:1348, in _MultiProcessingDataLoaderIter._next_data(self) [2023-09-25T15:50:45.479Z] 1346 else: [2023-09-25T15:50:45.479Z] 1347 del self._task_info[idx] [2023-09-25T15:50:45.479Z] -> 1348 return self._process_data(data) [2023-09-25T15:50:45.479Z] [2023-09-25T15:50:45.479Z] File /usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py:1374, in _MultiProcessingDataLoaderIter._process_data(self, data) [2023-09-25T15:50:45.479Z] 1372 self._try_put_index() [2023-09-25T15:50:45.479Z] 1373 if isinstance(data, ExceptionWrapper): [2023-09-25T15:50:45.479Z] -> 1374 data.reraise() [2023-09-25T15:50:45.479Z] 1375 return data [2023-09-25T15:50:45.479Z] [2023-09-25T15:50:45.479Z] File /usr/local/lib/python3.10/dist-packages/torch/_utils.py:697, in ExceptionWrapper.reraise(self) [2023-09-25T15:50:45.479Z] 693 except TypeError: [2023-09-25T15:50:45.479Z] 694 # If the exception takes multiple arguments, don't try to [2023-09-25T15:50:45.479Z] 695 # instantiate since we don't know how to [2023-09-25T15:50:45.479Z] 696 raise RuntimeError(msg) from None [2023-09-25T15:50:45.479Z] --> 697 raise exception [2023-09-25T15:50:45.479Z] [2023-09-25T15:50:45.479Z] RuntimeError: Caught RuntimeError in DataLoader worker process 0. [2023-09-25T15:50:45.479Z] Original Traceback (most recent call last): [2023-09-25T15:50:45.479Z] File "/home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/transforms/transform.py", line 141, in apply_transform [2023-09-25T15:50:45.479Z] return _apply_transform(transform, data, unpack_items, lazy, overrides, log_stats) [2023-09-25T15:50:45.479Z] File "/home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/transforms/transform.py", line 98, in _apply_transform [2023-09-25T15:50:45.479Z] return transform(data, lazy=lazy) if isinstance(transform, LazyTrait) else transform(data) [2023-09-25T15:50:45.479Z] File "/tmp/ipykernel_21374/3811870245.py", line 24, in __call__ [2023-09-25T15:50:45.479Z] fname = os.path.basename(data[key + "_meta_dict"]["filename_or_obj"]) [2023-09-25T15:50:45.479Z] KeyError: 'image_meta_dict' [2023-09-25T15:50:45.479Z] [2023-09-25T15:50:45.479Z] The above exception was the direct cause of the following exception: [2023-09-25T15:50:45.479Z] [2023-09-25T15:50:45.479Z] Traceback (most recent call last): [2023-09-25T15:50:45.479Z] File "/home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/transforms/transform.py", line 141, in apply_transform [2023-09-25T15:50:45.479Z] return _apply_transform(transform, data, unpack_items, lazy, overrides, log_stats) [2023-09-25T15:50:45.479Z] File "/home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/transforms/transform.py", line 98, in _apply_transform [2023-09-25T15:50:45.479Z] return transform(data, lazy=lazy) if isinstance(transform, LazyTrait) else transform(data) [2023-09-25T15:50:45.479Z] File "/home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/transforms/compose.py", line 335, in __call__ [2023-09-25T15:50:45.479Z] result = execute_compose( [2023-09-25T15:50:45.479Z] File "/home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/transforms/compose.py", line 111, in execute_compose [2023-09-25T15:50:45.479Z] data = apply_transform( [2023-09-25T15:50:45.479Z] File "/home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/transforms/transform.py", line 171, in apply_transform [2023-09-25T15:50:45.480Z] raise RuntimeError(f"applying transform {transform}") from e [2023-09-25T15:50:45.480Z] RuntimeError: applying transform <__main__.SaveSliced object at 0x7f566c30a260> [2023-09-25T15:50:45.480Z] [2023-09-25T15:50:45.480Z] The above exception was the direct cause of the following exception: [2023-09-25T15:50:45.480Z] [2023-09-25T15:50:45.480Z] Traceback (most recent call last): [2023-09-25T15:50:45.480Z] File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop [2023-09-25T15:50:45.480Z] data = fetcher.fetch(index) [2023-09-25T15:50:45.480Z] File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch [2023-09-25T15:50:45.480Z] data = [self.dataset[idx] for idx in possibly_batched_index] [2023-09-25T15:50:45.480Z] File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/fetch.py", line 51, in <listcomp> [2023-09-25T15:50:45.480Z] data = [self.dataset[idx] for idx in possibly_batched_index] [2023-09-25T15:50:45.480Z] File "/home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/data/dataset.py", line 112, in __getitem__ [2023-09-25T15:50:45.480Z] return self._transform(index) [2023-09-25T15:50:45.480Z] File "/home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/data/dataset.py", line 98, in _transform [2023-09-25T15:50:45.480Z] return apply_transform(self.transform, data_i) if self.transform is not None else data_i [2023-09-25T15:50:45.480Z] File "/home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/transforms/transform.py", line 171, in apply_transform [2023-09-25T15:50:45.480Z] raise RuntimeError(f"applying transform {transform}") from e [2023-09-25T15:50:45.480Z] RuntimeError: applying transform <monai.transforms.compose.Compose object at 0x7f566c30a2c0>
wyli
commented
1 year ago wyli
commented
1 year ago