Project-MONAI / tutorials

MONAI Tutorials
https://monai.io/started.html
Apache License 2.0
1.85k stars 681 forks source link

Issue in maisi_train_vae_tutorial #1760

Closed KumoLiu closed 3 months ago

KumoLiu commented 3 months ago
Running ./generative/maisi/maisi_train_vae_tutorial.ipynb
Checking PEP8 compliance...
Running notebook...
MONAI version: 1.3.1+29.g14b086b5
Numpy version: 1.24.4
Pytorch version: 2.3.0a0+40ec155e58.nv24.03
MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
MONAI rev id: 14b086b553693f5d344ff054f37d12ce6839da06
MONAI __file__: /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/__init__.py

Optional dependencies:
Pytorch Ignite version: 0.4.11
ITK version: 5.4.0
Nibabel version: 5.2.1
scikit-image version: 0.19.3
scipy version: 1.12.0
Pillow version: 7.0.0
Tensorboard version: 2.17.0
gdown version: 5.2.0
TorchVision version: 0.18.0a0
tqdm version: 4.66.2
lmdb version: 1.5.1
psutil version: 5.9.4
pandas version: 2.2.2
einops version: 0.7.0
transformers version: 4.40.2
mlflow version: 2.14.3
pynrrd version: 1.0.0
clearml version: 1.16.3rc2

For details about installing the optional dependencies, please visit:
    https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies

papermill  --progress-bar --log-output -k python3
/usr/local/lib/python3.10/dist-packages/papermill/iorw.py:149: UserWarning: the file is not specified with any extension : -
  warnings.warn(f"the file is not specified with any extension : {os.path.basename(path)}")

Executing:   0%|          | 0/28 [00:00<?, ?cell/s]
Executing:   4%|▎         | 1/28 [00:01<00:28,  1.07s/cell]
Executing:   7%|▋         | 2/28 [01:31<23:06, 53.34s/cell]
Executing:  14%|█▍        | 4/28 [01:37<08:49, 22.06s/cell]
Executing:  29%|██▊       | 8/28 [08:47<25:36, 76.85s/cell]
Executing:  71%|███████▏  | 20/28 [08:47<02:43, 20.50s/cell]
Executing:  89%|████████▉ | 25/28 [08:51<00:43, 14.51s/cell]
Executing:  93%|█████████▎| 26/28 [08:54<00:41, 20.55s/cell]
/usr/local/lib/python3.10/dist-packages/papermill/iorw.py:149: UserWarning: the file is not specified with any extension : -
  warnings.warn(f"the file is not specified with any extension : {os.path.basename(path)}")
Traceback (most recent call last):
  File "/usr/local/bin/papermill", line 8, in <module>
    sys.exit(papermill())
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/papermill/cli.py", line 235, in papermill
    execute_notebook(
  File "/usr/local/lib/python3.10/dist-packages/papermill/execute.py", line 131, in execute_notebook
    raise_for_execution_errors(nb, output_path)
  File "/usr/local/lib/python3.10/dist-packages/papermill/execute.py", line 251, in raise_for_execution_errors
    raise error
papermill.exceptions.PapermillExecutionError: 
---------------------------------------------------------------------------
Exception encountered at "In [13]":
---------------------------------------------------------------------------
OptionalImportError                       Traceback (most recent call last)
Cell In[13], line 11
      7     print("Use l1 loss")
      8 adv_loss = PatchAdversarialLoss(criterion="least_squares")
     10 loss_perceptual = (
---> 11     PerceptualLoss(spatial_dims=3, network_type="squeeze", is_fake_3d=True, fake_3d_ratio=0.2).eval().to(device)
     12 )
     14 # config optimizer and lr scheduler
     15 optimizer_g = torch.optim.Adam(params=autoencoder.parameters(), lr=args.lr, eps=1e-06 if args.amp else 1e-08)

File /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/losses/perceptual.py:125, in PerceptualLoss.__init__(self, spatial_dims, network_type, is_fake_3d, fake_3d_ratio, cache_dir, pretrained, pretrained_path, pretrained_state_dict_key, channel_wise)
    118     self.perceptual_function = TorchvisionModelPerceptualSimilarity(
    119         net=network_type,
    120         pretrained=pretrained,
    121         pretrained_path=pretrained_path,
    122         pretrained_state_dict_key=pretrained_state_dict_key,
    123     )
    124 else:
--> 125     self.perceptual_function = LPIPS(pretrained=pretrained, net=network_type, verbose=False)
    126 self.is_fake_3d = is_fake_3d
    127 self.fake_3d_ratio = fake_3d_ratio

File /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/utils/module.py:451, in optional_import.<locals>._LazyRaise.__call__(self, *_args, **_kwargs)
    446 def __call__(self, *_args, **_kwargs):
    447     """
    448     Raises:
    449         OptionalImportError: When you call this method.
    450     """
--> 451     raise self._exception

File /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/utils/module.py:402, in optional_import(module, version, version_checker, name, descriptor, version_args, allow_namespace_pkg, as_type)
    400     actual_cmd = f"import {module}"
    401 try:
--> 402     pkg = __import__(module)  # top level module
    403     the_module = import_module(module)
    404     if not allow_namespace_pkg:

File /usr/local/lib/python3.10/dist-packages/lpips/__init__.py:11
      8 # from torch.autograd import Variable
     10 from lpips.trainer import *
---> 11 from lpips.lpips import *
     13 def normalize_tensor(in_feat,eps=1e-10):
     14     norm_factor = torch.sqrt(torch.sum(in_feat**2,dim=1,keepdim=True))

File /usr/local/lib/python3.10/dist-packages/lpips/lpips.py:9
      7 from torch.autograd import Variable
      8 import numpy as np
----> 9 from . import pretrained_networks as pn
     10 import torch.nn
     12 import lpips

File /usr/local/lib/python3.10/dist-packages/lpips/pretrained_networks.py:3
      1 from collections import namedtuple
      2 import torch
----> 3 from torchvision import models as tv
      5 class squeezenet(torch.nn.Module):
      6     def __init__(self, requires_grad=False, pretrained=True):

File /usr/local/lib/python3.10/dist-packages/torchvision/__init__.py:6
      3 from modulefinder import Module
      5 import torch
----> 6 from torchvision import _meta_registrations, datasets, io, models, ops, transforms, utils
      8 from .extension import _HAS_OPS
     10 try:

File /usr/local/lib/python3.10/dist-packages/torchvision/_meta_registrations.py:26
     20         return fn
     22     return wrapper
     25 @register_meta("roi_align")
---> 26 def meta_roi_align(input, rois, spatial_scale, pooled_height, pooled_width, sampling_ratio, aligned):
     27     torch._check(rois.size(1) == 5, lambda: "rois must have shape as Tensor[K, 5]")
     28     torch._check(
     29         input.dtype == rois.dtype,
     30         lambda: (
   (...)
     33         ),
     34     )

File /usr/local/lib/python3.10/dist-packages/torchvision/_meta_registrations.py:18, in register_meta.<locals>.wrapper(fn)
     17 def wrapper(fn):
---> 18     if torchvision.extension._has_ops():
     19         get_meta_lib().impl(getattr(getattr(torch.ops.torchvision, op_name), overload_name), fn)
     20     return fn

OptionalImportError: from lpips import LPIPS (partially initialized module 'torchvision' has no attribute 'extension' (most likely due to a circular import)).

For details about installing the optional dependencies, please visit:
    https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies

real    8m55.442s
user    2m25.507s
sys 1m9.084s
Check failed!
Can-Zhao commented 3 months ago

This import is used in MONAI Core: monai/losses/perceptual.py Do we have any idea on how to fix it?

KumoLiu commented 3 months ago

Solved in the blossom. The issue is that xformers re-installed the torch and torchvision.