"RuntimeError: Trying to resize storage that is not resizable at .." when executing "python runners/run_mvtec.py"

GreatScherzo commented 3 years ago

🐛Bug

Executing "python runners/run_mvtec.py" in windows seems to cause "RuntimeError: Trying to resize storage that is not resizable at ..".
Used windows branch program.

Traceback

(Fcdd_Env) C:\Users\PC-2001-044\Desktop\fcdd-windows\fcdd-windows\python>python fcdd/runners/run_mvtec.py
Downloading ftp://guest:GU%2E205dldo@ftp.softronics.ch/mvtec_anomaly_detection/mvtec_anomaly_detection.tar.xz to C:\Users\PC-2001-044\Desktop\fcdd-windows\data\datasets\mvtec\mvtec_anomaly_detection.tar.xz
100.0%Extracting C:\Users\PC-2001-044\Desktop\fcdd-windows\data\datasets\mvtec\mvtec_anomaly_detection.tar.xz to C:\Users\PC-200~1\AppData\Local\Temp\tmpvdfnxt5a
Processing data for label bottle...
Processing data for label cable...
Processing data for label capsule...
Processing data for label carpet...
Processing data for label grid...
Processing data for label hazelnut...
Processing data for label leather...
Processing data for label metal_nut...
Processing data for label pill...
Processing data for label screw...
Processing data for label tile...
Processing data for label toothbrush...
Processing data for label transistor...
Processing data for label wood...
Processing data for label zipper...
Loading dataset from C:\Users\PC-2001-044\Desktop\fcdd-windows\data\datasets\mvtec\admvtec_240x240.pt...
Dataset complete.
Files already downloaded.
Loading dataset from C:\Users\PC-2001-044\Desktop\fcdd-windows\data\datasets\mvtec\admvtec_240x240.pt...
Dataset complete.
Downloading: "https://download.pytorch.org/models/vgg11_bn-6002323d.pth" to C:\Users\PC-2001-044\anaconda3\envs\Fcdd_Env\lib\site-packages\fcdd\models\..\..\..\data\models\vgg11_bn-6002323d.pth
100.0%
##### NET RECEPTION {'n': 28, 'j': 8, 'r': 62, 's': 3.5, 'img_shape': (3, 224, 224)} #####
Successfully saved code at C:\Users\PC-2001-044\Desktop\fcdd-windows\data\results\fcdd_20210206091247_mvtec_\normal_0\it_0\.\src.tar.gz
Generating dataset preview...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\PC-2001-044\anaconda3\envs\Fcdd_Env\lib\multiprocessing\spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "C:\Users\PC-2001-044\anaconda3\envs\Fcdd_Env\lib\multiprocessing\spawn.py", line 126, in _main
    self = reduction.pickle.load(from_parent)
  File "C:\Users\PC-2001-044\anaconda3\envs\Fcdd_Env\lib\site-packages\torch\multiprocessing\reductions.py", line 90, in rebuild_tensor
    t = torch._utils._rebuild_tensor(storage, storage_offset, size, stride)
  File "C:\Users\PC-2001-044\anaconda3\envs\Fcdd_Env\lib\site-packages\torch\_utils.py", line 131, in _rebuild_tensor
    return t.set_(storage, storage_offset, size, stride)
RuntimeError: Trying to resize storage that is not resizable at ..\aten\src\TH\THStorageFunctions.cpp:88
Plotting Many ROC for completed classes up to 0...
Traceback (most recent call last):
  File "C:\Users\PC-2001-044\anaconda3\envs\Fcdd_Env\lib\site-packages\torch\utils\data\dataloader.py", line 761, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "C:\Users\PC-2001-044\anaconda3\envs\Fcdd_Env\lib\queue.py", line 178, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "fcdd/runners/run_mvtec.py", line 19, in <module>
    runner.run()
  File "C:\Users\PC-2001-044\anaconda3\envs\Fcdd_Env\lib\site-packages\fcdd\runners\bases.py", line 203, in run
    self.run_classes(**vars(self.args))
  File "C:\Users\PC-2001-044\anaconda3\envs\Fcdd_Env\lib\site-packages\fcdd\runners\bases.py", line 221, in run_classes
    res = self.run_seeds(
  File "C:\Users\PC-2001-044\anaconda3\envs\Fcdd_Env\lib\site-packages\fcdd\runners\bases.py", line 181, in run_seeds
    res = self.run_one(
  File "C:\Users\PC-2001-044\anaconda3\envs\Fcdd_Env\lib\site-packages\fcdd\runners\bases.py", line 100, in run_one
    setup = trainer_setup(
  File "C:\Users\PC-2001-044\anaconda3\envs\Fcdd_Env\lib\site-packages\fcdd\training\setup.py", line 122, in trainer_setup
    'ds_preview', ds.preview(20), nrow=20,
  File "C:\Users\PC-2001-044\anaconda3\envs\Fcdd_Env\lib\site-packages\fcdd\datasets\bases.py", line 94, in preview
    for xb, yb, gtsb in loader:
  File "C:\Users\PC-2001-044\anaconda3\envs\Fcdd_Env\lib\site-packages\torch\utils\data\dataloader.py", line 345, in __next__
    data = self._next_data()
  File "C:\Users\PC-2001-044\anaconda3\envs\Fcdd_Env\lib\site-packages\torch\utils\data\dataloader.py", line 841, in _next_data
    idx, data = self._get_data()
  File "C:\Users\PC-2001-044\anaconda3\envs\Fcdd_Env\lib\site-packages\torch\utils\data\dataloader.py", line 798, in _get_data
    success, data = self._try_get_data()
  File "C:\Users\PC-2001-044\anaconda3\envs\Fcdd_Env\lib\site-packages\torch\utils\data\dataloader.py", line 774, in _try_get_data
    raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 13204) exited unexpectedly

Expected Behaviour

Was expecting for it to sample the MVTec Dataset without any problems

Environment

PyTorch Version: 1.4.0 Is Cuda Available?: Yes

OS: Windows 10 Pro

Python version: 3.8

CUDA runtime version: 10.2.141 GPU models and configuration: GPU 0: NVIDIA Quadro P1000 Nvidia driver version: 442.50

Additional Context

Thank you very much for taking time in assessing this issue :)

Edit

Sorry, i forgot to mention, i had problems with installing pytorch for the environment, then i went to install it by direct linking to the builds

(Fcdd_Env) C:\Users\PC-2001-044\Desktop\fcdd-windows\fcdd-windows\python>pip install .
Processing c:\users\pc-2001-044\desktop\fcdd-windows\fcdd-windows\python
Requirement already satisfied: matplotlib==3.1.3 in c:\users\pc-2001-044\appdata\roaming\python\python38\site-packages (from fcdd==0.6.1) (3.1.3)
Requirement already satisfied: torchvision==0.5.0 in c:\users\pc-2001-044\appdata\roaming\python\python38\site-packages (from fcdd==0.6.1) (0.5.0)
ERROR: Could not find a version that satisfies the requirement torch==1.4.0 (from fcdd)
ERROR: No matching distribution found for torch==1.4.0

(Fcdd_Env) C:\Users\PC-2001-044\Desktop\fcdd-windows\fcdd-windows\python>pip install https://download.pytorch.org/whl/cu101/torch-1.4.0-cp38-cp38-win_amd64.whl
Collecting torch==1.4.0
  Using cached https://download.pytorch.org/whl/cu101/torch-1.4.0-cp38-cp38-win_amd64.whl (796.9 MB)
Installing collected packages: torch
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
torchvision 0.5.0 requires numpy, which is not installed.
Successfully installed torch-1.4.0

(Fcdd_Env) C:\Users\PC-2001-044\Desktop\fcdd-windows\fcdd-windows\python>pip install .
Processing c:\users\pc-2001-044\desktop\fcdd-windows\fcdd-windows\python
Requirement already satisfied: matplotlib==3.1.3 in c:\users\pc-2001-044\appdata\roaming\python\python38\site-packages (from fcdd==0.6.1) (3.1.3)
Requirement already satisfied: torchvision==0.5.0 in c:\users\pc-2001-044\appdata\roaming\python\python38\site-packages (from fcdd==0.6.1) (0.5.0)
Requirement already satisfied: torch==1.4.0 in c:\users\pc-2001-044\anaconda3\envs\fcdd_env\lib\site-packages (from fcdd==0.6.1) (1.4.0)
Requirement already satisfied: opencv-python==4.2.0.32 in c:\users\pc-2001-044\appdata\roaming\python\python38\site-packages (from fcdd==0.6.1) (4.2.0.32)
Requirement already satisfied: six==1.14.0 in c:\users\pc-2001-044\appdata\roaming\python\python38\site-packages (from fcdd==0.6.1) (1.14.0)
Requirement already satisfied: scipy==1.4.1 in c:\users\pc-2001-044\appdata\roaming\python\python38\site-packages (from fcdd==0.6.1) (1.4.1)
Requirement already satisfied: scikit-image==0.16.2 in c:\users\pc-2001-044\appdata\roaming\python\python38\site-packages (from fcdd==0.6.1) (0.16.2)
Requirement already satisfied: scikit-learn==0.22.1 in c:\users\pc-2001-044\appdata\roaming\python\python38\site-packages (from fcdd==0.6.1) (0.22.1)
Requirement already satisfied: Pillow==7.0.0 in c:\users\pc-2001-044\appdata\roaming\python\python38\site-packages (from fcdd==0.6.1) (7.0.0)
Requirement already satisfied: pprofile==2.0.2 in c:\users\pc-2001-044\appdata\roaming\python\python38\site-packages (from fcdd==0.6.1) (2.0.2)
Requirement already satisfied: psutil==5.7.0 in c:\users\pc-2001-044\appdata\roaming\python\python38\site-packages (from fcdd==0.6.1) (5.7.0)
Requirement already satisfied: pyparsing==2.4.6 in c:\users\pc-2001-044\appdata\roaming\python\python38\site-packages (from fcdd==0.6.1) (2.4.6)
Requirement already satisfied: networkx==2.4 in c:\users\pc-2001-044\appdata\roaming\python\python38\site-packages (from fcdd==0.6.1) (2.4)
Requirement already satisfied: imageio==2.6.1 in c:\users\pc-2001-044\appdata\roaming\python\python38\site-packages (from fcdd==0.6.1) (2.6.1)
Requirement already satisfied: joblib==0.14.1 in c:\users\pc-2001-044\appdata\roaming\python\python38\site-packages (from fcdd==0.6.1) (0.14.1)
Requirement already satisfied: kiwisolver==1.1.0 in c:\users\pc-2001-044\appdata\roaming\python\python38\site-packages (from fcdd==0.6.1) (1.1.0)
Requirement already satisfied: kornia==0.2.0 in c:\users\pc-2001-044\appdata\roaming\python\python38\site-packages (from fcdd==0.6.1) (0.2.0)
Requirement already satisfied: setuptools in c:\users\pc-2001-044\anaconda3\envs\fcdd_env\lib\site-packages (from kiwisolver==1.1.0->fcdd==0.6.1) (52.0.0.post20210125)
Collecting numpy==1.18.1
  Using cached numpy-1.18.1-cp38-cp38-win_amd64.whl (12.8 MB)
Collecting python-dateutil==2.8.1
  Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting PyWavelets==1.1.1
  Downloading PyWavelets-1.1.1-cp38-cp38-win_amd64.whl (4.3 MB)
     |████████████████████████████████| 4.3 MB 3.3 MB/s
Collecting cycler>=0.10
  Using cached cycler-0.10.0-py2.py3-none-any.whl (6.5 kB)
Collecting decorator>=4.3.0
  Using cached decorator-4.4.2-py2.py3-none-any.whl (9.2 kB)
Building wheels for collected packages: fcdd
  Building wheel for fcdd (setup.py) ... done
  Created wheel for fcdd: filename=fcdd-0.6.1-py3-none-any.whl size=116241 sha256=c46c3f9294caaf6e171a8456c35860ebc8a09c572fdb9e2be9d880e7f3c787c1
  Stored in directory: C:\Users\PC-2001-044\AppData\Local\Temp\pip-ephem-wheel-cache-tmdctf0b\wheels\ac\7e\35\c763dfdc3e2ef819fad43e7ad60cedb088e9f8f8ec8e0d4b13
Successfully built fcdd
Installing collected packages: python-dateutil, numpy, decorator, cycler, PyWavelets, fcdd
Successfully installed PyWavelets-1.1.1 cycler-0.10.0 decorator-4.4.2 fcdd-0.6.1 numpy-1.18.1 python-dateutil-2.8.1

liznerski commented 3 years ago

Hey. Thanks for your report and detailed description, and sorry for the rather late response. We were able to locate the problem and I've just pushed a commit that should fix it. Note that this required to deactivate enlarging the MVTec-AD dataset, which we initially did to speed up the data loading process. This should not impact the performance, but you need ten times as many epochs as reported in the paper. I adjusted the default parameters, so you don't need to worry about that.

Thanks to @tkollmer for helping me to test this on Windows.

GreatScherzo commented 3 years ago

Thank you very much taking your time. Unfortunately, I'm still experiencing the same bug. Currently debugging for the underlying problem, will post an update if I find more about this

GreatScherzo commented 3 years ago

So apparently the bug occurs if you used Python 3.8 instead of 3.6.

When I was following the steps in installing fcdd, pip gives out an error saying that "it could not find a version that satisfies the requirement for torch==1.4.0" within fcdd, which is kind of ambiguous. I then tried installing torch 1.4.0 using conda, which tells me clearly that the current python version isn't compatible with torch 1.4. I then downgraded python, installed torch 1.4 through conda, and tried using the fcdd-windows of the previous version, and sure enough, the bug that you mentioned above #7 did occur. I then tried it with the patched version, and it worked fine.

Cheers!

liznerski / fcdd