Error on custom dataset training

[lidiahhhh]

I followed the instruction giude and tried to train a custom dataset. Dataset is as follows: fcdd\data\datasets\custom\train\dashboard\normal fcdd\data\datasets\custom\test\dashboard\normal fcdd\data\datasets\custom\test\dashboard\anomalous

The error i get is `python runners/run_custom.py

Plotting ROC for completed classes up to 0... Traceback (most recent call last): File ".\fcdd\python\fcdd\runners\run_custom.py", line 49, in runner.run() File "...\Anaconda3\lib\site-packages\fcdd-1.1.0-py3.9.egg\fcdd\runners\bases.py", line 203, in run self.run_classes(**vars(self.args)) File "...\Anaconda3\lib\site-packages\fcdd-1.1.0-py3.9.egg\fcdd\runners\bases.py", line 221, in run_classes res = self.run_seeds( File "...\Anaconda3\lib\site-packages\fcdd-1.1.0-py3.9.egg\fcdd\runners\bases.py", line 181, in run_seeds res = self.run_one( File "...\Anaconda3\lib\site-packages\fcdd-1.1.0-py3.9.egg\fcdd\runners\bases.py", line 100, in run_one setup = trainer_setup( File "...\Anaconda3\lib\site-packages\fcdd-1.1.0-py3.9.egg\fcdd\training\setup.py", line 104, in trainer_setup ds = load_dataset( File "...\Anaconda3\lib\site-packages\fcdd-1.1.0-py3.9.egg\fcdd\datasets__init.py", line 67, in load_dataset dataset = ADImageFolderDataset( File "...\Anaconda3\lib\site-packages\fcdd-1.1.0-py3.9.egg\fcdd\datasets\image_folder.py", line 95, in init self.mean, self.std = self.extract_mean_std(self.trainpath, normal_class) File "...\Anaconda3\lib\site-packages\fcdd-1.1.0-py3.9.egg\fcdd\datasets\image_folder.py", line 194, in extract_meanstd for x, in loader: File "...\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 359, in iter return self._get_iterator() File "...\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 305, in _get_iterator return _MultiProcessingDataLoaderIter(self) File "...\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 918, in init w.start() File "...\Anaconda3\lib\multiprocessing\process.py", line 121, in start self._popen = self._Popen(self) File "...\Anaconda3\lib\multiprocessing\context.py", line 224, in _Popen return _default_context.get_context().Process._Popen(process_obj) File "...\Anaconda3\lib\multiprocessing\context.py", line 327, in _Popen return Popen(process_obj) File "...\Anaconda3\lib\multiprocessing\popen_spawn_win32.py", line 93, in init__ reduction.dump(process_obj, to_child) File "...\Anaconda3\lib\multiprocessing\reduction.py", line 60, in dump ForkingPickler(file, protocol).dump(obj) AttributeError: Can't pickle local object 'ADImageFolderDataset.extract_mean_std..'

$ Traceback (most recent call last): File "", line 1, in File "...\Anaconda3\lib\multiprocessing\spawn.py", line 116, in spawn_main exitcode = _main(fd, parent_sentinel) File "...\Anaconda3\lib\multiprocessing\spawn.py", line 126, in _main self = reduction.pickle.load(from_parent) EOFError: Ran out of input `

[liznerski]

This seems to be the same issue as #18. You can either switch to Linux, or you use the windows branch of this repository and move the ADImageFolderDataset.extract_mean_std function our of the class.

[lidiahhhh]

Thanks for the reply.. I actually managed to successfully run the code on Google colab.

I have another question on the results, if you can help me: I'd expected the heatmaps of the normal class to be completly blue, but i got that the heatmaps have a lot of dark red spots.. Am i doing something wrong?

Thanks

[liznerski]

Well...

The heatmaps need to be normalized so that they can be interpreted as images; the values need to be in [0, 1]). We provide "locally normalized" heatmaps, where each each heatmap is normalized w.r.t. itself so that it has at least one fully red pixel (value 1) and one fully blue pixel (value 0). In this case, heatmaps for normal samples will always have red spots. We also provide "globally normalized" heatmaps, where each heatmap is normalized w.r.t to the complete set of samples (e.g., test set). If this set contains a sufficient amount of anomalies, there shouldn't be too many red spots in the "globally normalized" heatmaps for normal samples.

Another reason for your red spots could be that the quantile hyperparamter is set suboptimally. Incerasing it--so that less scores are mapped to 1--might help. Have a look at Appendix C of our paper for more details.

Hope that helps :)