search

RetroCirce / HTS-Audio-Transformer

The official code repo of "HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection"
https://arxiv.org/abs/2202.00874
MIT License
341 stars 62 forks source link

cannot pickle 'module' object #52

Open gillesmeyhi opened 9 months ago

gillesmeyhi commented 9 months ago

While trying to run the htsat_esc_training notebook, I get the following error which I do not seem to find a solution for... The error occurs when running trainer.fit()

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
File [c:\ProgramData\anaconda3\envs\htsat8\lib\site-packages\pytorch_lightning\trainer\call.py:44](file:///C:/ProgramData/anaconda3/envs/htsat8/lib/site-packages/pytorch_lightning/trainer/call.py:44), in _call_and_handle_interrupt(trainer, trainer_fn, *args, **kwargs)
     43         return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
---> 44     return trainer_fn(*args, **kwargs)
     46 except _TunerExitException:

File [c:\ProgramData\anaconda3\envs\htsat8\lib\site-packages\pytorch_lightning\trainer\trainer.py:581](file:///C:/ProgramData/anaconda3/envs/htsat8/lib/site-packages/pytorch_lightning/trainer/trainer.py:581), in Trainer._fit_impl(self, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path)
    575 ckpt_path = self._checkpoint_connector._select_ckpt_path(
    576     self.state.fn,
    577     ckpt_path,
    578     model_provided=True,
    579     model_connected=self.lightning_module is not None,
    580 )
--> 581 self._run(model, ckpt_path=ckpt_path)
    583 assert self.state.stopped

File [c:\ProgramData\anaconda3\envs\htsat8\lib\site-packages\pytorch_lightning\trainer\trainer.py:990](file:///C:/ProgramData/anaconda3/envs/htsat8/lib/site-packages/pytorch_lightning/trainer/trainer.py:990), in Trainer._run(self, model, ckpt_path)
    987 # ----------------------------
    988 # RUN THE TRAINER
    989 # ----------------------------
--> 990 results = self._run_stage()
    992 # ----------------------------
    993 # POST-Training CLEAN UP
    994 # ----------------------------

File [c:\ProgramData\anaconda3\envs\htsat8\lib\site-packages\pytorch_lightning\trainer\trainer.py:1036](file:///C:/ProgramData/anaconda3/envs/htsat8/lib/site-packages/pytorch_lightning/trainer/trainer.py:1036), in Trainer._run_stage(self)
   1035 with torch.autograd.set_detect_anomaly(self._detect_anomaly):
-> 1036     self.fit_loop.run()
   1037 return None

File [c:\ProgramData\anaconda3\envs\htsat8\lib\site-packages\pytorch_lightning\loops\fit_loop.py:194](file:///C:/ProgramData/anaconda3/envs/htsat8/lib/site-packages/pytorch_lightning/loops/fit_loop.py:194), in _FitLoop.run(self)
    193 def run(self) -> None:
--> 194     self.setup_data()
    195     if self.skip:

File [c:\ProgramData\anaconda3\envs\htsat8\lib\site-packages\pytorch_lightning\loops\fit_loop.py:258](file:///C:/ProgramData/anaconda3/envs/htsat8/lib/site-packages/pytorch_lightning/loops/fit_loop.py:258), in _FitLoop.setup_data(self)
    257 self._data_fetcher.setup(combined_loader)
--> 258 iter(self._data_fetcher)  # creates the iterator inside the fetcher
    259 max_batches = sized_len(combined_loader)

File [c:\ProgramData\anaconda3\envs\htsat8\lib\site-packages\pytorch_lightning\loops\fetchers.py:99](file:///C:/ProgramData/anaconda3/envs/htsat8/lib/site-packages/pytorch_lightning/loops/fetchers.py:99), in _PrefetchDataFetcher.__iter__(self)
     98 def __iter__(self) -> "_PrefetchDataFetcher":
---> 99     super().__iter__()
    100     if self.length is not None:
    101         # ignore pre-fetching, it's not necessary

File [c:\ProgramData\anaconda3\envs\htsat8\lib\site-packages\pytorch_lightning\loops\fetchers.py:48](file:///C:/ProgramData/anaconda3/envs/htsat8/lib/site-packages/pytorch_lightning/loops/fetchers.py:48), in _DataFetcher.__iter__(self)
     47 def __iter__(self) -> "_DataFetcher":
---> 48     self.iterator = iter(self.combined_loader)
     49     self.reset()

File [c:\ProgramData\anaconda3\envs\htsat8\lib\site-packages\pytorch_lightning\utilities\combined_loader.py:335](file:///C:/ProgramData/anaconda3/envs/htsat8/lib/site-packages/pytorch_lightning/utilities/combined_loader.py:335), in CombinedLoader.__iter__(self)
    334 iterator = cls(self.flattened, self._limits)
--> 335 iter(iterator)
    336 self._iterator = iterator

File [c:\ProgramData\anaconda3\envs\htsat8\lib\site-packages\pytorch_lightning\utilities\combined_loader.py:87](file:///C:/ProgramData/anaconda3/envs/htsat8/lib/site-packages/pytorch_lightning/utilities/combined_loader.py:87), in _MaxSizeCycle.__iter__(self)
     86 def __iter__(self) -> Self:
---> 87     super().__iter__()
     88     self._consumed = [False] * len(self.iterables)

File [c:\ProgramData\anaconda3\envs\htsat8\lib\site-packages\pytorch_lightning\utilities\combined_loader.py:40](file:///C:/ProgramData/anaconda3/envs/htsat8/lib/site-packages/pytorch_lightning/utilities/combined_loader.py:40), in _ModeIterator.__iter__(self)
     39 def __iter__(self) -> Self:
---> 40     self.iterators = [iter(iterable) for iterable in self.iterables]
     41     self._idx = 0

File [c:\ProgramData\anaconda3\envs\htsat8\lib\site-packages\pytorch_lightning\utilities\combined_loader.py:40](file:///C:/ProgramData/anaconda3/envs/htsat8/lib/site-packages/pytorch_lightning/utilities/combined_loader.py:40), in (.0)
     39 def __iter__(self) -> Self:
---> 40     self.iterators = [iter(iterable) for iterable in self.iterables]
     41     self._idx = 0

File [c:\ProgramData\anaconda3\envs\htsat8\lib\site-packages\torch\utils\data\dataloader.py:438](file:///C:/ProgramData/anaconda3/envs/htsat8/lib/site-packages/torch/utils/data/dataloader.py:438), in DataLoader.__iter__(self)
    437 else:
--> 438     return self._get_iterator()

File [c:\ProgramData\anaconda3\envs\htsat8\lib\site-packages\torch\utils\data\dataloader.py:386](file:///C:/ProgramData/anaconda3/envs/htsat8/lib/site-packages/torch/utils/data/dataloader.py:386), in DataLoader._get_iterator(self)
    385 self.check_worker_number_rationality()
--> 386 return _MultiProcessingDataLoaderIter(self)

File [c:\ProgramData\anaconda3\envs\htsat8\lib\site-packages\torch\utils\data\dataloader.py:1039](file:///C:/ProgramData/anaconda3/envs/htsat8/lib/site-packages/torch/utils/data/dataloader.py:1039), in _MultiProcessingDataLoaderIter.__init__(self, loader)
   1033 # NB: Process.start() actually take some time as it needs to
   1034 #     start a process and pass the arguments over via a pipe.
   1035 #     Therefore, we only add a worker to self._workers list after
   1036 #     it started, so that we do not call .join() if program dies
   1037 #     before it starts, and __del__ tries to join but will get:
   1038 #     AssertionError: can only join a started process.
-> 1039 w.start()
   1040 self._index_queues.append(index_queue)

File [c:\ProgramData\anaconda3\envs\htsat8\lib\multiprocessing\process.py:121](file:///C:/ProgramData/anaconda3/envs/htsat8/lib/multiprocessing/process.py:121), in BaseProcess.start(self)
    120 _cleanup()
--> 121 self._popen = self._Popen(self)
    122 self._sentinel = self._popen.sentinel

File [c:\ProgramData\anaconda3\envs\htsat8\lib\multiprocessing\context.py:224](file:///C:/ProgramData/anaconda3/envs/htsat8/lib/multiprocessing/context.py:224), in Process._Popen(process_obj)
    222 @staticmethod
    223 def _Popen(process_obj):
--> 224     return _default_context.get_context().Process._Popen(process_obj)

File [c:\ProgramData\anaconda3\envs\htsat8\lib\multiprocessing\context.py:327](file:///C:/ProgramData/anaconda3/envs/htsat8/lib/multiprocessing/context.py:327), in SpawnProcess._Popen(process_obj)
    326 from .popen_spawn_win32 import Popen
--> 327 return Popen(process_obj)

File [c:\ProgramData\anaconda3\envs\htsat8\lib\multiprocessing\popen_spawn_win32.py:93](file:///C:/ProgramData/anaconda3/envs/htsat8/lib/multiprocessing/popen_spawn_win32.py:93), in Popen.__init__(self, process_obj)
     92     reduction.dump(prep_data, to_child)
---> 93     reduction.dump(process_obj, to_child)
     94 finally:

File [c:\ProgramData\anaconda3\envs\htsat8\lib\multiprocessing\reduction.py:60](file:///C:/ProgramData/anaconda3/envs/htsat8/lib/multiprocessing/reduction.py:60), in dump(obj, file, protocol)
     59 '''Replacement for pickle.dump() using ForkingPickler.'''
---> 60 ForkingPickler(file, protocol).dump(obj)

TypeError: cannot pickle 'module' object
gillesmeyhi commented 9 months ago

If you cannot reproduce this error, can you please share your environment details?

RetroCirce commented 9 months ago

Hi, sorry for the late reply. You can refer to this issue. Basically the reason is about the environment and some hyperparameter changes.

My environment (when I did this project) is torch 1.7 + pytorch lightning 1.5.9, you can see on this yaml file.