Can't Pickle Error when evaluating convnext on HSN

[paulpeyret-biophonia]

Hello,

Thanks for this great preprint and for this lib that looks promising!

I am trying to reproduce the evaluation of HSN dataset with convnext pretrained on XCL dataset.

I am working on branch neurips2024 .

My OS is Windows 11. I installed the packages using conda and pip and I add to update my torchvision, torch and torchaudio version to be compatible with my A5000 GPU drivers version (551.61) and cuda version (12.4) already installed. torchaudio 2.4.1+cu124 torchvision 0.19.1+cu124 torch 2.4.1+cu124

All the preprocessing works fine and I manage to load the datasets, preprocess and one hot encode them. It seems like the error happens when It loads the model.

Maybe I missed something in the setup process ? I am not so familiar with multiprocessing.

Thanks for you help.

Here is the error log when i execute python birdset/eval.py experiment="birdset_neurips24/HSN/LT/convnext.yaml"

...
[2024-09-10 10:50:39,330][birdset.datamodule.base_datamodule][INFO] - test
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
Error executing job with overrides: ['experiment=birdset_neurips24/HSN/LT/convnext.yaml']
Traceback (most recent call last):
  File "C:\Users\Username\Dev\BirdSet\birdset\eval.py", line 82, in eval
    trainer.test(
  File "C:\Users\Username\miniconda3\envs\birdset\lib\site-packages\lightning\pytorch\trainer\trainer.py", line 754, in test
    return call._call_and_handle_interrupt(
  File "C:\Users\Username\miniconda3\envs\birdset\lib\site-packages\lightning\pytorch\trainer\call.py", line 44, in _call_and_handle_interrupt    
    return trainer_fn(*args, **kwargs)
  File "C:\Users\Username\miniconda3\envs\birdset\lib\site-packages\lightning\pytorch\trainer\trainer.py", line 794, in _test_impl
    results = self._run(model, ckpt_path=ckpt_path)
  File "C:\Users\Username\miniconda3\envs\birdset\lib\site-packages\lightning\pytorch\trainer\trainer.py", line 989, in _run
    results = self._run_stage()
  File "C:\Users\Username\miniconda3\envs\birdset\lib\site-packages\lightning\pytorch\trainer\trainer.py", line 1028, in _run_stage
    return self._evaluation_loop.run()
  File "C:\Users\Username\miniconda3\envs\birdset\lib\site-packages\lightning\pytorch\loops\utilities.py", line 182, in _decorator
    return loop_run(self, *args, **kwargs)
  File "C:\Users\Username\miniconda3\envs\birdset\lib\site-packages\lightning\pytorch\loops\evaluation_loop.py", line 112, in run
    self.reset()
  File "C:\Users\Username\miniconda3\envs\birdset\lib\site-packages\lightning\pytorch\loops\evaluation_loop.py", line 230, in reset
    iter(data_fetcher)  # creates the iterator inside the fetcher
  File "C:\Users\Username\miniconda3\envs\birdset\lib\site-packages\lightning\pytorch\loops\fetchers.py", line 99, in __iter__
    super().__iter__()
  File "C:\Users\Username\miniconda3\envs\birdset\lib\site-packages\lightning\pytorch\loops\fetchers.py", line 48, in __iter__
    self.iterator = iter(self.combined_loader)
  File "C:\Users\Username\miniconda3\envs\birdset\lib\site-packages\lightning\pytorch\utilities\combined_loader.py", line 335, in __iter__        
    iter(iterator)
  File "C:\Users\Username\miniconda3\envs\birdset\lib\site-packages\lightning\pytorch\utilities\combined_loader.py", line 144, in __iter__        
    self._load_current_iterator()
  File "C:\Users\Username\miniconda3\envs\birdset\lib\site-packages\lightning\pytorch\utilities\combined_loader.py", line 160, in _load_current_iterator
    self.iterators = [iter(self.iterables[self._iterator_idx])]
  File "C:\Users\Username\miniconda3\envs\birdset\lib\site-packages\torch\utils\data\dataloader.py", line 435, in __iter__
    self._iterator = self._get_iterator()
  File "C:\Users\Username\miniconda3\envs\birdset\lib\site-packages\torch\utils\data\dataloader.py", line 388, in _get_iterator
    return _MultiProcessingDataLoaderIter(self)
  File "C:\Users\Username\miniconda3\envs\birdset\lib\site-packages\torch\utils\data\dataloader.py", line 1038, in __init__
    w.start()
  File "C:\Users\Username\miniconda3\envs\birdset\lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
  File "C:\Users\Username\miniconda3\envs\birdset\lib\multiprocessing\context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Users\Username\miniconda3\envs\birdset\lib\multiprocessing\context.py", line 327, in _Popen
    return Popen(process_obj)
  File "C:\Users\Username\miniconda3\envs\birdset\lib\multiprocessing\popen_spawn_win32.py", line 93, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Users\Username\miniconda3\envs\birdset\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'MultilabelMix.__init__.<locals>.<lambda>'. Did you mean: '_return_value'?

[raphaelschwinger]

Hey @paulpeyret-biophonia,

Thanks for reaching out to us!

Unfortunately, I can not reproduce your error. I suspect some errors with your python environment / installed packages. Maybe some of the following steps could help:

• Eval on CPU only: Change line 7 in configs/experiment/birdset_neurips24/HSN/LT/convnext.yaml to:

    - override /trainer: default.yaml

  And reduce the batch size and number of workers in line 44-47 to:

    loaders:
      test:
        batch_size: 1
        num_workers: 1

  This will take quite some time.

• Our workstations and server are running Linux, you might be closer to our setup by developing inside a docker container: See https://github.com/raphaelschwinger/cuda-python-devcontainer/tree/5adcc63118789be6445dea6671ddcf35498df78e

[paulpeyret-biophonia]

Hi @raphaelschwinger

Thanks for you answer. I will try what you suggest + the dev_container option and let you know how it goes. 👍

[paulpeyret-biophonia]

The dev container worked fine. Thanks !

[raphaelschwinger]

@paulpeyret-biophonia Great! I am closing this issue then!