Error while running create_heatmaps.py --> ValueError: ctypes objects containing pointers cannot be pickled

[AlmashAlam]

I tried running the command python create_heatmaps.py --config config_template.yaml

And ran into this error

Traceback (most recent call last):
  File "create_heatmaps.py", line 289, in <module>
    _, _, wsi_object = compute_from_patches(wsi_object=wsi_object,
  File "D:\CLAM\vis_utils\heatmap_utils.py", line 64, in compute_from_patches
    for idx, (roi, coords) in enumerate(roi_loader):
  File "E:\anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 359, in __iter__
    return self._get_iterator()
  File "E:\anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 305, in _get_iterator
    return _MultiProcessingDataLoaderIter(self)
  File "E:\anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 918, in __init__
    w.start()
  File "E:\anaconda3\lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
  File "E:\anaconda3\lib\multiprocessing\context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "E:\anaconda3\lib\multiprocessing\context.py", line 327, in _Popen
    return Popen(process_obj)
  File "E:\anaconda3\lib\multiprocessing\popen_spawn_win32.py", line 93, in __init__
    reduction.dump(process_obj, to_child)
  File "E:\anaconda3\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
ValueError: ctypes objects containing pointers cannot be pickled

System configuration:

• torchvision==0.10.0+cu111
• OS: Windows 10

Could you please provide any guidance on how to resolve this error?

[fedshyvana]

https://github.com/MSKCC-Computational-Pathology/MIL-nature-medicine-2019/issues/4 I feel like this is the same/similar issue with using Pytorch dataloader with openslide. If you can, running the code in Linux should solve the issue.

[fedshyvana]

Maybe you can try editing vis_utils/heatmap_utils.py, line 55:

roi_loader = get_simple_loader(roi_dataset, batch_size=batch_size, num_workers=8)

to use num_workers = 0, and see if the issue gets resolved although it will potentially make the code run a lot slower.