search

aleju / imgaug

Image augmentation for machine learning experiments.
http://imgaug.readthedocs.io
MIT License
14.42k stars 2.44k forks source link

AddToHue _LUT_CACHE is NoneType #738

Open BlGene opened 3 years ago

BlGene commented 3 years ago

When running the moco code here with imgaug augmentations, using the following guide to making them compatible with pytorch transforms here I get the following error (see below).

It looks to me like the _LUT_CACHE of AddToHue gets truncated by the pickeling of the spawn.

I managed to make the error go away by changing the AddToHue backend away from cv2.

class ImgAugTransform:
  def __init__(self):
      # augmentation
      self.aug = iaa.Sequential([
          ...
          iaa.AddToHue((-10, 10)),
      ], random_order=True)
      self.aug[-1].backend = "numpy" 
Traceback (most recent call last):
  File "main_moco.py", line 437, in <module>
    main()
  File "main_moco.py", line 132, in main
    mp.spawn(main_worker, nprocs=ngpus_per_node, args=(ngpus_per_node, args))
  File "XXX/miniconda3/envs/fh-ss/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 200, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "XXX/miniconda3/envs/fh-ss/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 158, in start_processes
    while not context.join():
  File "XXX/miniconda3/envs/fh-ss/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 119, in join
    raise Exception(msg)
Exception: 

-- Process 3 terminated with the following error:
Traceback (most recent call last):
  File "XXX/miniconda3/envs/fh-ss/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
    fn(i, *args)
  File "XXX/main_moco.py", line 301, in main_worker
    train(train_loader, model, criterion, optimizer, epoch, args)
  File "XXX/main_moco.py", line 328, in train
    for i, (images, _) in enumerate(train_loader):
  File "XXX/miniconda3/envs/fh-ss/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 363, in __next__
    data = self._next_data()
  File "XXX/miniconda3/envs/fh-ss/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 989, in _next_data
    return self._process_data(data)
  File "XXX/miniconda3/envs/fh-ss/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 1014, in _process_data
    data.reraise()
  File "XXX/miniconda3/envs/fh-ss/lib/python3.6/site-packages/torch/_utils.py", line 395, in reraise
    raise self.exc_type(msg)
TypeError: Caught TypeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "XXX/miniconda3/envs/fh-ss/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 185, in _worker_loop
    data = fetcher.fetch(index)
  File "XXX/miniconda3/envs/fh-ss/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "XXX/miniconda3/envs/fh-ss/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "XXX/DatasetUnsupervisedExemplar.py", line 204, in __getitem__
    sample1 = self.transform(sample1)
  File "XXX/miniconda3/envs/fh-ss/lib/python3.6/site-packages/torchvision/transforms/transforms.py", line 61, in __call__
    img = t(img)
  File "XXX/DatasetUnsupervisedExemplar.py", line 245, in __call__
    return self.aug.augment_image(img)
  File "XXX/miniconda3/envs/fh-ss/lib/python3.6/site-packages/imgaug/augmenters/meta.py", line 771, in augment_image
    return self.augment_images([image], hooks=hooks)[0]
  File "XXX/miniconda3/envs/fh-ss/lib/python3.6/site-packages/imgaug/augmenters/meta.py", line 825, in augment_images
    hooks=hooks
  File "XXX/miniconda3/envs/fh-ss/lib/python3.6/site-packages/imgaug/augmenters/meta.py", line 645, in augment_batch_
    hooks=hooks)
  File "XXX/miniconda3/envs/fh-ss/lib/python3.6/site-packages/imgaug/augmenters/meta.py", line 3127, in _augment_batch_
    hooks=hooks
  File "XXX/miniconda3/envs/fh-ss/lib/python3.6/site-packages/imgaug/augmenters/meta.py", line 645, in augment_batch_
    hooks=hooks)
  File "XXX/miniconda3/envs/fh-ss/lib/python3.6/site-packages/imgaug/augmenters/color.py", line 2471, in _augment_batch_
    image_hsv, hue_i, saturation_i)
  File "XXX/miniconda3/envs/fh-ss/lib/python3.6/site-packages/imgaug/augmenters/color.py", line 2501, in _transform_image_cv2
    table_hue = cls._LUT_CACHE[0]
TypeError: 'NoneType' object is not subscriptable

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "XXX/miniconda3/envs/fh-ss/lib/python3.6/multiprocessing/spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "XXX/miniconda3/envs/fh-ss/lib/python3.6/multiprocessing/spawn.py", line 115, in _main
    self = reduction.pickle.load(from_parent)
_pickle.UnpicklingError: pickle data was truncated
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "XXX/miniconda3/envs/fh-ss/lib/python3.6/multiprocessing/spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "XXX/miniconda3/envs/fh-ss/lib/python3.6/multiprocessing/spawn.py", line 115, in _main
    self = reduction.pickle.load(from_parent)
_pickle.UnpicklingError: pickle data was truncated
XXX/miniconda3/envs/fh-ss/lib/python3.6/multiprocessing/semaphore_tracker.py:143: UserWarning: semaphore_tracker: There appear to be 32 leaked semaphores to clean up at shutdown
  len(cache))
XXX/miniconda3/envs/fh-ss/lib/python3.6/multiprocessing/semaphore_tracker.py:143: UserWarning: semaphore_tracker: There appear to be 32 leaked semaphores to clean up at shutdown
  len(cache))
shuishiwojiade commented 3 years ago

Set num_workers of dataloader to 0.