Using mixup and mosaic at the same time, with 0 < p < 1

Xonxt commented 1 year ago

Search before asking

[X] I have searched the YOLOv8 issues and found no similar bug report.

YOLOv8 Component

Training

Bug

When setting the probabilities of both mixup and mosaic to a value 0 < p < 1, notice the strict "<" here (so, neither 0.0 nor 1.0, but in-between), the detection-training crashes with the following exception:

  File "C:\Program Files\Python39\lib\site-packages\ultralytics\yolo\engine\trainer.py", line 313, in _do_train
    for i, batch in pbar:
  File "C:\Program Files\Python39\lib\site-packages\tqdm\std.py", line 1195, in __iter__
    for obj in iterable:
  File "C:\Program Files\Python39\lib\site-packages\ultralytics\yolo\data\build.py", line 38, in __iter__
    yield next(self.iterator)
  File "C:\Program Files\Python39\lib\site-packages\torch\utils\data\dataloader.py", line 628, in __next__
    data = self._next_data()
  File "C:\Program Files\Python39\lib\site-packages\torch\utils\data\dataloader.py", line 1333, in _next_data
    return self._process_data(data)
  File "C:\Program Files\Python39\lib\site-packages\torch\utils\data\dataloader.py", line 1359, in _process_data
    data.reraise()
  File "C:\Program Files\Python39\lib\site-packages\torch\_utils.py", line 543, in reraise
    raise exception
TypeError: Caught TypeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "C:\Program Files\Python39\lib\site-packages\torch\utils\data\_utils\worker.py", line 302, in _worker_loop
    data = fetcher.fetch(index)
  File "C:\Program Files\Python39\lib\site-packages\torch\utils\data\_utils\fetch.py", line 61, in fetch
    return self.collate_fn(data)
  File "C:\Program Files\Python39\lib\site-packages\ultralytics\yolo\data\dataset.py", line 192, in collate_fn
    value = torch.stack(value, 0)
TypeError: expected Tensor as element 1 in argument 0, but got tuple

referring of course to the def collate_fn(batch): function in ultralytics/data/dataset.py.

Example:

mixup: 0.5
mosaic: 0.5

Environment

Ultralytics YOLOv8.0.136 Python-3.9.13 torch-1.13.1+cu116 CUDA:0 (NVIDIA GeForce RTX 2080 Ti, 11264MiB)
OS: Windows 10
Python 3.9.13

Minimal Reproducible Example

mixup: 0.5
mosaic: 0.5

Additional

No response

Are you willing to submit a PR?

[ ] Yes I'd like to help by submitting a PR!

glenn-jocher commented 1 year ago

@Xonxt hello,

Thank you for the detailed description and carefully logging the issue. We appreciate your efforts to debug the problem.

The error you're talking about tends to occur when two types of image augmentations - in this case, mixup and mosaic - are applied simultaneously with probabilities that are neither 0 nor 1 but lie between them. At this juncture, it seems like the DataLoader is attempting to stack tuples rather than Tensor objects, leading to the TypeError you've reported.

This situation happens due to the fundamental differences between Mixup and Mosaic techniques. While Mixup creates a blend of two randomly chosen images, Mosaic takes four images and stitches them into a single one. Therefore, using both concurrently might result in conflicting data structure expectations, causing the crash.

Comprehensive testing and debugging are needed to fix this error. As of now, a typical workaround would be to avoid such simultaneous use or to ensure at least one of their probabilities is set to 0.

The Ultralytics team will review this issue and consider the necessary enhancements in the subsequent updates of the YOLOv8 model. As always, we value our community's feedback and appreciate your input.

Please continue to monitor our repository for updates and improvements.

Best Regards, Glenn Jocher.

Laughing-q commented 1 year ago

@Xonxt The training with mixup=0.5 and mosaic=0.5 goes fine to me:

github-actions[bot] commented 1 year ago

👋 Hello there! We wanted to give you a friendly reminder that this issue has not had any recent activity and may be closed soon, but don't worry - you can always reopen it if needed. If you still have any questions or concerns, please feel free to let us know how we can help.

For additional resources and information, please see the links below:

Docs: https://docs.ultralytics.com
HUB: https://hub.ultralytics.com
Community: https://community.ultralytics.com

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLO 🚀 and Vision AI ⭐

ultralytics / ultralytics