search

roboflow / notebooks

Examples and tutorials on using SOTA computer vision models and techniques. Learn everything from old-school ResNet, through YOLO and object-detection transformers like DETR, to the latest models like Grounding DINO and SAM.
https://roboflow.com/models
5.09k stars 788 forks source link

ValueError: Caught ValueError in DataLoader worker process 0. #53

Closed Raiseku closed 1 year ago

Raiseku commented 1 year ago

Search before asking

Notebook name

https://colab.research.google.com/github/roboflow-ai/notebooks/blob/main/notebooks/train-yolov8-instance-segmentation-on-custom-dataset.ipynb

Bug


/content
Downloading https://github.com/ultralytics/assets/releases/download/v0.0.0/yolov8s-seg.pt to yolov8s-seg.pt...
100% 22.8M/22.8M [00:00<00:00, 150MB/s]

Ultralytics YOLOv8.0.11 🚀 Python-3.8.10 torch-1.13.1+cu116 CUDA:0 (Tesla T4, 15110MiB)
yolo/engine/trainer: task=segment, mode=train, model=yolov8s-seg.pt, data=/content/datasets/TicketDetection-3/data.yaml, epochs=15, patience=50, batch=16, imgsz=640, save=True, cache=False, device=, workers=8, project=None, name=None, exist_ok=False, pretrained=False, optimizer=SGD, verbose=False, seed=0, deterministic=True, single_cls=False, image_weights=False, rect=False, cos_lr=False, close_mosaic=10, resume=False, overlap_mask=True, mask_ratio=4, dropout=0.0, val=True, save_json=False, save_hybrid=False, conf=None, iou=0.7, max_det=300, half=False, dnn=False, plots=True, source=None, show=False, save_txt=False, save_conf=False, save_crop=False, hide_labels=False, hide_conf=False, vid_stride=1, line_thickness=3, visualize=False, augment=False, agnostic_nms=False, retina_masks=False, format=torchscript, keras=False, optimize=False, int8=False, dynamic=False, simplify=False, opset=17, workspace=4, nms=False, lr0=0.01, lrf=0.01, momentum=0.937, weight_decay=0.0005, warmup_epochs=3.0, warmup_momentum=0.8, warmup_bias_lr=0.1, box=7.5, cls=0.5, dfl=1.5, fl_gamma=0.0, label_smoothing=0.0, nbs=64, hsv_h=0.015, hsv_s=0.7, hsv_v=0.4, degrees=0.0, translate=0.1, scale=0.5, shear=0.0, perspective=0.0, flipud=0.0, fliplr=0.5, mosaic=1.0, mixup=0.0, copy_paste=0.0, cfg=None, hydra={'output_subdir': None, 'run': {'dir': '.'}}, v5loader=False, save_dir=runs/segment/train
Downloading https://ultralytics.com/assets/Arial.ttf to /root/.config/Ultralytics/Arial.ttf...
100% 755k/755k [00:00<00:00, 87.0MB/s]
Overriding model.yaml nc=80 with nc=1

                   from  n    params  module                                       arguments                     
  0                  -1  1       928  ultralytics.nn.modules.Conv                  [3, 32, 3, 2]                 
  1                  -1  1     18560  ultralytics.nn.modules.Conv                  [32, 64, 3, 2]                
  2                  -1  1     29056  ultralytics.nn.modules.C2f                   [64, 64, 1, True]             
  3                  -1  1     73984  ultralytics.nn.modules.Conv                  [64, 128, 3, 2]               
  4                  -1  2    197632  ultralytics.nn.modules.C2f                   [128, 128, 2, True]           
  5                  -1  1    295424  ultralytics.nn.modules.Conv                  [128, 256, 3, 2]              
  6                  -1  2    788480  ultralytics.nn.modules.C2f                   [256, 256, 2, True]           
  7                  -1  1   1180672  ultralytics.nn.modules.Conv                  [256, 512, 3, 2]              
  8                  -1  1   1838080  ultralytics.nn.modules.C2f                   [512, 512, 1, True]           
  9                  -1  1    656896  ultralytics.nn.modules.SPPF                  [512, 512, 5]                 
 10                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']          
 11             [-1, 6]  1         0  ultralytics.nn.modules.Concat                [1]                           
 12                  -1  1    591360  ultralytics.nn.modules.C2f                   [768, 256, 1]                 
 13                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']          
 14             [-1, 4]  1         0  ultralytics.nn.modules.Concat                [1]                           
 15                  -1  1    148224  ultralytics.nn.modules.C2f                   [384, 128, 1]                 
 16                  -1  1    147712  ultralytics.nn.modules.Conv                  [128, 128, 3, 2]              
 17            [-1, 12]  1         0  ultralytics.nn.modules.Concat                [1]                           
 18                  -1  1    493056  ultralytics.nn.modules.C2f                   [384, 256, 1]                 
 19                  -1  1    590336  ultralytics.nn.modules.Conv                  [256, 256, 3, 2]              
 20             [-1, 9]  1         0  ultralytics.nn.modules.Concat                [1]                           
 21                  -1  1   1969152  ultralytics.nn.modules.C2f                   [768, 512, 1]                 
 22        [15, 18, 21]  1   2770931  ultralytics.nn.modules.Segment               [1, 32, 128, [128, 256, 512]] 
YOLOv8s-seg summary: 261 layers, 11790483 parameters, 11790467 gradients, 42.7 GFLOPs

Transferred 411/417 items from pretrained weights
optimizer: SGD(lr=0.01) with parameter groups 66 weight(decay=0.0), 77 weight(decay=0.0005), 76 bias
train: Scanning /content/datasets/TicketDetection-3/train/labels... 150 images, 0 backgrounds, 0 corrupt: 100% 150/150 [00:00<00:00, 1433.15it/s]
train: New cache created: /content/datasets/TicketDetection-3/train/labels.cache
albumentations: Blur(p=0.01, blur_limit=(3, 7)), MedianBlur(p=0.01, blur_limit=(3, 7)), ToGray(p=0.01), CLAHE(p=0.01, clip_limit=(1, 4.0), tile_grid_size=(8, 8))
val: Scanning /content/datasets/TicketDetection-3/valid/labels... 14 images, 0 backgrounds, 0 corrupt: 100% 14/14 [00:00<00:00, 637.08it/s]
val: New cache created: /content/datasets/TicketDetection-3/valid/labels.cache
Image sizes 640 train, 640 val
Using 2 dataloader workers
Logging results to runs/segment/train
Starting training for 15 epochs...

      Epoch    GPU_mem   box_loss   seg_loss   cls_loss   dfl_loss  Instances       Size
  0% 0/10 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/usr/local/bin/yolo", line 8, in <module>
    sys.exit(entrypoint())
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/cli.py", line 148, in entrypoint
    cli(cfg)
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/cli.py", line 84, in cli
    func(cfg)
  File "/usr/local/lib/python3.8/dist-packages/hydra/main.py", line 79, in decorated_main
    return task_function(cfg_passthrough)
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/v8/segment/train.py", line 153, in train
    model.train(**cfg)
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/engine/model.py", line 203, in train
    self.trainer.train()
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/engine/trainer.py", line 185, in train
    self._do_train(int(os.getenv("RANK", -1)), world_size)
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/engine/trainer.py", line 285, in _do_train
    for i, batch in pbar:
  File "/usr/local/lib/python3.8/dist-packages/tqdm/std.py", line 1195, in __iter__
    for obj in iterable:
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/dataloader.py", line 628, in __next__
    data = self._next_data()
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/dataloader.py", line 1333, in _next_data
    return self._process_data(data)
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/dataloader.py", line 1359, in _process_data
    data.reraise()
  File "/usr/local/lib/python3.8/dist-packages/torch/_utils.py", line 543, in reraise
    raise exception
ValueError: Caught ValueError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
    data = fetcher.fetch(index)
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/_utils/fetch.py", line 58, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/_utils/fetch.py", line 58, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/data/base.py", line 179, in __getitem__
    return self.transforms(self.get_label_info(index))
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/data/augment.py", line 48, in __call__
    data = t(data)
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/data/augment.py", line 48, in __call__
    data = t(data)
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/data/augment.py", line 361, in __call__
    i = self.box_candidates(box1=instances.bboxes.T,
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/data/augment.py", line 375, in box_candidates
    return (w2 > wh_thr) & (h2 > wh_thr) & (w2 * h2 / (w1 * h1 + eps) > area_thr) & (ar < ar_thr)  # candidates
ValueError: operands could not be broadcast together with shapes (3,) (4,)

Environment

Google Colab

Minimal Reproducible Example

!nvidia-smi
import os
HOME = os.getcwd()
print(HOME)
!pip install ultralytics==8.0.11

from IPython import display
display.clear_output()

import ultralytics
ultralytics.checks()
from ultralytics import YOLO

from IPython.display import display, Image
!mkdir {HOME}/datasets
%cd {HOME}/datasets
!pip install roboflow
display.clear_output()
from roboflow import Roboflow
rf = Roboflow(api_key="#####")
project = rf.workspace("muni").project("ticketdetection")
dataset = project.version(3).download("yolov5")
%cd {HOME}

!yolo task=segment mode=train model=yolov8s-seg.pt data={dataset.location}/data.yaml epochs=15 imgsz=640

Additional

The other important error i think is: ValueError: operands could not be broadcast together with shapes (3,) (4,)

Can someone help me?

Are you willing to submit a PR?

github-actions[bot] commented 1 year ago

👋 Hello @Raiseku, thank you for leaving an issue on Roboflow Notebooks.

🐞 Bug reports

If you are filing a bug report, please be as detailed as possible. This will help us more easily diagnose and resolve the problem you are facing. To learn more about contributing, check out our Contributing Guidelines.

If you require support with custom code that is not part of Roboflow Notebooks, please reach out on the Roboflow Forum or on the GitHub Discussions page associated with this repository.

💬 Get in touch

Do you have more questions about Roboflow that we haven't responded to yet? Feel free to ask them on the Roboflow Discuss forum. Our developer advocates and community team actively respond to questions there.

To ask questions about Notebooks, head over to the GitHub Discussions section of this repository.

SkalskiP commented 1 year ago

@Raiseku could you try changing:

!yolo task=segment mode=train model=yolov8s-seg.pt data={dataset.location}/data.yaml epochs=15 imgsz=640

to:

!yolo task=segment mode=train model=yolov8s-seg.pt data={dataset.location}/data.yaml epochs=15 imgsz=640 v5loader=True

?

Raiseku commented 1 year ago

Thanks for the answer! i changed the code to: !yolo task=segment mode=train model=yolov8s-seg.pt data={dataset.location}/data.yaml epochs=15 imgsz=640 v5loader=True

but now its giving me this error:

/content
Downloading https://github.com/ultralytics/assets/releases/download/v0.0.0/yolov8s-seg.pt to yolov8s-seg.pt...
100% 22.8M/22.8M [00:01<00:00, 17.2MB/s]

Ultralytics YOLOv8.0.11 🚀 Python-3.8.10 torch-1.13.1+cu116 CUDA:0 (Tesla T4, 15110MiB)
yolo/engine/trainer: task=segment, mode=train, model=yolov8s-seg.pt, data=/content/datasets/TicketDetection-3/data.yaml, epochs=15, patience=50, batch=16, imgsz=640, save=True, cache=False, device=, workers=8, project=None, name=None, exist_ok=False, pretrained=False, optimizer=SGD, verbose=False, seed=0, deterministic=True, single_cls=False, image_weights=False, rect=False, cos_lr=False, close_mosaic=10, resume=False, overlap_mask=True, mask_ratio=4, dropout=0.0, val=True, save_json=False, save_hybrid=False, conf=None, iou=0.7, max_det=300, half=False, dnn=False, plots=True, source=None, show=False, save_txt=False, save_conf=False, save_crop=False, hide_labels=False, hide_conf=False, vid_stride=1, line_thickness=3, visualize=False, augment=False, agnostic_nms=False, retina_masks=False, format=torchscript, keras=False, optimize=False, int8=False, dynamic=False, simplify=False, opset=17, workspace=4, nms=False, lr0=0.01, lrf=0.01, momentum=0.937, weight_decay=0.0005, warmup_epochs=3.0, warmup_momentum=0.8, warmup_bias_lr=0.1, box=7.5, cls=0.5, dfl=1.5, fl_gamma=0.0, label_smoothing=0.0, nbs=64, hsv_h=0.015, hsv_s=0.7, hsv_v=0.4, degrees=0.0, translate=0.1, scale=0.5, shear=0.0, perspective=0.0, flipud=0.0, fliplr=0.5, mosaic=1.0, mixup=0.0, copy_paste=0.0, cfg=None, hydra={'output_subdir': None, 'run': {'dir': '.'}}, v5loader=True, save_dir=runs/segment/train
Downloading https://ultralytics.com/assets/Arial.ttf to /root/.config/Ultralytics/Arial.ttf...
100% 755k/755k [00:00<00:00, 135MB/s]
Overriding model.yaml nc=80 with nc=1

                   from  n    params  module                                       arguments                     
  0                  -1  1       928  ultralytics.nn.modules.Conv                  [3, 32, 3, 2]                 
  1                  -1  1     18560  ultralytics.nn.modules.Conv                  [32, 64, 3, 2]                
  2                  -1  1     29056  ultralytics.nn.modules.C2f                   [64, 64, 1, True]             
  3                  -1  1     73984  ultralytics.nn.modules.Conv                  [64, 128, 3, 2]               
  4                  -1  2    197632  ultralytics.nn.modules.C2f                   [128, 128, 2, True]           
  5                  -1  1    295424  ultralytics.nn.modules.Conv                  [128, 256, 3, 2]              
  6                  -1  2    788480  ultralytics.nn.modules.C2f                   [256, 256, 2, True]           
  7                  -1  1   1180672  ultralytics.nn.modules.Conv                  [256, 512, 3, 2]              
  8                  -1  1   1838080  ultralytics.nn.modules.C2f                   [512, 512, 1, True]           
  9                  -1  1    656896  ultralytics.nn.modules.SPPF                  [512, 512, 5]                 
 10                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']          
 11             [-1, 6]  1         0  ultralytics.nn.modules.Concat                [1]                           
 12                  -1  1    591360  ultralytics.nn.modules.C2f                   [768, 256, 1]                 
 13                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']          
 14             [-1, 4]  1         0  ultralytics.nn.modules.Concat                [1]                           
 15                  -1  1    148224  ultralytics.nn.modules.C2f                   [384, 128, 1]                 
 16                  -1  1    147712  ultralytics.nn.modules.Conv                  [128, 128, 3, 2]              
 17            [-1, 12]  1         0  ultralytics.nn.modules.Concat                [1]                           
 18                  -1  1    493056  ultralytics.nn.modules.C2f                   [384, 256, 1]                 
 19                  -1  1    590336  ultralytics.nn.modules.Conv                  [256, 256, 3, 2]              
 20             [-1, 9]  1         0  ultralytics.nn.modules.Concat                [1]                           
 21                  -1  1   1969152  ultralytics.nn.modules.C2f                   [768, 512, 1]                 
 22        [15, 18, 21]  1   2770931  ultralytics.nn.modules.Segment               [1, 32, 128, [128, 256, 512]] 
YOLOv8s-seg summary: 261 layers, 11790483 parameters, 11790467 gradients, 42.7 GFLOPs

Transferred 411/417 items from pretrained weights
optimizer: SGD(lr=0.01) with parameter groups 66 weight(decay=0.0), 77 weight(decay=0.0005), 76 bias
albumentations: Blur(p=0.01, blur_limit=(3, 7)), MedianBlur(p=0.01, blur_limit=(3, 7)), ToGray(p=0.01), CLAHE(p=0.01, clip_limit=(1, 4.0), tile_grid_size=(8, 8))
train: Scanning /content/datasets/TicketDetection-3/train/labels... 150 images, 0 backgrounds, 0 corrupt: 100% 150/150 [00:00<00:00, 1414.79it/s]
train: New cache created: /content/datasets/TicketDetection-3/train/labels.cache
val: Scanning /content/datasets/TicketDetection-3/valid/labels... 14 images, 0 backgrounds, 0 corrupt: 100% 14/14 [00:00<00:00, 422.50it/s]
val: New cache created: /content/datasets/TicketDetection-3/valid/labels.cache
Image sizes 640 train, 640 val
Using 2 dataloader workers
Logging results to runs/segment/train
Starting training for 15 epochs...

      Epoch    GPU_mem   box_loss   seg_loss   cls_loss   dfl_loss  Instances       Size
  0% 0/10 [00:07<?, ?it/s]
Traceback (most recent call last):
  File "/usr/local/bin/yolo", line 8, in <module>
    sys.exit(entrypoint())
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/cli.py", line 148, in entrypoint
    cli(cfg)
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/cli.py", line 84, in cli
    func(cfg)
  File "/usr/local/lib/python3.8/dist-packages/hydra/main.py", line 79, in decorated_main
    return task_function(cfg_passthrough)
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/v8/segment/train.py", line 153, in train
    model.train(**cfg)
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/engine/model.py", line 203, in train
    self.trainer.train()
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/engine/trainer.py", line 185, in train
    self._do_train(int(os.getenv("RANK", -1)), world_size)
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/engine/trainer.py", line 303, in _do_train
    self.loss, self.loss_items = self.criterion(preds, batch)
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/v8/segment/train.py", line 45, in criterion
    return self.compute_loss(preds, batch)
  File "/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/v8/segment/train.py", line 91, in __call__
    masks = batch["masks"].to(self.device).float()
KeyError: 'masks'
Sentry is attempting to send 2 pending error messages
Waiting up to 2 seconds
Press Ctrl-C to quit
SkalskiP commented 1 year ago

From what I can see YOLOv8 team is working on solution for that problem right now: https://github.com/ultralytics/ultralytics/issues/548#issuecomment-1402241153

Laughing-q commented 1 year ago

@Raiseku It has been solved in https://github.com/ultralytics/ultralytics/pull/598 please upgrade you ultralytics package. :)

SkalskiP commented 1 year ago

@Laughing-q thanks for keeping us updated