search

facebookresearch / hydra

Hydra is a framework for elegantly configuring complex applications
https://hydra.cc
MIT License
8.81k stars 635 forks source link

Error executing job with overrides: ['task=detect', 'mode=train', 'model=yolov8s.pt', 'data=ChePaiJianCe\\data.yaml', 'epochs=100', 'imgsz=640', 'device=0'][Bug] #2552

Closed DarkAdu closed 1 year ago

DarkAdu commented 1 year ago

yolov8

Everything is normal when using cpu to train, but it will report an error if it is replaced by gpu

C:\Users\Cola>yolo task=detect mode=train model=yolov8s.pt data=ChePaiJianCe\data.yaml epochs=100 imgsz=640 device=0
yolo\engine\trainer: task=detect, mode=train, model=yolov8s.pt, data=ChePaiJianCe\data.yaml, epochs=100, patience=50, batch=16, imgsz=640, save=True, cache=False, device=0, workers=8, project=None, name=None, exist_ok=False, pretrained=False, optimizer=SGD, verbose=False, seed=0, deterministic=True, single_cls=False, image_weights=False, rect=False, cos_lr=False, close_mosaic=10, resume=False, overlap_mask=True, mask_ratio=4, dropout=0.0, val=True, save_json=False, save_hybrid=False, conf=None, iou=0.7, max_det=300, half=False, dnn=False, plots=True, source=None, show=False, save_txt=False, save_conf=False, save_crop=False, hide_labels=False, hide_conf=False, vid_stride=1, line_thickness=3, visualize=False, augment=False, agnostic_nms=False, retina_masks=False, format=torchscript, keras=False, optimize=False, int8=False, dynamic=False, simplify=False, opset=17, workspace=4, nms=False, lr0=0.01, lrf=0.01, momentum=0.937, weight_decay=0.0005, warmup_epochs=3.0, warmup_momentum=0.8, warmup_bias_lr=0.1, box=7.5, cls=0.5, dfl=1.5, fl_gamma=0.0, label_smoothing=0.0, nbs=64, hsv_h=0.015, hsv_s=0.7, hsv_v=0.4, degrees=0.0, translate=0.1, scale=0.5, shear=0.0, perspective=0.0, flipud=0.0, fliplr=0.5, mosaic=1.0, mixup=0.0, copy_paste=0.0, hydra={'output_subdir': None, 'run': {'dir': '.'}}, v5loader=False, save_dir=runs\detect\train6
Ultralytics YOLOv8.0.3  Python-3.9.13 torch-1.13.1+cu117 CUDA:0 (NVIDIA GeForce RTX 3070 Ti Laptop GPU, 8192MiB)
Overriding model.yaml nc=80 with nc=5

                   from  n    params  module                                       arguments
  0                  -1  1       928  ultralytics.nn.modules.Conv                  [3, 32, 3, 2]
  1                  -1  1     18560  ultralytics.nn.modules.Conv                  [32, 64, 3, 2]
  2                  -1  1     29056  ultralytics.nn.modules.C2f                   [64, 64, 1, True]
  3                  -1  1     73984  ultralytics.nn.modules.Conv                  [64, 128, 3, 2]
  4                  -1  2    197632  ultralytics.nn.modules.C2f                   [128, 128, 2, True]
  5                  -1  1    295424  ultralytics.nn.modules.Conv                  [128, 256, 3, 2]
  6                  -1  2    788480  ultralytics.nn.modules.C2f                   [256, 256, 2, True]
  7                  -1  1   1180672  ultralytics.nn.modules.Conv                  [256, 512, 3, 2]
  8                  -1  1   1838080  ultralytics.nn.modules.C2f                   [512, 512, 1, True]
  9                  -1  1    656896  ultralytics.nn.modules.SPPF                  [512, 512, 5]
 10                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']
 11             [-1, 6]  1         0  ultralytics.nn.modules.Concat                [1]
 12                  -1  1    591360  ultralytics.nn.modules.C2f                   [768, 256, 1]
 13                  -1  1         0  torch.nn.modules.upsampling.Upsample         [None, 2, 'nearest']
 14             [-1, 4]  1         0  ultralytics.nn.modules.Concat                [1]
 15                  -1  1    148224  ultralytics.nn.modules.C2f                   [384, 128, 1]
 16                  -1  1    147712  ultralytics.nn.modules.Conv                  [128, 128, 3, 2]
 17            [-1, 12]  1         0  ultralytics.nn.modules.Concat                [1]
 18                  -1  1    493056  ultralytics.nn.modules.C2f                   [384, 256, 1]
 19                  -1  1    590336  ultralytics.nn.modules.Conv                  [256, 256, 3, 2]
 20             [-1, 9]  1         0  ultralytics.nn.modules.Concat                [1]
 21                  -1  1   1969152  ultralytics.nn.modules.C2f                   [768, 512, 1]
 22        [15, 18, 21]  1   2117983  ultralytics.nn.modules.Detect                [5, [128, 256, 512]]
Model summary: 225 layers, 11137535 parameters, 11137519 gradients, 28.7 GFLOPs

Transferred 349/355 items from pretrained weights
optimizer: SGD(lr=0.01) with parameter groups 57 weight(decay=0.0), 64 weight(decay=0.0005), 63 bias
train: Scanning C:\Users\Cola\ChePaiJianCe\train\labels.cache... 243 images, 0 backgrounds, 0 corrupt: 100%|██████████|
val: Scanning C:\Users\Cola\ChePaiJianCe\valid\labels.cache... 21 images, 0 backgrounds, 0 corrupt: 100%|██████████| 21
Image sizes 640 train, 640 val
Using 8 dataloader workers
Logging results to runs\detect\train6
Starting training for 100 epochs...

      Epoch    GPU_mem   box_loss   cls_loss   dfl_loss  Instances       Size
      1/100      3.86G      1.201      5.759      1.288          7        640: 100%|██████████| 16/16 [00:07<00:00,  2.
                 Class     Images  Instances      Box(P          R      mAP50  mAP50-95):   0%|          | 0/1 [00:06<?
Error executing job with overrides: ['task=detect', 'mode=train', 'model=yolov8s.pt', 'data=ChePaiJianCe\\data.yaml', 'epochs=100', 'imgsz=640', 'device=0']
Traceback (most recent call last):
  File "C:\Users\Cola\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\Cola\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\Cola\AppData\Local\Programs\Python\Python39\Scripts\yolo.exe\__main__.py", line 7, in <module>
  File "C:\Users\Cola\AppData\Local\Programs\Python\Python39\lib\site-packages\hydra\main.py", line 90, in decorated_main
    _run_hydra(
  File "C:\Users\Cola\AppData\Local\Programs\Python\Python39\lib\site-packages\hydra\_internal\utils.py", line 394, in _run_hydra
    _run_app(
  File "C:\Users\Cola\AppData\Local\Programs\Python\Python39\lib\site-packages\hydra\_internal\utils.py", line 457, in _run_app
    run_and_report(
  File "C:\Users\Cola\AppData\Local\Programs\Python\Python39\lib\site-packages\hydra\_internal\utils.py", line 222, in run_and_report
    raise ex
  File "C:\Users\Cola\AppData\Local\Programs\Python\Python39\lib\site-packages\hydra\_internal\utils.py", line 219, in run_and_report
    return func()
  File "C:\Users\Cola\AppData\Local\Programs\Python\Python39\lib\site-packages\hydra\_internal\utils.py", line 458, in <lambda>
    lambda: hydra.run(
  File "C:\Users\Cola\AppData\Local\Programs\Python\Python39\lib\site-packages\hydra\_internal\hydra.py", line 132, in run
    _ = ret.return_value
  File "C:\Users\Cola\AppData\Local\Programs\Python\Python39\lib\site-packages\hydra\core\utils.py", line 260, in return_value
    raise self._return_value
  File "C:\Users\Cola\AppData\Local\Programs\Python\Python39\lib\site-packages\hydra\core\utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "C:\Users\Cola\AppData\Local\Programs\Python\Python39\lib\site-packages\ultralytics\yolo\cli.py", line 52, in cli
    func(cfg)
  File "C:\Users\Cola\AppData\Local\Programs\Python\Python39\lib\site-packages\hydra\main.py", line 79, in decorated_main
    return task_function(cfg_passthrough)
  File "C:\Users\Cola\AppData\Local\Programs\Python\Python39\lib\site-packages\ultralytics\yolo\v8\detect\train.py", line 206, in train
    model.train(**cfg)
  File "C:\Users\Cola\AppData\Local\Programs\Python\Python39\lib\site-packages\ultralytics\yolo\engine\model.py", line 193, in train
    self.trainer.train()
  File "C:\Users\Cola\AppData\Local\Programs\Python\Python39\lib\site-packages\ultralytics\yolo\engine\trainer.py", line 177, in train
    self._do_train(int(os.getenv("RANK", -1)), world_size)
  File "C:\Users\Cola\AppData\Local\Programs\Python\Python39\lib\site-packages\ultralytics\yolo\engine\trainer.py", line 332, in _do_train
    self.metrics, self.fitness = self.validate()
  File "C:\Users\Cola\AppData\Local\Programs\Python\Python39\lib\site-packages\ultralytics\yolo\engine\trainer.py", line 418, in validate
    metrics = self.validator(self)
  File "C:\Users\Cola\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "C:\Users\Cola\AppData\Local\Programs\Python\Python39\lib\site-packages\ultralytics\yolo\engine\validator.py", line 151, in __call__
    preds = self.postprocess(preds)
  File "C:\Users\Cola\AppData\Local\Programs\Python\Python39\lib\site-packages\ultralytics\yolo\v8\detect\val.py", line 62, in postprocess
    preds = ops.non_max_suppression(preds,
  File "C:\Users\Cola\AppData\Local\Programs\Python\Python39\lib\site-packages\ultralytics\yolo\utils\ops.py", line 212, in non_max_suppression
    i = torchvision.ops.nms(boxes, scores, iou_thres)  # NMS
  File "C:\Users\Cola\AppData\Local\Programs\Python\Python39\lib\site-packages\torchvision\ops\boxes.py", line 41, in nms
    return torch.ops.torchvision.nms(boxes, scores, iou_threshold)
  File "C:\Users\Cola\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\_ops.py", line 442, in __call__
    return self._op(*args, **kwargs or {})
NotImplementedError: Could not run 'torchvision::nms' with arguments from the 'CUDA' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'torchvision::nms' is only available for these backends: [CPU, QuantizedCPU, BackendSelect, Python, FuncTorchDynamicLayerBackMode, Functionalize, Named, Conjugate, Negative, ZeroTensor, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, AutogradMPS, AutogradXPU, AutogradHPU, AutogradLazy, Tracer, AutocastCPU, AutocastCUDA, FuncTorchBatched, FuncTorchVmapMode, Batched, VmapMode, FuncTorchGradWrapper, PythonTLSSnapshot, FuncTorchDynamicLayerFrontMode, PythonDispatcher].

CPU: registered at C:\Users\circleci\project\torchvision\csrc\ops\cpu\nms_kernel.cpp:112 [kernel]
QuantizedCPU: registered at C:\Users\circleci\project\torchvision\csrc\ops\quantized\cpu\qnms_kernel.cpp:124 [kernel]
BackendSelect: fallthrough registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\core\BackendSelectFallbackKernel.cpp:3 [backend fallback]
Python: registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\core\PythonFallbackKernel.cpp:140 [backend fallback]
FuncTorchDynamicLayerBackMode: registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\functorch\DynamicLayer.cpp:488 [backend fallback]
Functionalize: registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\FunctionalizeFallbackKernel.cpp:291 [backend fallback]
Named: registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\core\NamedRegistrations.cpp:7 [backend fallback]
Conjugate: registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\ConjugateFallback.cpp:18 [backend fallback]
Negative: registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\NegateFallback.cpp:18 [backend fallback]
ZeroTensor: registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\ZeroTensorFallback.cpp:86 [backend fallback]
ADInplaceOrView: fallthrough registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\core\VariableFallbackKernel.cpp:64 [backend fallback]
AutogradOther: fallthrough registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\core\VariableFallbackKernel.cpp:35 [backend fallback]
AutogradCPU: fallthrough registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\core\VariableFallbackKernel.cpp:39 [backend fallback]
AutogradCUDA: fallthrough registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\core\VariableFallbackKernel.cpp:47 [backend fallback]
AutogradXLA: fallthrough registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\core\VariableFallbackKernel.cpp:51 [backend fallback]
AutogradMPS: fallthrough registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\core\VariableFallbackKernel.cpp:59 [backend fallback]
AutogradXPU: fallthrough registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\core\VariableFallbackKernel.cpp:43 [backend fallback]
AutogradHPU: fallthrough registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\core\VariableFallbackKernel.cpp:68 [backend fallback]
AutogradLazy: fallthrough registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\core\VariableFallbackKernel.cpp:55 [backend fallback]
Tracer: registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\torch\csrc\autograd\TraceTypeManual.cpp:296 [backend fallback]
AutocastCPU: fallthrough registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\autocast_mode.cpp:482 [backend fallback]
AutocastCUDA: fallthrough registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\autocast_mode.cpp:324 [backend fallback]
FuncTorchBatched: registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\functorch\LegacyBatchingRegistrations.cpp:743 [backend fallback]
FuncTorchVmapMode: fallthrough registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\functorch\VmapModeRegistrations.cpp:28 [backend fallback]
Batched: registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\BatchingRegistrations.cpp:1064 [backend fallback]
VmapMode: fallthrough registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\VmapModeRegistrations.cpp:33 [backend fallback]
FuncTorchGradWrapper: registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\functorch\TensorWrapper.cpp:189 [backend fallback]
PythonTLSSnapshot: registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\core\PythonFallbackKernel.cpp:148 [backend fallback]
FuncTorchDynamicLayerFrontMode: registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\functorch\DynamicLayer.cpp:484 [backend fallback]
PythonDispatcher: registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\core\PythonFallbackKernel.cpp:144 [backend fallback]
omry commented 1 year ago

Sorry, but we can't debug your program for you.