ultralytics / ultralytics

Ultralytics YOLO11 πŸš€
https://docs.ultralytics.com
GNU Affero General Public License v3.0
36.56k stars 7.05k forks source link

Benchmark crashes with YOLOv6 customized model #18883

Closed EmmanuelMess closed 3 weeks ago

EmmanuelMess commented 3 weeks ago

Search before asking

Ultralytics YOLO Component

Other

Bug

I have this model:

%%writefile yolov6-face.yaml
# Ultralytics YOLO πŸš€, AGPL-3.0 license
# YOLOv6 object detection model with P3-P5 outputs. For Usage examples see https://docs.ultralytics.com/models/yolov6

# Parameters
nc: 1 # number of classes
activation: nn.ReLU() # (optional) model default activation function
scales: # model compound scaling constants, i.e. 'model=yolov6n.yaml' will call yolov8.yaml with scale 'n'
  # [depth, width, max_channels]
  p: [0.33, 0.25, 8] # nano is [0.33, 0.25, 1024]

# YOLOv6-3.0s backbone
backbone:
  # [from, repeats, module, args]
  - [-1, 1, Conv, [64, 3, 2]] # 0-P1/2
  - [-1, 1, Conv, [128, 3, 2]] # 1-P2/4
  - [-1, 6, Conv, [128, 3, 1]]
  - [-1, 1, Conv, [256, 3, 2]] # 3-P3/8
  - [-1, 12, Conv, [256, 3, 1]]
  - [-1, 1, Conv, [512, 3, 2]] # 5-P4/16
  - [-1, 18, Conv, [512, 3, 1]]
  - [-1, 1, Conv, [1024, 3, 2]] # 7-P5/32
  - [-1, 6, Conv, [1024, 3, 1]]
  - [-1, 1, SPPF, [1024, 5]] # 9

# YOLOv6-3.0s head
head:
  - [-1, 1, Conv, [256, 1, 1]]
  - [-1, 1, nn.ConvTranspose2d, [256, 2, 2, 0]]
  - [[-1, 6], 1, Concat, [1]] # cat backbone P4
  - [-1, 1, Conv, [256, 3, 1]]
  - [-1, 9, Conv, [256, 3, 1]] # 14

  - [-1, 1, Conv, [128, 1, 1]]
  - [-1, 1, nn.ConvTranspose2d, [128, 2, 2, 0]]
  - [[-1, 4], 1, Concat, [1]] # cat backbone P3
  - [-1, 1, Conv, [128, 3, 1]]
  - [-1, 9, Conv, [128, 3, 1]] # 19

  - [[14, 19], 1, Detect, [nc]] # Detect(P3, P4, P5)

That I train with this code:

model = YOLO("./yolov6-face.yaml")

r = model.train(data="face-detection-dataset.yaml", epochs=1, imgsz='192,320', single_cls=True, plots=True, batch=500)

from ultralytics.utils.benchmarks import benchmark
model.benchmark(data="face-detection-dataset.yaml", imgsz='192,320', device="cpu")

But running the benchmark crashes with:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[7], line 2
      1 from ultralytics.utils.benchmarks import benchmark
----> 2 model.benchmark(data="face-detection-dataset.yaml", imgsz='192,320', device="cpu")

File /opt/conda/lib/python3.10/site-packages/ultralytics/engine/model.py:682, in Model.benchmark(self, **kwargs)
    680 custom = {"verbose": False}  # method defaults
    681 args = {**DEFAULT_CFG_DICT, **self.model.args, **custom, **kwargs, "mode": "benchmark"}
--> 682 return benchmark(
    683     model=self,
    684     data=kwargs.get("data"),  # if no 'data' argument passed set data=None for default datasets
    685     imgsz=args["imgsz"],
    686     half=args["half"],
    687     int8=args["int8"],
    688     device=args["device"],
    689     verbose=kwargs.get("verbose"),
    690 )

File /opt/conda/lib/python3.10/site-packages/ultralytics/utils/benchmarks.py:172, in benchmark(model, data, imgsz, half, int8, device, verbose, eps)
    169 check_yolo(device=device)  # print system info
    170 df = pd.DataFrame(y, columns=["Format", "Status❔", "Size (MB)", key, "Inference time (ms/im)", "FPS"])
--> 172 name = Path(model.ckpt_path).name
    173 s = f"\nBenchmarks complete for {name} on {data} at imgsz={imgsz} ({time.time() - t0:.2f}s)\n{df}\n"
    174 LOGGER.info(s)

File /opt/conda/lib/python3.10/pathlib.py:960, in Path.__new__(cls, *args, **kwargs)
    958 if cls is Path:
    959     cls = WindowsPath if os.name == 'nt' else PosixPath
--> 960 self = cls._from_parts(args)
    961 if not self._flavour.is_supported:
    962     raise NotImplementedError("cannot instantiate %r on your system"
    963                               % (cls.__name__,))

File /opt/conda/lib/python3.10/pathlib.py:594, in PurePath._from_parts(cls, args)
    589 @classmethod
    590 def _from_parts(cls, args):
    591     # We need to call _parse_args on the instance, so as to get the
    592     # right flavour.
    593     self = object.__new__(cls)
--> 594     drv, root, parts = self._parse_args(args)
    595     self._drv = drv
    596     self._root = root

File /opt/conda/lib/python3.10/pathlib.py:578, in PurePath._parse_args(cls, args)
    576     parts += a._parts
    577 else:
--> 578     a = os.fspath(a)
    579     if isinstance(a, str):
    580         # Force-cast str subclasses to str (issue #21127)
    581         parts.append(str(a))

TypeError: expected str, bytes or os.PathLike object, not NoneType

Running with:

from ultralytics.utils.benchmarks import benchmark
benchmark(model="/kaggle/working/runs/detect/train/weights/best.pt", data="face-detection-dataset.yaml", imgsz='192,320', device="cpu")

Mostly returns NaN even though I can actually export to Tensorflow Lite and NCNN (and probably others):

Setup complete βœ… (4 CPUs, 31.4 GB RAM, 6100.3/8062.4 GB disk)

Benchmarks complete for best.pt on face-detection-dataset.yaml at imgsz=192,320 (536.99s)
                   Format Status❔  Size (MB)  metrics/mAP50-95(B)  Inference time (ms/im)     FPS
0                 PyTorch       βœ…        0.3               0.2027                   16.46   60.73
1             TorchScript       ❎        0.7                  NaN                     NaN     NaN
2                    ONNX       ❎        0.5                  NaN                     NaN     NaN
3                OpenVINO       ❎        0.6                  NaN                     NaN     NaN
4                TensorRT       ❌        0.0                  NaN                     NaN     NaN
5                  CoreML       ❎        0.3                  NaN                     NaN     NaN
6   TensorFlow SavedModel       ❎        1.4                  NaN                     NaN     NaN
7     TensorFlow GraphDef       ❎        0.5                  NaN                     NaN     NaN
8         TensorFlow Lite       ❎        0.5                  NaN                     NaN     NaN
9     TensorFlow Edge TPU       ❎        0.3                  NaN                     NaN     NaN
10          TensorFlow.js       ❎        0.5                  NaN                     NaN     NaN
11           PaddlePaddle       ❎        1.0                  NaN                     NaN     NaN
12                    MNN       ❎        0.5                  NaN                     NaN     NaN
13                   NCNN       βœ…        0.5               0.0003                    5.97  167.53
14                    IMX       ❌        0.0                  NaN                     NaN     NaN
15                   RKNN       ❌        0.0                  NaN                     NaN     NaN

Environment

Ultralytics 8.3.67 πŸš€ Python-3.10.14 torch-2.4.0 CUDA:0 (Tesla P100-PCIE-16GB, 16269MiB)
Setup complete βœ… (4 CPUs, 31.4 GB RAM, 6095.9/8062.4 GB disk)

OS                  Linux-6.6.56+-x86_64-with-glibc2.35
Environment         Kaggle
Python              3.10.14
Install             pip
RAM                 31.35 GB
Disk                6095.9/8062.4 GB
CPU                 Intel Xeon 2.00GHz
CPU count           4
GPU                 Tesla P100-PCIE-16GB, 16269MiB
GPU count           1
CUDA                12.3

numpy               βœ… 1.26.4>=1.23.0
numpy               βœ… 1.26.4<2.0.0; sys_platform == "darwin"
matplotlib          βœ… 3.7.5>=3.3.0
opencv-python       βœ… 4.10.0.84>=4.6.0
pillow              βœ… 11.0.0>=7.1.2
pyyaml              βœ… 6.0.2>=5.3.1
requests            βœ… 2.32.3>=2.23.0
scipy               βœ… 1.14.1>=1.4.1
torch               βœ… 2.4.0>=1.8.0
torch               βœ… 2.4.0!=2.4.0,>=1.8.0; sys_platform == "win32"
torchvision         βœ… 0.19.0>=0.9.0
tqdm                βœ… 4.66.4>=4.64.0
psutil              βœ… 5.9.3
py-cpuinfo          βœ… 9.0.0
pandas              βœ… 2.2.3>=1.1.4
seaborn             βœ… 0.12.2>=0.11.0
ultralytics-thop    βœ… 2.0.14>=2.0.0

Minimal Reproducible Example

dataset:

%%writefile face-detection-dataset.yaml
# CC0: Public Domain license
# Face-Detection-Dataset dataset by Fares Elmenshawii
# Documentation: https://www.kaggle.com/datasets/fareselmenshawii/face-detection-dataset

# Train/val/test sets as 1) dir: path/to/imgs, 2) file: path/to/imgs.txt, or 3) list: [path/to/imgs1, path/to/imgs2, ..]
path: /kaggle/input/face-detection-dataset # dataset root dir
train: images/train # train images (relative to 'path')
val: images/val # val images (relative to 'path')
test: # test images (optional)

# Classes
names:
  0: face

# Download script/URL (optional)
download: https://storage.googleapis.com/kaggle-data-sets/3345370/5891144/bundle/archive.zip

model:

%%writefile yolov6-face.yaml
# Ultralytics YOLO πŸš€, AGPL-3.0 license
# YOLOv6 object detection model with P3-P5 outputs. For Usage examples see https://docs.ultralytics.com/models/yolov6

# Parameters
nc: 1 # number of classes
activation: nn.ReLU() # (optional) model default activation function
scales: # model compound scaling constants, i.e. 'model=yolov6n.yaml' will call yolov8.yaml with scale 'n'
  # [depth, width, max_channels]
  p: [0.33, 0.25, 8] # nano is [0.33, 0.25, 1024]

# YOLOv6-3.0s backbone
backbone:
  # [from, repeats, module, args]
  - [-1, 1, Conv, [64, 3, 2]] # 0-P1/2
  - [-1, 1, Conv, [128, 3, 2]] # 1-P2/4
  - [-1, 6, Conv, [128, 3, 1]]
  - [-1, 1, Conv, [256, 3, 2]] # 3-P3/8
  - [-1, 12, Conv, [256, 3, 1]]
  - [-1, 1, Conv, [512, 3, 2]] # 5-P4/16
  - [-1, 18, Conv, [512, 3, 1]]
  - [-1, 1, Conv, [1024, 3, 2]] # 7-P5/32
  - [-1, 6, Conv, [1024, 3, 1]]
  - [-1, 1, SPPF, [1024, 5]] # 9

# YOLOv6-3.0s head
head:
  - [-1, 1, Conv, [256, 1, 1]]
  - [-1, 1, nn.ConvTranspose2d, [256, 2, 2, 0]]
  - [[-1, 6], 1, Concat, [1]] # cat backbone P4
  - [-1, 1, Conv, [256, 3, 1]]
  - [-1, 9, Conv, [256, 3, 1]] # 14

  - [-1, 1, Conv, [128, 1, 1]]
  - [-1, 1, nn.ConvTranspose2d, [128, 2, 2, 0]]
  - [[-1, 4], 1, Concat, [1]] # cat backbone P3
  - [-1, 1, Conv, [128, 3, 1]]
  - [-1, 9, Conv, [128, 3, 1]] # 19

  - [[14, 19], 1, Detect, [nc]] # Detect(P3, P4, P5)

code:

model = YOLO("./yolov6-face.yaml")

r = model.train(data="face-detection-dataset.yaml", epochs=1, imgsz='192,320', single_cls=True, plots=True, batch=500)

from ultralytics.utils.benchmarks import benchmark
model.benchmark(data="face-detection-dataset.yaml", imgsz='192,320', device="cpu")

Additional

Exporting works just fine.

Are you willing to submit a PR?

UltralyticsAssistant commented 3 weeks ago

πŸ‘‹ Hello @EmmanuelMess, thank you for your interest in Ultralytics πŸš€! We appreciate your detailed report and enthusiasm for using YOLO models. It looks like you're encountering issues while benchmarking your customized YOLOv6 model. Let's work together to address this!

We recommend starting with the following steps:

If this issue is a πŸ› Bug, it would be extremely helpful if you could provide a minimum reproducible example to help us debug it more effectively. You've already shared many details (thank you!) but please confirm if these are sufficient to reproduce the error as-is.

If this is a training or benchmarking-related ❓ Question, consider reviewing our Training Tips or join the community for further assistance.

Community Support ❀️

Meanwhile, feel free to join the Ultralytics community where you can engage with other users while awaiting an engineer's input:

Status Information

Ultralytics CI
This badge indicates the status of our Continuous Integration (CI) tests. A green badge ensures all tests are passing for verified operations across platforms.

An Ultralytics engineer will review this issue and assist you shortly. Thank you for your patience! πŸš€

Y-T-G commented 3 weeks ago

You should load the best.pt or last.pt after training.

EmmanuelMess commented 3 weeks ago

You should load the best.pt or last.pt after training.

I did, and I get a lot of NaN results, even if the exports work. I mention it in the issue.

Y-T-G commented 3 weeks ago

If export succeeds and val fails, that's when you would see NaN and ❎. As opposed to ❌ which indicates export failed.

You should check the logs for errors.

UltralyticsAssistant commented 3 weeks ago

Thanks for raising this issue! A potential fix has been merged through PR #18894 by @Y-T-G with contributions from @glenn-jocher. πŸŽ‰

Key Changes Introduced in the PR:

This PR focuses on improving model file handling to enhance clarity and flexibility during benchmarking. The changes include:

To test whether the changes in this PR resolve the issue, you can:

Please give the updates a try and share feedback on whether they address the issue for you. Your input is invaluable in ensuring the fix works as intended.

Thanks again for reporting the issue and helping us improve! πŸ™ If the issue persists, don't hesitate to let us know, and we'll continue investigating. πŸš€

EmmanuelMess commented 3 weeks ago

If export succeeds and val fails, that's when you would see NaN and ❎. As opposed to ❌ which indicates export failed.

You should check the logs for errors.

Thanks! I was wondering what that meant. Will post the logs to another issue.

EmmanuelMess commented 3 weeks ago

Opened #18904

glenn-jocher commented 2 weeks ago

Thank you for opening #18904! We'll review it and get back to you with updates or further assistance.