UltralyticsAssistant
commented
2 weeks ago 👋 Hello @eVen-gits, thank you for your detailed report regarding the crash with a large amount of background images. We're excited to help you out! 🚀
For bug reports, it's very helpful if you can provide a minimum reproducible example. This will allow us to more effectively diagnose and resolve the issue.
To get started, please ensure you're using the latest version of the ultralytics package and all its requirements in a Python environment of version 3.8 or higher with PyTorch version 1.8 or higher:
pip install -U ultralyticsIn case you are facing dataset-related issues with slicing or format transformations, please ensure your data preprocessing steps and outputs, like annotations, adhere closely to the expected formats after every stage. This can be key when errors appear unexpectedly during model training.
You can also try running your training in one of our recommended environments to verify it’s not an issue with your local setup. Here are some options with all dependencies preinstalled:
- Notebooks with free GPU:
Feel free to continue exploring the fantastic resources in our Docs, including specific guidance that might relate to your issue.
And remember, the Ultralytics community is here to support you! For real-time interaction, join us on Discord 🎧 or share your experience with others on Reddit.
This is an automated response, but an Ultralytics engineer will also look into your issue shortly. Meanwhile, additional insights you provide may speed up the process. Thanks again for reaching out! 😊
Y-T-G
eVen-gits
glenn-jocher
Search before asking
Ultralytics YOLO Component
No response
Bug
I am working with large images, and I am using sahi to slice these images into tiles. This produces a COCO style dataset.
This dataset is then transformed to a
yolodataset, usingsahi coco yolov5command. This produces a yolo style dataset that still needs preprocessing/fixes. Here is the general procedure:sahi yolo slicesahi coco yolov5data.ymltodata.yamland fix pathsThe crash appears to be happening here:
Longer output (some repeated CUDA spam ommited)
``` >>> yolo train \ batch=64 device=0,1 imgsz=640 epochs=100 patience=100 \ model=yolo11l \ data=More information on above procedures:
1) Slice the dataset
Using command
2) Transform the dataset format from COCO to YOLO
3) Split the data in train/test/val
4) Fix yolo annotations
5) Move files into corresponding folders
This can be achieved with simple bash commands:
6) Update and rename
data.ymltodata.yamland fix pathsFile contents should be somehting like this:
The final structure should look something like this: ├── test │ ├── images │ └── labels ├── train │ ├── images │ └── labels ├── val │ ├── images │ └── labels └── data.yaml
7) Attempt to train
This will produce a crash outlined above. Within the original sliced dataset, there are roughly this many images: train: 76000 images, 69060 backgrounds, 0 corrupt: 100%|██████████| 76000/76000 val: 9500 images, 8642 backgrounds, 0 corrupt: 100%|██████████| 9500/9500
Now here is an interesting part:
If I add an additional step in preprocessing, removing all background images from sliced dataset, the training will work fine. This can be done using
cocojsonlibrary, like so:python3 -m cocojson.run.remove_empty <annotations_json> --out <output_dataset_json_path>This will produce a significantly smaller subset, but it will also not contain any background images. I do not believe this to be conceptually correct, as you also want negative samples in training as to not have positive sample bias, right?
The resulting dataset has this size: train: 6915 images, 0 backgrounds, 0 corrupt: 100%|██████████| 6915/6915 val: 864 images, 0 backgrounds, 0 corrupt: 100%|██████████| 864/864
Please, advise what can be done. Is this an issue with the size of the dataset, or the ratio of positive/negative samples, or is it something entirely different.
Environment
Minimal Reproducible Example
Please, see code snippets above. It might be difficult to provide sample dataset, due to the size.
Additional
No response
Are you willing to submit a PR?