Poor results during inference

[FasterthanLi]

Search before asking

• [X] I have searched the YOLOv8 issues and discussions and found no similar questions.

Question

Hello Ultralytics Team,

I am experiencing an issue with the inference phase of my YOLOv8m model, which I fine-tuned using the German Traffic Sign Recognition Benchmark dataset. The model exhibits satisfactory performance during both training and validation phases, accurately recognizing traffic signs. However, when I use the model to perform inference on my own dataset, which consists of images not included in the German Traffic Sign Recognition Benchmark, the results significantly deteriorate.

Steps to Reproduce:

Fine-tune YOLOv8m using the German Traffic Sign Recognition Benchmark. Validate the model using the benchmark's validation set—note good performance. Run inference using a separate set of images collected independently (not from the benchmark dataset). Expected Behavior: The model should maintain a reasonable level of accuracy and performance when applied to similar images not included in the training dataset, especially since the images pertain to the same domain (traffic signs).

Actual Behavior: The model's performance is poor when running inference on the new images, indicating a significant drop in detection accuracy compared to the benchmark dataset.

Request: Could you please provide insights or recommendations on how to improve the model's robustness and performance on unseen data that is stylistically and contextually similar to the training data? Any suggestions for adjusting the training parameters or preprocessing steps would also be appreciated.

Thank you for your support and looking forward to your suggestions.

Below is the code snippet from inference.py that handles the inference process: `# Nessearay imports import torch from ultralytics import YOLO import cv2 import os from torchvision import transforms

Device setup

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

Load model

model = YOLO('path/to/best.pt').to(device)

Directory containing the images

dir_image_path = "path/to/images"

Directory to save the results

save_dir = "path/to/save"

Ensure the save directory exists

os.makedirs(save_dir, exist_ok=True)

Function to preprocess each image using OpenCV

def preprocess_image(image_path):

Read image

image = cv2.imread(image_path)
if image is None:
    return None  # Skip files that can't be opened

# Convert image to RGB format
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

# Convert image to float and scale to [0, 1]
image = image.astype('float32') / 255.0

# Convert the image to a tensor
image = torch.from_numpy(image).permute(2, 0, 1)

# Normalize the image using defined mean and std
normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])

# Normalize the image
image = normalize(image)

# Clamp the image to fit the model's expected input range
image = torch.clamp(image, 0, 1)

# Add batch dimension
return image.unsqueeze(0)

Process each image in the directory and predict using the model

for filename in os.listdir(dir_image_path): if not filename.lower().endswith(('.png', '.jpg', '.jpeg')):

Skip non-image files

    continue 

# Load the image
file_path = os.path.join(dir_image_path, filename)

# Preprocess the image
preprocessed_image = preprocess_image(file_path)
if preprocessed_image is None:
    # Skip files that can't be opened
    print(f"Skipping file {filename} as it could not be loaded.")

    # Skip files that can't be opened
    continue 

# Predict using the model
results = model(preprocessed_image, imgsz=640, stream=False)

# Process the results to extract bounding boxes and probabilities
for result in results:

    # Boxes object for bounding box outputs
    boxes = result.boxes

    # Probs object for classification outputs  
    probs = result.probs  

    # Save the annotated image with bounding boxes and labels
    save_path = os.path.join(save_dir, filename.replace('.jpg', '_result.jpg').replace('.jpeg', '_result.jpeg').replace('.png', '_result.png'))

    # Save annotated results to disk
    result.save(filename=save_path) 

    # Print the save path
    print(f"Result saved for image {filename} at {save_path}")

Print the results

print("Inference completed successfully.")`

Additional

No response

[github-actions[bot]]

👋 Hello @FasterthanLi, thank you for your interest in Ultralytics YOLOv8 🚀! We recommend a visit to the Docs for new users where you can find many Python and CLI usage examples and where many of the most common questions may already be answered.

If this is a 🐛 Bug Report, please provide a minimum reproducible example to help us debug it.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset image examples and training logs, and verify you are following our Tips for Best Training Results.

Join the vibrant Ultralytics Discord 🎧 community for real-time conversations and collaborations. This platform offers a perfect space to inquire, showcase your work, and connect with fellow Ultralytics users.

Install

Pip install the ultralytics package including all requirements in a Python>=3.8 environment with PyTorch>=1.8.

pip install ultralytics

Environments

YOLOv8 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all Ultralytics CI tests are currently passing. CI tests verify correct operation of all YOLOv8 Modes and Tasks on macOS, Windows, and Ubuntu every 24 hours and on every commit.

[glenn-jocher]

@FasterthanLi hey there! 👋 It sounds like you're encountering a classic case of model generalization issues, which isn't uncommon when models are trained and then used on data that slightly differs in distribution from the training set. Here are a few suggestions that might help improve your model’s performance on unseen data:

1. Data Augmentation: Consider augmenting your training data with transformations that match variations in your new dataset. This can include geometric transformations, color space adjustments, and more. YOLOv8 offers easy options for augmentation that you can leverage during training.

2. Fine-tuning on New Data: If possible, label a subset of your new dataset and include it in your training process. This can help the model adapt better to the new data distribution.

3. Model Calibration: Sometimes, the confidence thresholds tuned on the validation set might not translate well to new data. You might want to experiment with adjusting the confidence threshold (conf) during inference to see if it helps in improving results.

4. Post-processing Adjustments: Tweaking non-maximum suppression (NMS) settings or other post-processing steps might help in reducing false positives/negatives on the new data.

Your preprocessing pipeline looks solid, but ensure that the augmentation and normalization steps are consistent with what the model was trained on.

For more detailed advice or further assistance, our community might have more insights to offer, so exploring similar questions in our discussions can be beneficial. Good luck, and we're here to support you on this journey! 🚀

[FasterthanLi]

@glenn-jocher Thank you for the answer, I definitely will incorporate the suggestions you listed, however, I have a question should I apply normalization for images before training? I thought YOLOv8 does it automatically. Here is the link to the post https://github.com/ultralytics/ultralytics/issues/6979#issuecomment-1854540453.

[glenn-jocher]

Hey there! 🌟 You're right, YOLOv8 automatically handles normalization as part of its preprocessing steps, so there's no need for you to apply normalization before training your images. This is managed internally to ensure consistency across different datasets and to make training more straightforward for you. If you're implementing custom preprocessing, remember to keep it aligned with YOLOv8 standards for best results. Happy training! 😊

[FasterthanLi]

Thank you for the answer!

[glenn-jocher]

@FasterthanLi you're welcome! If you have any more questions or run into issues, feel free to reach out. Happy coding! 😊