Add/Modify YOLOv8 backbone structure

[chloewxrn]

I am trying to modify the YOLOv8 backbone by adding the attention module inside. However, I keep getting key error. I modified the .yaml file, modules.py and tasks.py but still not working. Anyone can help to solve this issue?

[github-actions[bot]]

👋 Hello @chloewxrn, thank you for your interest in YOLOv8 🚀! We recommend a visit to the YOLOv8 Docs for new users where you can find many Python and CLI usage examples and where many of the most common questions may already be answered.

If this is a 🐛 Bug Report, please provide a minimum reproducible example to help us debug it.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset image examples and training logs, and verify you are following our Tips for Best Training Results.

Install

Pip install the ultralytics package including all requirements in a Python>=3.7 environment with PyTorch>=1.7.

pip install ultralytics

Environments

YOLOv8 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all Ultralytics CI tests are currently passing. CI tests verify correct operation of all YOLOv8 Modes and Tasks on macOS, Windows, and Ubuntu every 24 hours and on every commit.

[glenn-jocher]

@chloewxrn we don't provide support for code customization as we're very busy maintaining and developing, but if you post your model YAML and a reproducible example of the problem that would make it easier for a community member to engage with you.

[github-actions[bot]]

👋 Hello there! We wanted to give you a friendly reminder that this issue has not had any recent activity and may be closed soon, but don't worry - you can always reopen it if needed. If you still have any questions or concerns, please feel free to let us know how we can help.

For additional resources and information, please see the links below:

• Docs: https://docs.ultralytics.com
• HUB: https://hub.ultralytics.com
• Community: https://community.ultralytics.com

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLO 🚀 and Vision AI ⭐

[JinsJinsAgain]

Which kinds of files you modified..?

I already modified YOLOv8 network and do training now.

[glenn-jocher]

@JinsJinsAgain thank you for reaching out and for your interest in YOLOv8! It's great to hear that you have already modified the YOLOv8 network and are now training it.

In order to better understand your situation and provide effective assistance, could you please specify which files you have modified in YOLOv8? This will help us to provide you with the necessary guidance and support.

We look forward to hearing from you soon!

[JinsJinsAgain]

1.yolov8/ultralytics/models/v8/yolov8.yaml 2.yolov8/ultralytics/nn/tasks.py 3.yolov8/ultralytics/nn/modules/conv.py 4.yolov8/ultralytics/nn/modules/block.py 5.yolov8/ultralytics/nn/modules/init.py

I modified these 5 files, This is the method how to modify yolov8 architecture in my situation.

1. If we make new classes in conv.py and block.py files.
2. then you have to change init.py and tasks.py files to import classes made by yourself.
3. Finally, you can add your new layer to the yolov8.yaml file.

[glenn-jocher]

@JinsJinsAgain thank you for providing the information about the files you have modified in YOLOv8! It seems like you have made changes to the following files:

1. yolov8/ultralytics/models/v8/yolov8.yaml
2. yolov8/ultralytics/nn/tasks.py
3. yolov8/ultralytics/nn/modules/conv.py
4. yolov8/ultralytics/nn/modules/block.py
5. yolov8/ultralytics/nn/modules/__init__.py

To modify the YOLOv8 architecture with your changes, you can follow these steps:

1. Create new classes in the conv.py and block.py files to define the desired layers or modules.
2. Update the __init__.py and tasks.py files to import the newly created classes from conv.py and block.py.
3. Finally, add the new layer or module to the yolov8.yaml file to include it in the YOLOv8 architecture.

By following these steps, you should be able to customize the YOLOv8 architecture according to your requirements.

Please note that this is a high-level explanation, and the exact implementation details may depend on your specific modifications. Make sure to review and test your changes thoroughly to ensure the desired functionality.

If you have any further questions or need additional assistance, please feel free to ask. We're here to help!

[github-actions[bot]]

👋 Hello there! We wanted to give you a friendly reminder that this issue has not had any recent activity and may be closed soon, but don't worry - you can always reopen it if needed. If you still have any questions or concerns, please feel free to let us know how we can help.

For additional resources and information, please see the links below:

• Docs: https://docs.ultralytics.com
• HUB: https://hub.ultralytics.com
• Community: https://community.ultralytics.com

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLO 🚀 and Vision AI ⭐

[glenn-jocher]

Hi @AT9991,

To train your modified YOLOv8 architecture, you need to follow these steps:

1. After modifying the necessary files to define your new model architecture, you should have a new YAML file that specifies your new architecture.

2. Ensure that your training data is properly structured. It should be in the format expected by YOLOv8. Usually this involves having annotated images in a specific directory structure.

3. Use the YOLOv8 training routine, but do so with the YAML file that specifies your custom model. This generally involves a command at the command-line where you specify your custom YAML file and possibly other parameters, such as batch size and number of epochs.

4. Monitor the training process. The training routine will print out loss statistics and should save model checkpoints at regular intervals, so you can halt training and resume from a checkpoint if necessary.

Remember, the devil is in the details. There could be many reasons why training isn't working, and it's hard for us to debug without additional information.

If you follow these general steps and are still running into issues, feel free to provide more specific information about the problems you're encountering so we can better assist you. Thank you!

[Rehamelshiekh]

@glenn-jocher i want to modify the architecture of yolov8 but i don' know from where should i start i'm new in the yolo. specially i want to modify the backbone arcitecture. can youplease help me. and how to modify files in my colab.

[MohammadSadati]

I can't find yolov8/ultralytics/models/v8/yolov8.yaml I think you change the location to yolov8/ultralytics/cfg/models/v8/yolov8.yaml am I right? if I want to change the backbone of yolov8, I should change the lines under backbone:?

[glenn-jocher]

Hi @MohammadSadati,

You are correct that the structure of the repository has evolved, and some files may have moved from their original locations. The .yaml architecture file for the YOLOv8 model is indeed more likely to be found at yolov8/ultralytics/cfg/models/v8/yolov8.yaml.

If your modification aims specifically at the backbone architecture of YOLOv8, then yes, you'll need to adjust the backbone: section in the YOLOv8 .yaml file. Please ensure to correctly define your desired architecture in accordance with the Ultralytics build sequence rules and keep in mind that changes in the backbone might need corresponding changes in the head or neck sections as well.

When you make changes to the yolov8.yaml file and use it to train your model, this new configuration will define the architecture of the model to be trained during your experiment. Never hesitate to ask here if you encounter some difficulties during your journey!

[MohammadSadati]

hi @glenn-jocher as I said, I want to change the backbone of yolov8 to resnet50

In the first step I added these two classes to ultralytics/nn/modules/conv.py

class ResnetStart(nn.Module):
    def __init__(self, in_channels = 3, out_channels = 64 , stride=2):
        self.conv1 = nn.Conv2d(in_channels, out_channels, kernel_size=7, stride=2, padding=3)
        self.batch_norm1 = nn.BatchNorm2d(out_channels)
        self.relu = nn.ReLU()
        self.max_pool = nn.MaxPool2d(kernel_size = 3, stride=2, padding=1)

    def forward(self, x):
        x = self.relu(self.batch_norm1(self.conv1(x)))
        x = self.max_pool(x)
        return x 

class ResnetConv(nn.Module):
    def __init__(self, in_channels, out_channels , stride=1, e = 4):

        self.conv1 = nn.Conv2d(in_channels, out_channels, kernel_size=1, stride=1, padding=0)
        self.batch_norm1 = nn.BatchNorm2d(out_channels)

        self.conv2 = nn.Conv2d(out_channels, out_channels, kernel_size=3, stride=stride, padding=1)
        self.batch_norm2 = nn.BatchNorm2d(out_channels)

        self.conv3 = nn.Conv2d(out_channels, out_channels*self.expansion, kernel_size=1, stride=1, padding=0)
        self.batch_norm3 = nn.BatchNorm2d(out_channels*self.expansion)

        self.relu = nn.ReLU()

        def forward(self, x):
            x = self.relu(self.batch_norm1(self.conv1(x)))
            x = self.relu(self.batch_norm2(self.conv2(x)))
            x = self.conv3(x)
            x = self.batch_norm3(x)
            return x

then I add 'ResnetStart' , 'ResnetConv' to _ all _ list and ultralytics/nn/modules/init.py file .

in second step I change the backbone: in ultralytics/cfg/models/v8/yolov8.yaml file to :

- [-1, 1, 'ResnetStart', [3, 64, 2]]  # 0
  - [[-1, 0], 1, 'ResnetConv', [3, 64, 1]]   # 1 
  - [[-1, 1], 1, 'ResnetConv', [3, 64, 1]]   # 2
  - [[-1, 2], 1, 'ResnetConv', [3, 64, 1]]   # 3
  - [[-1, 3], 1, 'ResnetConv', [3, 128, 2]]  # 4
  - [[-1, 4], 1, 'ResnetConv', [3, 128, 1]]  # 5
  - [[-1, 5], 1, 'ResnetConv', [3, 128, 1]]  # 6
  - [[-1, 6], 1, 'ResnetConv', [3, 128, 1]]  # 7
  - [[-1, 7], 1, 'ResnetConv', [3, 256, 2]]  # 8
  - [[-1, 8], 1, 'ResnetConv', [3, 256, 1]]  # 9 
  - [[-1, 9], 1, 'ResnetConv', [3, 256, 1]]  # 10
  - [[-1, 10], 1, 'ResnetConv', [3, 256, 1]]  # 11
  - [[-1, 11], 1, 'ResnetConv', [3, 256, 2]]  # 12
  - [[-1, 12], 1, 'ResnetConv', [3, 256, 1]]  # 13
  - [[-1, 13], 1, 'ResnetConv', [3, 512, 1]]  # 14
  - [[-1, 14], 1, 'ResnetConv', [3, 512, 2]]  # 15
  - [[-1, 15], 1, 'ResnetConv', [3, 512, 1]]  # 16

I know I should change the head too, but I want to know I am in the right way or not?

thanks in advance.

[glenn-jocher]

Hello @MohammadSadati,

Based on your description, it seems like you are on the right path to modifying the base architecture of YOLOv8 to utilize ResNet50.

1. First, you have correctly added new module classes in the correct conv.py file. Ensure that the functions are correctly implemented and correspond to the structures you need in ResNet50.
2. You have correctly imported the new classes you created in the init.py file. This is the right place to import the new classes so that they can be recognized in the architecture definition yaml.
3. Finally, you have correctly started modifying the .yaml file to reflect the Resnet50 structure. It's crucial to ensure all the modules are referenced correctly with parameters as you have done it.

Please remember to update the Head of the architecture in your YAML file as the Head needs to align with your new backbone structure. Also, ensure the parameters in the Yaml file are correct, such as in_channels and out_channels for each layer, matching the Resnet50 structure.

Refactor and test each module independently to ensure they work as expected, and incrementally build up your architecture to help manage complexity and diagnose any issues.

Good luck with your experiment!

[saikrishna0203]

pls tell me steps to change backbone architecture of yolov8 to resnet model till training ,please

[saikrishna0203]

@MohammadSadati bro pls tell me steps from starting to end till training what to do each step to change yolo8 architecture

[jsjoel]

WARNING ⚠️ no model scale passed. Assuming scale='n'. Traceback (most recent call last): File "/usr/local/bin/yolo", line 8, in sys.exit(entrypoint()) File "/usr/local/lib/python3.10/dist-packages/ultralytics/cfg/init.py", line 541, in entrypoint model = YOLO(model, task=task) File "/usr/local/lib/python3.10/dist-packages/ultralytics/engine/model.py", line 132, in init self._new(model, task=task, verbose=verbose) File "/usr/local/lib/python3.10/dist-packages/ultralytics/engine/model.py", line 197, in _new self.model = (model or self._smart_load("model"))(cfg_dict, verbose=verbose and RANK == -1) # build model File "/usr/local/lib/python3.10/dist-packages/ultralytics/nn/tasks.py", line 278, in init self.model, self.save = parse_model(deepcopy(self.yaml), ch=ch, verbose=verbose) # model, savelist File "/usr/local/lib/python3.10/dist-packages/ultralytics/nn/tasks.py", line 759, in parse_model m = getattr(torch.nn, m[3:]) if "nn." in m else globals()[m] # get module KeyError: 'ODConv_3rd'

after doing the four steps. This error is raised

[glenn-jocher]

@jsjoel to add an attention module to the YOLOv8 backbone, you'll need to ensure that your modifications are consistent across the model configuration file (.yaml), the model definition (modules.py), and any other relevant files. Here are some steps to guide you:

1. Model Configuration File (.yaml):

   • Define your attention module within the .yaml file. Make sure to use the correct indentation and syntax.
   • Add the attention module to the backbone at the appropriate location within the architecture.

2. Model Definition (modules.py):

   • Implement the attention module class if it's not already available in the codebase.
   • Ensure that the module's forward method is correctly defined and that it interfaces properly with the rest of the network.

3. Tasks File (tasks.py):

   • If you've made changes that affect the overall model structure, ensure that these are reflected in any tasks or functions that build or manipulate the model.

4. Key Error:

   • A key error typically indicates a mismatch between the keys used in the state dictionary of the model and the keys expected during model construction.
   • Double-check the names of the parameters and layers in your attention module and ensure they match the keys in the .yaml file and the model's state dictionary.

5. Debugging:

   • Use print statements or a debugger to inspect the state dictionary keys and the expected keys at the point where the error occurs.
   • Verify that the attention module is correctly instantiated and integrated into the model.

6. Testing:

   • Write a simple test case to instantiate the model with the attention module and pass a dummy input through it to ensure everything is working as expected.

If you continue to face issues, consider providing more specific details such as the exact error message, the code snippet where the error occurs, and the relevant sections of the .yaml file and modules.py. This will help the community to better understand the problem and provide more targeted assistance.

[AjanthaAravind]

@glenn-jocher After changing the backbone with the attention mechanism, how to run the .yaml file in colab. is it enough to upload the .yaml file only to colab? how the changes reflect in that file?

[glenn-jocher]

@AjanthaAravind to modify the YOLOv8 backbone structure and add an attention module, you need to ensure that your changes are consistent across the model configuration file (.yaml), the model definition code (typically in models.py or modules.py), and potentially the training script (tasks.py or similar).

Here's a step-by-step guide to help you integrate an attention module into the YOLOv8 backbone:

1. Update the .yaml Configuration File:

   • Open the .yaml file that defines the architecture of your YOLOv8 model.
   • Locate the backbone section and add a new entry for your attention module. Make sure to give it a unique name that does not conflict with existing keys.
   • Define the parameters of your attention module within this entry.

2. Modify the Model Definition Code:

   • In models.py or modules.py, import the attention module you wish to use. If it's a custom module, ensure that it's correctly implemented and accessible.
   • Add code to instantiate the attention module within the backbone class. This typically involves adding it to the __init__ method and ensuring it's called in the forward pass.
   • Ensure that the input and output dimensions of the attention module match the surrounding layers in the backbone.

3. Adjust the Training Script:

   • In tasks.py or the equivalent training script, make sure that the model is being loaded with the updated .yaml configuration.
   • If you've added new hyperparameters or settings specific to the attention module, ensure they are being parsed and passed to the model correctly.

4. Troubleshoot Key Errors:

   • A key error usually indicates a mismatch between the keys used in the .yaml file and the keys expected in the model definition code.
   • Double-check that the names of the modules and their parameters in the .yaml file match exactly with those in the model definition.
   • If you're loading pretrained weights, ensure that the keys in the state dictionary match the names of the parameters in your modified model. If they don't, you may need to either adjust the keys or train the model from scratch.

5. Test the Model:

   • Before training, instantiate the model with the modified configuration and print the model summary to verify that the attention module is correctly integrated and that the dimensions align as expected.
   • Run a forward pass with dummy data to ensure there are no runtime errors.

6. Train the Model:

   • Once you've confirmed that the model is set up correctly, proceed with training. Monitor the training process to ensure that the attention module is improving the model's performance as intended.

If you continue to face issues, please provide the exact error message and the relevant sections of the code where the error occurs. This will help in diagnosing the problem more accurately. Remember that modifying deep learning models can be complex, and it often requires careful debugging to ensure all components are compatible.

[AjanthaAravind]

@glenn-jocher Thank you

[glenn-jocher]

@AjanthaAravind to modify the YOLOv8 backbone structure to include an attention module, you'll need to ensure that your changes are consistent across the model configuration file (.yaml), the model definition code (typically in models.py or modules.py), and potentially the training script (tasks.py or similar).

Here's a general approach to adding an attention module to the YOLOv8 backbone:

1. Update the YAML Configuration File:

   • Open the .yaml file that defines the model architecture.
   • Locate the backbone section and add a new entry for your attention module. Ensure that the indentation and format are consistent with the rest of the file.
   • Assign a unique name to the module and specify the parameters it requires.

2. Implement the Attention Module:

   • In modules.py (or wherever the model components are defined), implement your attention module as a new class or function.
   • Ensure that the module adheres to PyTorch's nn.Module interface if it's a class.
   • The module should take the necessary inputs and apply the attention mechanism, returning the modified feature maps.

3. Integrate the Attention Module in the Model:

   • In the same file where the YOLOv8 model is defined (could be models.py or modules.py), import your attention module.
   • Modify the backbone construction code to instantiate and integrate your attention module at the desired location within the backbone.

4. Modify the Training Script if Necessary:

   • If you've made changes that affect the training process (e.g., new hyperparameters, different input/output shapes), update tasks.py or the relevant training script accordingly.
   • Ensure that any new arguments or configurations are parsed and passed to the model correctly.

5. Debugging Key Errors:

   • Key errors typically occur when there's a mismatch between the keys used in the state dictionary of the model and the keys expected during various operations like loading weights or building the model.
   • Double-check the names you've used in the .yaml file and ensure they match the keys in your model's state dictionary.
   • If you're loading pretrained weights, ensure that the new attention module's weights are handled appropriately, as they won't be present in the original pretrained state dictionary.

6. Testing:

   • After making the changes, instantiate the model with the updated configuration and verify that it can perform a forward pass without errors.
   • It's a good practice to write unit tests for the new attention module to ensure it behaves as expected.

Here's an example of how you might define an attention module in the YAML file:

# Example of adding an attention module in the .yaml file
backbone:
  # ... existing layers ...
  - name: AttentionModule
    params:
      in_channels: 256
      out_channels: 256
      # ... other parameters for your attention module ...

And an example of implementing the attention module in modules.py:

import torch
import torch.nn as nn

class AttentionModule(nn.Module):
    def __init__(self, in_channels, out_channels, ...):
        super().__init__()
        # Define attention mechanism components here
        # ...

    def forward(self, x):
        # Apply attention mechanism and return the result
        # ...
        return x

# Then, in the model definition, instantiate and use the AttentionModule

Remember to carefully review the error messages you're getting, as they often provide clues about what's causing the issue. If you're still encountering problems, consider sharing the specific error message and the relevant code snippets for more targeted assistance.

[SauirbekA]

Hello guys, if you are adding a new class for yolov8 architecture please don't forget to add them on those 4 places

1) ultralytics/nn/tasks.py, def parse_model, line 807

2) ultralytics/nn/tasks.py, from ultralytics , line 10

3) ultralytics/nn/modules/init.py, from .block OR from .head OR from .conv, and __all__

4) ultralytics/nn/modules/conv.py, ultralytics/nn/modules/block.py, ultralytics/nn/modules/head.py

[glenn-jocher]

@SauirbekA hey there! 👋 Thanks for the heads-up on integrating new classes into the YOLOv8 architecture. You're spot on! When adding new classes, it's crucial to update those specific areas to ensure everything works seamlessly.

Just a quick example for clarity: if you've created a new FancyBlock class in block.py, you'd do something like this:

# In ultralytics/nn/modules/block.py
class FancyBlock(nn.Module):
    # Your implementation here

# In ultralytics/nn/modules/__init__.py
from .block import FancyBlock
__all__ = ['FancyBlock', ...]  # Add 'FancyBlock' to the list

And don't forget to import and use your new class in tasks.py as needed. If you run into any snags, feel free to drop a question here. Happy coding! 😊👨‍💻

[enazari]

Hi @glenn-jocher,

Thank you for your great work. I'm curious about modifying a pre-trained model's architecture. We know that we can configure the architecture using the parameters in the model's YAML file. When customizing these settings, it creates a new model with randomly initialized weights. My question is, it possible to create a scaled-down version of the existing nano model, with fewer parameters, while still leveraging the weights from pre-trained models? Thank you for considering my question.

[glenn-jocher]

@enazari hi there! 👋

Absolutely, you can scale down a pre-trained model and still leverage the weights. You'd modify the YAML to reduce the model's width and depth, then load the pre-trained weights with model.load_state_dict(torch.load(weights), strict=False). This will ignore non-matching keys, allowing you to use the weights that do fit your scaled-down architecture.

Keep in mind that layers with changed dimensions won't benefit from pre-trained weights and will be randomly initialized. Fine-tuning the modified model on your dataset is a good idea to regain performance lost due to the architecture change.

Happy coding! 😊🚀

[enazari]

Hi @glenn-jocher,

Thank you for your swift reply.

I have a question aboutmodel.load_state_dict(torch.load(weights), strict=False) ; how can I obtain the weights of a YOLOv8 model to pass to PyTorch?

In more detail:

#first I initialize a new model:
from ultralytics import YOLO
raw_model = YOLO('yolov8n-pose.yaml') 

#then I initialize a fine-tuned model:
fine_tuned_model = YOLO('yolov8n-pose.pt') 

#now, how do I access the weights of the fine-tuned model to apply to raw_model?
raw_model.load_state_dict(torch.load(fine_tuned_model.weights), strict=False) #fine_tuned_model.weights?

Thank you!

[glenn-jocher]

@enazari hi there! 👋

To load the weights from a fine-tuned YOLOv8 model, you'll want to point directly to the .pt file when using torch.load(). Here's how you can do it:

# Load the weights from the fine-tuned model's .pt file
weights = torch.load('path/to/yolov8n-pose.pt')

# Apply these weights to your raw model
raw_model.load_state_dict(weights, strict=False)

Just replace 'path/to/yolov8n-pose.pt' with the actual path to your fine-tuned model's weights file. This should do the trick! 😊

Happy coding! 🚀

[enazari]

Thanks once more @glenn-jocher!

[glenn-jocher]

@enazari you're welcome! To load weights into your model, just point to the .pt file like this:

# Load your raw model
raw_model = YOLO('yolov8n-pose.yaml')

# Load weights from the fine-tuned model
weights = torch.load('path/to/yolov8n-pose.pt')

# Apply weights to the raw model
raw_model.load_state_dict(weights['model'].state_dict(), strict=False)

Replace 'path/to/yolov8n-pose.pt' with your model's file path. This will transfer the weights to your new model configuration. Happy coding! 😊👨‍💻

[enazari]

Hi @glenn-jocher,

Thanks for your advice. I applied the method you suggested; I loaded the weights of a pre-trained YOLOv8-pose nano model into a randomly initialized model that has essentially the YOLOv8-pose nano structure. Subsequently, I trained this model, now equipped with pre-trained weights, using my dataset. Despite this approach, I was unable to achieve the same outcomes as when directly training a pre-trained model.

In more detail, I got the YOLOv8-pose nano structure yaml file from https://github.com/ultralytics/ultralytics/blob/main/ultralytics/cfg/models/v8/yolov8-pose.yaml.

First I initialized a raw model:

from ultralytics import YOLO

model = YOLO('yolov8-pose.yaml') 

Then I loaded model with the pre-trained yolovy8-pose:

import torch 

weights = torch.load('yolov8n-pose.pt')

#method1:
model.load_state_dict(weights, strict=False)

#method2:
# model.load_state_dict(weights['model'].state_dict(), strict=False)

Running either method1 or 2, I got the following message:

_IncompatibleKeys(missing_keys=['model.model.0.conv.weight', 'model.model.0.bn.weight', 'model.model.0.bn.bias', 'model.model.0.bn.running_mean', 'model.model.0.bn.running_var', 'model.model.1.conv.weight', 'model.model.1.bn.weight', 'model.model.1.bn.bias', 'model.model.1.bn.running_mean', 'model.model.1.bn.running_var', 'model.model.2.cv1.conv.weight', 'model.model.2.cv1.bn.weight', 'model.model.2.cv1.bn.bias', 'model.model.2.cv1.bn.running_mean', 'model.model.2.cv1.bn.running_var', 'model.model.2.cv2.conv.weight', 'model.model.2.cv2.bn.weight', 'model.model.2.cv2.bn.bias', 'model.model.2.cv2.bn.running_mean', 'model.model.2.cv2.bn.running_var', 'model.model.2.m.0.cv1.conv.weight', 'model.model.2.m.0.cv1.bn.weight', 'model.model.2.m.0.cv1.bn.bias', 'model.model.2.m.0.cv1.bn.running_mean', 'model.model.2.m.0.cv1.bn.running_var', 'model.model.2.m.0.cv2.conv.weight', 'model.model.2.m.0.cv2.bn.weight', 'model.model.2.m.0.cv2.bn.bias', 'model.model.2.m.0.cv2.bn.running_mean', 'model.model.2.m.0.cv2.bn.running_var', 'model.model.3.conv.weight', 'model.model.3.bn.weight', 'model.model.3.bn.bias', 'model.model.3.bn.running_mean', 'model.model.3.bn.running_var', 'model.model.4.cv1.conv.weight', 'model.model.4.cv1.bn.weight', 'model.model.4.cv1.bn.bias', 'model.model.4.cv1.bn.running_mean', 'model.model.4.cv1.bn.running_var', 'model.model.4.cv2.conv.weight', 'model.model.4.cv2.bn.weight', 'model.model.4.cv2.bn.bias', 'model.model.4.cv2.bn.running_mean', 'model.model.4.cv2.bn.running_var', 'model.model.4.m.0.cv1.conv.weight', 'model.model.4.m.0.cv1.bn.weight', 'model.model.4.m.0.cv1.bn.bias', 'model.model.4.m.0.cv1.bn.running_mean', 'model.model.4.m.0.cv1.bn.running_var', 'model.model.4.m.0.cv2.conv.weight', 'model.model.4.m.0.cv2.bn.weight', 'model.model.4.m.0.cv2.bn.bias', 'model.model.4.m.0.cv2.bn.running_mean', 'model.model.4.m.0.cv2.bn.running_var', 'model.model.4.m.1.cv1.conv.weight', 'model.model.4.m.1.cv1.bn.weight', 'model.model.4.m.1.cv1.bn.bias', 'model.model.4.m.1.cv1.bn.running_mean', 'model.model.4.m.1.cv1.bn.running_var', 'model.model.4.m.1.cv2.conv.weight', 'model.model.4.m.1.cv2.bn.weight', 'model.model.4.m.1.cv2.bn.bias', 'model.model.4.m.1.cv2.bn.running_mean', 'model.model.4.m.1.cv2.bn.running_var', 'model.model.5.conv.weight', 'model.model.5.bn.weight', 'model.model.5.bn.bias', 'model.model.5.bn.running_mean', 'model.model.5.bn.running_var', 'model.model.6.cv1.conv.weight', 'model.model.6.cv1.bn.weight', 'model.model.6.cv1.bn.bias', 'model.model.6.cv1.bn.running_mean', 'model.model.6.cv1.bn.running_var', 'model.model.6.cv2.conv.weight', 'model.model.6.cv2.bn.weight', 'model.model.6.cv2.bn.bias', 'model.model.6.cv2.bn.running_mean', 'model.model.6.cv2.bn.running_var', 'model.model.6.m.0.cv1.conv.weight', 'model.model.6.m.0.cv1.bn.weight', 'model.model.6.m.0.cv1.bn.bias', 'model.model.6.m.0.cv1.bn.running_mean', 'model.model.6.m.0.cv1.bn.running_var', 'model.model.6.m.0.cv2.conv.weight', 'model.model.6.m.0.cv2.bn.weight', 'model.model.6.m.0.cv2.bn.bias', 'model.model.6.m.0.cv2.bn.running_mean', 'model.model.6.m.0.cv2.bn.running_var', 'model.model.6.m.1.cv1.conv.weight', 'model.model.6.m.1.cv1.bn.weight', 'model.model.6.m.1.cv1.bn.bias', 'model.model.6.m.1.cv1.bn.running_mean', 'model.model.6.m.1.cv1.bn.running_var', 'model.model.6.m.1.cv2.conv.weight', 'model.model.6.m.1.cv2.bn.weight', 'model.model.6.m.1.cv2.bn.bias', 'model.model.6.m.1.cv2.bn.running_mean', 'model.model.6.m.1.cv2.bn.running_var', 'model.model.7.conv.weight', 'model.model.7.bn.weight', 'model.model.7.bn.bias', 'model.model.7.bn.running_mean', 'model.model.7.bn.running_var', 'model.model.8.cv1.conv.weight', 'model.model.8.cv1.bn.weight', 'model.model.8.cv1.bn.bias', 'model.model.8.cv1.bn.running_mean', 'model.model.8.cv1.bn.running_var', 'model.model.8.cv2.conv.weight', 'model.model.8.cv2.bn.weight', 'model.model.8.cv2.bn.bias', 'model.model.8.cv2.bn.running_mean', 'model.model.8.cv2.bn.running_var', 'model.model.8.m.0.cv1.conv.weight', 'model.model.8.m.0.cv1.bn.weight', 'model.model.8.m.0.cv1.bn.bias', 'model.model.8.m.0.cv1.bn.running_mean', 'model.model.8.m.0.cv1.bn.running_var', 'model.model.8.m.0.cv2.conv.weight', 'model.model.8.m.0.cv2.bn.weight', 'model.model.8.m.0.cv2.bn.bias', 'model.model.8.m.0.cv2.bn.running_mean', 'model.model.8.m.0.cv2.bn.running_var', 'model.model.9.cv1.conv.weight', 'model.model.9.cv1.bn.weight', 'model.model.9.cv1.bn.bias', 'model.model.9.cv1.bn.running_mean', 'model.model.9.cv1.bn.running_var', 'model.model.9.cv2.conv.weight', 'model.model.9.cv2.bn.weight', 'model.model.9.cv2.bn.bias', 'model.model.9.cv2.bn.running_mean', 'model.model.9.cv2.bn.running_var', 'model.model.12.cv1.conv.weight', 'model.model.12.cv1.bn.weight', 'model.model.12.cv1.bn.bias', 'model.model.12.cv1.bn.running_mean', 'model.model.12.cv1.bn.running_var', 'model.model.12.cv2.conv.weight', 'model.model.12.cv2.bn.weight', 'model.model.12.cv2.bn.bias', 'model.model.12.cv2.bn.running_mean', 'model.model.12.cv2.bn.running_var', 'model.model.12.m.0.cv1.conv.weight', 'model.model.12.m.0.cv1.bn.weight', 'model.model.12.m.0.cv1.bn.bias', 'model.model.12.m.0.cv1.bn.running_mean', 'model.model.12.m.0.cv1.bn.running_var', 'model.model.12.m.0.cv2.conv.weight', 'model.model.12.m.0.cv2.bn.weight', 'model.model.12.m.0.cv2.bn.bias', 'model.model.12.m.0.cv2.bn.running_mean', 'model.model.12.m.0.cv2.bn.running_var', 'model.model.15.cv1.conv.weight', 'model.model.15.cv1.bn.weight', 'model.model.15.cv1.bn.bias', 'model.model.15.cv1.bn.running_mean', 'model.model.15.cv1.bn.running_var', 'model.model.15.cv2.conv.weight', 'model.model.15.cv2.bn.weight', 'model.model.15.cv2.bn.bias', 'model.model.15.cv2.bn.running_mean', 'model.model.15.cv2.bn.running_var', 'model.model.15.m.0.cv1.conv.weight', 'model.model.15.m.0.cv1.bn.weight', 'model.model.15.m.0.cv1.bn.bias', 'model.model.15.m.0.cv1.bn.running_mean', 'model.model.15.m.0.cv1.bn.running_var', 'model.model.15.m.0.cv2.conv.weight', 'model.model.15.m.0.cv2.bn.weight', 'model.model.15.m.0.cv2.bn.bias', 'model.model.15.m.0.cv2.bn.running_mean', 'model.model.15.m.0.cv2.bn.running_var', 'model.model.16.conv.weight', 'model.model.16.bn.weight', 'model.model.16.bn.bias', 'model.model.16.bn.running_mean', 'model.model.16.bn.running_var', 'model.model.18.cv1.conv.weight', 'model.model.18.cv1.bn.weight', 'model.model.18.cv1.bn.bias', 'model.model.18.cv1.bn.running_mean', 'model.model.18.cv1.bn.running_var', 'model.model.18.cv2.conv.weight', 'model.model.18.cv2.bn.weight', 'model.model.18.cv2.bn.bias', 'model.model.18.cv2.bn.running_mean', 'model.model.18.cv2.bn.running_var', 'model.model.18.m.0.cv1.conv.weight', 'model.model.18.m.0.cv1.bn.weight', 'model.model.18.m.0.cv1.bn.bias', 'model.model.18.m.0.cv1.bn.running_mean', 'model.model.18.m.0.cv1.bn.running_var', 'model.model.18.m.0.cv2.conv.weight', 'model.model.18.m.0.cv2.bn.weight', 'model.model.18.m.0.cv2.bn.bias', 'model.model.18.m.0.cv2.bn.running_mean', 'model.model.18.m.0.cv2.bn.running_var', 'model.model.19.conv.weight', 'model.model.19.bn.weight', 'model.model.19.bn.bias', 'model.model.19.bn.running_mean', 'model.model.19.bn.running_var', 'model.model.21.cv1.conv.weight', 'model.model.21.cv1.bn.weight', 'model.model.21.cv1.bn.bias', 'model.model.21.cv1.bn.running_mean', 'model.model.21.cv1.bn.running_var', 'model.model.21.cv2.conv.weight', 'model.model.21.cv2.bn.weight', 'model.model.21.cv2.bn.bias', 'model.model.21.cv2.bn.running_mean', 'model.model.21.cv2.bn.running_var', 'model.model.21.m.0.cv1.conv.weight', 'model.model.21.m.0.cv1.bn.weight', 'model.model.21.m.0.cv1.bn.bias', 'model.model.21.m.0.cv1.bn.running_mean', 'model.model.21.m.0.cv1.bn.running_var', 'model.model.21.m.0.cv2.conv.weight', 'model.model.21.m.0.cv2.bn.weight', 'model.model.21.m.0.cv2.bn.bias', 'model.model.21.m.0.cv2.bn.running_mean', 'model.model.21.m.0.cv2.bn.running_var', 'model.model.22.cv2.0.0.conv.weight', 'model.model.22.cv2.0.0.bn.weight', 'model.model.22.cv2.0.0.bn.bias', 'model.model.22.cv2.0.0.bn.running_mean', 'model.model.22.cv2.0.0.bn.running_var', 'model.model.22.cv2.0.1.conv.weight', 'model.model.22.cv2.0.1.bn.weight', 'model.model.22.cv2.0.1.bn.bias', 'model.model.22.cv2.0.1.bn.running_mean', 'model.model.22.cv2.0.1.bn.running_var', 'model.model.22.cv2.0.2.weight', 'model.model.22.cv2.0.2.bias', 'model.model.22.cv2.1.0.conv.weight', 'model.model.22.cv2.1.0.bn.weight', 'model.model.22.cv2.1.0.bn.bias', 'model.model.22.cv2.1.0.bn.running_mean', 'model.model.22.cv2.1.0.bn.running_var', 'model.model.22.cv2.1.1.conv.weight', 'model.model.22.cv2.1.1.bn.weight', 'model.model.22.cv2.1.1.bn.bias', 'model.model.22.cv2.1.1.bn.running_mean', 'model.model.22.cv2.1.1.bn.running_var', 'model.model.22.cv2.1.2.weight', 'model.model.22.cv2.1.2.bias', 'model.model.22.cv2.2.0.conv.weight', 'model.model.22.cv2.2.0.bn.weight', 'model.model.22.cv2.2.0.bn.bias', 'model.model.22.cv2.2.0.bn.running_mean', 'model.model.22.cv2.2.0.bn.running_var', 'model.model.22.cv2.2.1.conv.weight', 'model.model.22.cv2.2.1.bn.weight', 'model.model.22.cv2.2.1.bn.bias', 'model.model.22.cv2.2.1.bn.running_mean', 'model.model.22.cv2.2.1.bn.running_var', 'model.model.22.cv2.2.2.weight', 'model.model.22.cv2.2.2.bias', 'model.model.22.cv3.0.0.conv.weight', 'model.model.22.cv3.0.0.bn.weight', 'model.model.22.cv3.0.0.bn.bias', 'model.model.22.cv3.0.0.bn.running_mean', 'model.model.22.cv3.0.0.bn.running_var', 'model.model.22.cv3.0.1.conv.weight', 'model.model.22.cv3.0.1.bn.weight', 'model.model.22.cv3.0.1.bn.bias', 'model.model.22.cv3.0.1.bn.running_mean', 'model.model.22.cv3.0.1.bn.running_var', 'model.model.22.cv3.0.2.weight', 'model.model.22.cv3.0.2.bias', 'model.model.22.cv3.1.0.conv.weight', 'model.model.22.cv3.1.0.bn.weight', 'model.model.22.cv3.1.0.bn.bias', 'model.model.22.cv3.1.0.bn.running_mean', 'model.model.22.cv3.1.0.bn.running_var', 'model.model.22.cv3.1.1.conv.weight', 'model.model.22.cv3.1.1.bn.weight', 'model.model.22.cv3.1.1.bn.bias', 'model.model.22.cv3.1.1.bn.running_mean', 'model.model.22.cv3.1.1.bn.running_var', 'model.model.22.cv3.1.2.weight', 'model.model.22.cv3.1.2.bias', 'model.model.22.cv3.2.0.conv.weight', 'model.model.22.cv3.2.0.bn.weight', 'model.model.22.cv3.2.0.bn.bias', 'model.model.22.cv3.2.0.bn.running_mean', 'model.model.22.cv3.2.0.bn.running_var', 'model.model.22.cv3.2.1.conv.weight', 'model.model.22.cv3.2.1.bn.weight', 'model.model.22.cv3.2.1.bn.bias', 'model.model.22.cv3.2.1.bn.running_mean', 'model.model.22.cv3.2.1.bn.running_var', 'model.model.22.cv3.2.2.weight', 'model.model.22.cv3.2.2.bias', 'model.model.22.dfl.conv.weight', 'model.model.22.cv4.0.0.conv.weight', 'model.model.22.cv4.0.0.bn.weight', 'model.model.22.cv4.0.0.bn.bias', 'model.model.22.cv4.0.0.bn.running_mean', 'model.model.22.cv4.0.0.bn.running_var', 'model.model.22.cv4.0.1.conv.weight', 'model.model.22.cv4.0.1.bn.weight', 'model.model.22.cv4.0.1.bn.bias', 'model.model.22.cv4.0.1.bn.running_mean', 'model.model.22.cv4.0.1.bn.running_var', 'model.model.22.cv4.0.2.weight', 'model.model.22.cv4.0.2.bias', 'model.model.22.cv4.1.0.conv.weight', 'model.model.22.cv4.1.0.bn.weight', 'model.model.22.cv4.1.0.bn.bias', 'model.model.22.cv4.1.0.bn.running_mean', 'model.model.22.cv4.1.0.bn.running_var', 'model.model.22.cv4.1.1.conv.weight', 'model.model.22.cv4.1.1.bn.weight', 'model.model.22.cv4.1.1.bn.bias', 'model.model.22.cv4.1.1.bn.running_mean', 'model.model.22.cv4.1.1.bn.running_var', 'model.model.22.cv4.1.2.weight', 'model.model.22.cv4.1.2.bias', 'model.model.22.cv4.2.0.conv.weight', 'model.model.22.cv4.2.0.bn.weight', 'model.model.22.cv4.2.0.bn.bias', 'model.model.22.cv4.2.0.bn.running_mean', 'model.model.22.cv4.2.0.bn.running_var', 'model.model.22.cv4.2.1.conv.weight', 'model.model.22.cv4.2.1.bn.weight', 'model.model.22.cv4.2.1.bn.bias', 'model.model.22.cv4.2.1.bn.running_mean', 'model.model.22.cv4.2.1.bn.running_var', 'model.model.22.cv4.2.2.weight', 'model.model.22.cv4.2.2.bias'], unexpected_keys=['epoch', 'best_fitness', 'ema', 'updates', 'optimizer', 'train_args', 'date', 'version'])

Then I trained the model: model.train(...) and did not get the same results compared to when training a pre-trained model:

from ultralytics import YOLO

model = YOLO('yolov8n-pose.pt') 

model.train(...)

It appears the issue stems from a mismatch between the YAML file for the model structure I have and the actual structure of the pre-trained yolov8n-pose.pt file. Could you guide me on where to find the corresponding model YAML file for the pre-trained YOLOv8-pose nano version?

I do appreciate your time and help.

[glenn-jocher]

Hey @enazari,

It looks like the structure defined in your YAML file doesn't match the pre-trained yolov8n-pose.pt model's architecture, hence the key mismatches. The pre-trained model's weights expect a certain architecture, and if your YAML file defines a different one, the weights won't load correctly.

The YAML file for the pre-trained models is typically included in the repository or the release assets. If you can't find the exact YAML file for the yolov8n-pose model, it might be worth checking the official Ultralytics YOLOv8 repository again or reaching out on the issues page for the correct file.

For the best results, ensure that the YAML file defining the model architecture matches the pre-trained model's structure. Once you have the correct YAML file, you should be able to load the weights without the key errors and proceed with training on your dataset.

Here's a quick check you can do:

from ultralytics import YOLO

# Load the pre-trained model
model = YOLO('yolov8n-pose.pt') 

# Check the model architecture
print(model.model)

This will print out the architecture of the pre-trained model, which you can compare with the structure defined in your YAML file.

Hope this helps! 🚀

[enazari]

I resolved the issue I was having by running: raw_model.model.load_state_dict(pretrained_model.model.state_dict(), strict= False)

This time after calling load_state_dict, the following is printed: <All keys matched successfully>

Here is the complete code:

from ultralytics import YOLO

pretrained_model = YOLO('yolov8n-pose.pt')

raw_model = YOLO('yolov8-pose.yaml') #obtained from https://github.com/ultralytics/ultralytics/blob/main/ultralytics/cfg/models/v8/yolov8-pose.yaml

raw_model.model.load_state_dict(pretrained_model.model.state_dict(), strict=False)

Despite ensuring that the weights of both models are identical post-loading, I am unable to achieve comparable training outcomes. Training the pretrained_model yields significantly better results than when training the loaded raw_model. I have verified that the weights of two models are the same after loading but are still unable to replicate the training results. Could you shed light on any additional elements, aside from the initial weights, that might be causing this variation in results?

Thank you.

[glenn-jocher]

@enazari hey there!

Great to hear you've resolved the key mismatch issue! 🎉 The code snippet you've shared looks spot on for transferring weights.

If you're seeing different training outcomes, it might be due to factors like different hyperparameters, training data variations, or even random seeds affecting the initialization of new layers. Also, check if the learning rate scheduler and optimizer states are being transferred, as these can significantly impact training.

Here's a quick snippet to ensure optimizer states are aligned if you haven't done this already:

raw_model.optimizer.load_state_dict(pretrained_model.optimizer.state_dict())

Keep in mind that any layers in raw_model not present in pretrained_model will have random weights, which could also affect performance.

Hope this helps, and happy fine-tuning! 🚀

[enazari]

Hi @glenn-jocher ,

Upon implementing your/your llm's suggestion, I faced this error:

AttributeError: 'YOLO' object has no attribute 'optimizer'

It would be great if you could let me know how I could access the optimizer state of the pre-trained model. Thanks.

[glenn-jocher]

@enazari hi there! 😊

It looks like you're trying to access the optimizer directly from the YOLO object, which isn't directly exposed. For handling the optimizer, you typically work within the training script where the optimizer is defined and used.

If you're working with custom training scripts or modifying the training process, you'd set up the optimizer there, something like this:

import torch.optim as optim

optimizer = optim.Adam(raw_model.parameters(), lr=0.001)

For transferring optimizer states from one training session to another (especially when fine-tuning or continuing training), you'd usually save and load the optimizer state along with the model weights:

# Saving
torch.save({
    'model_state_dict': model.state_dict(),
    'optimizer_state_dict': optimizer.state_dict(),
    ...
}, PATH)

# Loading
checkpoint = torch.load(PATH)
model.load_state_dict(checkpoint['model_state_dict'])
optimizer.load_state_dict(checkpoint['optimizer_state_dict'])

If you're using the Ultralytics YOLOv8 framework directly, adjustments to the optimizer are typically handled internally during the training process, and you might not need to manually access or modify the optimizer state.

Hope this clears things up! Let me know if you have any more questions.

[tcc940308]

Hi @glenn-jocher, I want to print out the features after extraction from the backbone, but I'm not sure which file needs modification. This is because I intend to perform some data processing after the backbone extraction. Could you offer some advice? Thank you!

[glenn-jocher]

@tcc940308 hey there! 😊

Sure thing, you can tap into the backbone features right after the extraction process by modifying models/yolo.py (or the equivalent if you're working with a different structure). Look for the forward() function within the model definition. Right after the backbone processing, you should see where the features are passed on for further processing (like to the neck or head of the network).

Here's a simplified example:

def forward(self, x):
    # x is your input image tensor
    features = self.backbone(x)  # Backbone feature extraction

    # Print or process your features here
    print(features.shape)
    # Your data processing code here

    # Continue with the rest of model processing...
    output = self.neck(features)
    return output

Just insert your processing code right after the features are extracted by the backbone. Keep in mind, printing out the features directly might slow down your processing significantly if you're doing it every iteration, especially with large datasets or deep models.

Hope this helps you out! If you need more specific advice, feel free to ask. Happy coding!

[RamPraveen2710]

from ultralytics import YOLO

Load the pre-trained model

model =

!yolo \ task=segment \ mode=train \ model='yolov8-cls-resnet50.yaml' \ imgsz=640 \ data=trashcan_inst_material.yaml \ epochs={8} \ batch=16 \ name=yolov8_res_seg\ exist_ok=True \ amp=False

hey i was using the configuration file "yolov8-cls-resnet50.yaml" for my object segmentation dataset but i am getting the below error can u pls say

Traceback (most recent call last): File "/usr/local/bin/yolo", line 8, in sys.exit(entrypoint()) File "/usr/local/lib/python3.10/dist-packages/ultralytics/cfg/init.py", line 582, in entrypoint getattr(model, mode)(overrides) # default args from model File "/usr/local/lib/python3.10/dist-packages/ultralytics/engine/model.py", line 673, in train self.trainer.train() File "/usr/local/lib/python3.10/dist-packages/ultralytics/engine/trainer.py", line 198, in train self._do_train(world_size) File "/usr/local/lib/python3.10/dist-packages/ultralytics/engine/trainer.py", line 370, in _do_train self.loss, self.loss_items = self.model(batch) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in _call_impl return forward_call(args, kwargs) File "/usr/local/lib/python3.10/dist-packages/ultralytics/nn/tasks.py", line 88, in forward return self.loss(x, *args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/ultralytics/nn/tasks.py", line 264, in loss self.criterion = self.init_criterion() File "/usr/local/lib/python3.10/dist-packages/ultralytics/nn/tasks.py", line 371, in init_criterion return v8SegmentationLoss(self) File "/usr/local/lib/python3.10/dist-packages/ultralytics/utils/loss.py", line 256, in init super().init(model) File "/usr/local/lib/python3.10/dist-packages/ultralytics/utils/loss.py", line 159, in init self.stride = m.stride # model strides File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1688, in getattr raise AttributeError(f"'{type(self).name}' object has no attribute '{name}'") AttributeError: 'Classify' object has no attribute 'stride' terminate called without an active exception

[glenn-jocher]

Hey there! 🌟 It looks like you're trying to train an object segmentation model using a classification model configuration file (yolov8-cls-resnet50.yaml). For segmentation tasks, you'll need to use a configuration file specifically designed for segmentation, as classification configurations lack necessary attributes like stride, which is essential for segmentation models.

Here's a quick tip: make sure you're using a segmentation-oriented YAML file. If your dataset is for segmentation, the model configuration must also support segmentation. The error you're encountering indicates a mismatch in the configuration and task types. 😊

For object segmentation, you'd typically use a YAML configuration that's akin to something designed for segmentation tasks. Unfortunately, yolov8-cls-resnet50.yaml is tailored for classification, thus causing the error. If you're looking for segmentation, try finding or creating a YAML configuration meant for segment tasks. The official Ultralytics repository might have examples or more information on segmentation configurations.

Hope this helps, and feel free to reach out if you have more questions!

[tcc940308]

@tcc940308 hey there! 😊

Sure thing, you can tap into the backbone features right after the extraction process by modifying models/yolo.py (or the equivalent if you're working with a different structure). Look for the forward() function within the model definition. Right after the backbone processing, you should see where the features are passed on for further processing (like to the neck or head of the network).

Here's a simplified example:

def forward(self, x):
    # x is your input image tensor
    features = self.backbone(x)  # Backbone feature extraction

    # Print or process your features here
    print(features.shape)
    # Your data processing code here

    # Continue with the rest of model processing...
    output = self.neck(features)
    return output

Just insert your processing code right after the features are extracted by the backbone. Keep in mind, printing out the features directly might slow down your processing significantly if you're doing it every iteration, especially with large datasets or deep models.

Hope this helps you out! If you need more specific advice, feel free to ask. Happy coding!

Hello @glenn-jocher ,

Thank you for your reply.

However, I am still confused. I couldn't find any Python file named model.py in the models folder. Therefore, I am unable to locate the function you mentioned, def forward(self, x): Is this the correct path? https://github.com/ultralytics/ultralytics/tree/main/ultralytics/models/yolo Did I overlook something?

Thank you!

[glenn-jocher]

Hey @tcc940308! Glad you reached out. 😄 My apologies for the confusion earlier. It looks like there was a misunderstanding in my previous instruction regarding the file path and names for modifying the model architecture.

The actual file and function you're looking to modify may differ based on the exact version of the YOLOv8 implementation you're using. Typically, for customization like adding attention mechanisms or accessing backbone features, you'd look into the file where the model's layers and forward pass are defined.

Given the URL you shared, it seems there might have been updates or changes in the repository structure. I recommend checking for a file named something similar to yolo.py or model.py within the repository. It's possible that the architecture definitions or forward functions might be located in differently named files or within a sub-directory.

If you're diving into model customization, you might want to explore files like yolo.py (if available) or other Python files in the models directory that define the neural network's class. The forward method will likely be part of the main model class within one of these files.

If you're still unable to find the right place to add your modifications, searching through the repository for keywords related to "model" or "forward" might help locate the correct file. Also, considering reaching out in the repository's discussions or issues for guidance specific to the latest structure they have.

Hope this points you in the right direction! Keep experimenting and don't hesitate to ping if you have more questions. Happy coding! 🚀

[KomisD]

Hey @glenn-jocher Cheers to your great work answering peoples questions!

I wanted to ask the community something else that I didn't find any relevant info on it. To be more specific I want to create a new architecture for the Yolo model that utilize a two stream architecture. I want to have a custom module that take the initial input passed into the model as well.

I created a yaml file like this:

nc: 6 # number of classes
scales: # model compound scaling constants, i.e. 'model=yolov8n.yaml' will call yolov8.yaml with scale 'n'
  # [depth, width, max_channels]
  s: [0.33, 0.50, 1024] # YOLOv8s summary: 225 layers, 11166560 parameters, 11166544 gradients,  28.8 GFLOPs

# YOLOv8.0n backbone
backbone:
  # [from, repeats, module, args]
  - [-1, 1, Conv, [64, 3, 2]] # 0-P1/2
  - [-1, 1, Conv, [128, 3, 2]] # 1-P2/4
  - [-1, 3, C2f, [128, True]]
  - [-1, 1, Conv, [256, 3, 2]] # 3-P3/8
  - [-1, 6, C2f, [256, True]]
  - [-1, 1, Conv, [512, 3, 2]] # 5-P4/16
  - [-1, 6, C2f, [512, True]]
  - [-1, 1, Conv, [1024, 3, 2]] # 7-P5/32
  - [-1, 3, C2f, [1024, True]]
  - [-1, 1, SPPF, [1024, 5]] # 9
  - [-11, 10, Preprocess, [128]]

# YOLOv8.0n head
head:
  - [-2, 1, nn.Upsample, [None, 2, "nearest"]]
  - [[-1, 5], 1, Concat, [1]] # cat backbone P4
  - [-1, 3, C2f, [512]] # 12

  - [-1, 1, nn.Upsample, [None, 2, "nearest"]]
  - [[-1, 4], 1, Concat, [1]] # cat backbone P3
  - [-1, 3, C2f, [256]] # 15 (P3/8-small)

  - [-1, 1, LCBHAM, [256]]
  - [[-1, 12], 1, Concat, [1]] # cat head P4
  - [-1, 3, C2f, [512]] # 18 (P4/16-medium)

  - [-1, 1, Conv, [512, 3, 2]]
  - [[-1, 9], 1, Concat, [1]] # cat head P5
  - [-1, 3, C2f, [1024]] # 21 (P5/32-large)

  - [[16, 19, 22], 1, Detect, [nc]] # Detect(P3, P4, P5)

I want the Preprocess layer to take the same input as - [-1, 1, Conv, [64, 3, 2]] # 0-P1/2 does, input[1, 3, 256, 256] in my case.

When I try to run above code I get this error:

  Traceback (most recent call last):
  File "c:\Users\dcube1\Desktop\Thesis\Training.py", line 5, in <module>
    model = YOLO('yolov8s-Second_stream.yaml') # Number of classes updated to 6
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\dcube1\anaconda3\envs\gpu_env\Lib\site-packages\ultralytics\models\yolo\model.py", line 23, in __init__
    super().__init__(model=model, task=task, verbose=verbose)
  File "C:\Users\dcube1\anaconda3\envs\gpu_env\Lib\site-packages\ultralytics\engine\model.py", line 149, in __init__
    self._new(model, task=task, verbose=verbose)
  File "C:\Users\dcube1\anaconda3\envs\gpu_env\Lib\site-packages\ultralytics\engine\model.py", line 218, in _new
    self.model = (model or self._smart_load("model"))(cfg_dict, verbose=verbose and RANK == -1)  # build model
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\dcube1\anaconda3\envs\gpu_env\Lib\site-packages\ultralytics\nn\tasks.py", line 294, in __init__
    self.model, self.save = parse_model(deepcopy(self.yaml), ch=ch, verbose=verbose)  # model, savelist
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\dcube1\anaconda3\envs\gpu_env\Lib\site-packages\ultralytics\nn\tasks.py", line 898, in parse_model
    c1, c2 = ch[f], args[0]
             ~~^^^
IndexError: list index out of range

I would appreciate any help or any resources that might be helpful for my case.

[glenn-jocher]

@KomisD hey there! 😊

It seems like the issue you're encountering is due to how the model is parsing the layers from your YAML configuration. Specifically, the Preprocess layer trying to access the same input as the first convolutional layer is causing confusion because of how the layer indices are being resolved. The error message IndexError: list index out of range usually occurs when trying to access a model layer or a channel dimension that doesn't exist according to the specified index.

To achieve the two-stream architecture where the Preprocess layer takes the same input as the initial convolution layer, try directly referencing the input layer by its index. In YOLO architectures defined by .yaml files, the -1 index refers to the output of the previous layer. To reference the initial input for a later layer, you typically need to use the index 0 since the input image is considered the first item in the layer list during model parsing.

However, in your case, using [-11, 10, Preprocess, [128]] seems to attempt a complex operation that might not be supported directly by the parsing logic as expected. The key is ensuring your custom module (Preprocess in this case) can properly interpret this instruction, which may require custom modifications in the parsing code (parse_model function) within nn/tasks.py to handle this special case where a layer needs to fetch the original input beyond simple sequential processing.

As a high-level suggestion, ensure your custom Preprocess layer and any custom handling logic are correctly implemented in the code that interprets this YAML configuration. You might need to adjust the parse_model function or wherever the model architecture is constructed from the YAML to properly handle this non-standard layer behavior.

Short example on how to reference initial input, assuming Preprocess expects it and is custom implemented to handle this:

backbone:
  ...
  - [0, 1, Preprocess, [128]] # Now directly says, "take layer 0's output as input"

Ensure Preprocess is properly coded to handle what input[1, 3, 256, 256] refers to. Unfortunately, without a direct look into how Preprocess is implemented, this suggestion is more about ensuring you're pointing your custom layer to the right input in the YAML and have the necessary logic in your model parsing code to understand this reference.

For more detailed assistance, it's beneficial to dive deeper into how the model and layers are implemented in ultralytics and whether other mechanisms or special configurations are needed for your specific architecture design.

Keep experimenting, and don't hesitate to reach out in the community forums or issues for more specific guidance! 🚀

[RamPraveen2710]

i have seen many research papers have modified i.e.improved yolov8 as they custamized for example below is one of the abstract of a research paper

Our algorithm focuses on three aspects. Firstly, we replace the original C2f module with Deformable Convnets v2 to enhance the adaptive ability of the target region in the convolution check feature map and extract the target region’s features more accurately. Secondly, we introduce SimAm, a non-parametric attention mechanism, which can deduce and assign three-dimensional attention weights without adding network parameters. Lastly, weoptimize the loss function by replacing the CIoU loss function with the Wise-IoU loss function. Wenamedour new algorithm DSW-YOLOv8n, which is an acronym of Deformable Convnets v2, SimAm, and Wise-IoU of the improved YOLOv8n(DSW-YOLOv8n). To conduct our experiments, we created our own dataset of underwater target detection for experimentation. Meanwhile, we also utilized the Pascal VOC dataset to evaluate our approach. The mAP@0.5 and mAP@0.5:0.95 of the original YOLOv8n algorithm on underwater target detection were 88.6% and 51.8%, respectively, and the DSW-YOLOv8n algorithm mAP@0.5 and mAP@0.5:0.95 can reach 91.8% and 55.9%.

i also want to improve the yolov8s to custamize according to my project on underwater trash object detection can u please help regarding

[glenn-jocher]

Hey there! 😊

Awesome to hear about your interest in customizing YOLOv8 for your underwater trash object detection project! It sounds like you're inspired by some cutting-edge modifications and ready to make your version shine.

Based on the abstract you mentioned, here are some brief pointers to get you started on customizing YOLOv8:

1. Deformable Convnets v2: You can integrate this into YOLOv8 by modifying the convolution layers in the model's definition. In PyTorch, you might use something like torchvision.ops.deform_conv2d to implement deformable convolutions.

2. SimAm: For a non-parametric attention mechanism like SimAm, you can incorporate it into the feature extraction layers. Essentially, you'd apply it after convolutional layers to focus on important features dynamically. Here's a rough idea:

   # Assuming `x` is the output of a convolution layer
   attention = SimAm(x)  # Apply your SimAm function here
   x = x * attention  # Scale features by attention

3. Wise-IoU Loss: Replacing CIoU with Wise-IoU means adjusting the loss calculation part of your code. This would involve defining the Wise-IoU loss function and replacing the existing IoU computation in the training loop.

Remember, integrating these changes will require familiarity with the model architecture and possibly tweaking the training pipeline to accommodate your custom dataset and loss function.

For experimenting and evaluating changes like DSW-YOLOv8n, ensure you have a valid dataset that represents your problem space well, such as your underwater dataset, and use a rigorous validation approach to quantify improvements.

Feel free to dive into the YOLOv8 source code. Decipher how the modules are constructed, and don't hesitate to experiment with these enhancements. The community and documentation can be great resources if you get stuck or need more specific advice.

Happy coding, and may your detection model make a splash! 🚀

[JoanneLin168]

Hey so I have made a modified YOLO model by inserting custom blocks into the backbone. When training, the summary does show that these blocks are inserted.

I wanted to use the pretrained weights from yolov8-seg.pt for my modified architecture so that it is faster to train. I have also written some code to shift the keys of the indexes of the weights down to allow for the new blocks that have been inserted.

However, when training the loss is higher than expected and the mAP is much lower than expected (the mAP is very close to 0, suggesting no weights were loaded). My custom blocks have all of the weights initially set to 0, and has a residual connection so in theory they should initially have no impact on the feature maps.

Am I inserting the weights in correctly? If I am, then what may be the mistake I have?

Below is my code:

import torch
import ultralytics
from ultralytics import YOLO

def load_weights(model, old_weights, block_idxs):

    new_weights = {}
    sorted_names = list(old_weights.keys())
    sorted_names.sort(key=lambda x: int(x.split('.')[1]))

    separated_modules = {}
    old_indices = []
    i = 0
    # 1. separate the modules
    # 2. for each module, determine the offset for the next module
    while i < len(sorted_names):
        name = sorted_names[i]
        module_index = int(name.split('.')[1])

        if module_index not in old_indices:
            separated_modules[module_index] = []
            old_indices.append(module_index)

        separated_modules[module_index].append(name)

        i += 1

    offset = 0
    last_index = 0
    for m in separated_modules:
        new_module_index = m + offset
        if new_module_index > last_index:
            last_index = new_module_index

        if new_module_index in block_idxs:
            offset += 1
            new_module_index += 1

        # Update every name in the module
        for name in separated_modules[m]:
            new_name = name.split('.')
            new_name[1] = str(new_module_index)
            new_name = '.'.join(new_name)
            new_weights[new_name] = old_weights[name]

    a = list(model.model.state_dict().keys())
    a.sort(key=lambda x: int(x.split('.')[1]))
    b = list(new_weights.keys())
    b.sort(key=lambda x: int(x.split('.')[1]))

    # assert that b is a subset of a
    assert set(b).issubset(set(a))

    model.model.load_state_dict(new_weights, strict=False)  # load a pretrained YOLOv8n segmentation model

    return model

if __name__ == '__main__':
    torch.backends.cudnn.benchmark = True
    torch.backends.cudnn.deterministic = True

    pretrained_model = YOLO('yolov8n-seg.pt').model
    weights = pretrained_model.state_dict()  # load weights from the pretrained YOLOv8n segmentation model

    model = YOLO('yolov8n-seg-custom.yaml')

    block_idxs = [5, 8, 11]
    model = load_weights(model, weights, block_idxs)

    # Train the model
    model = model.train(
        name='custom',

        data='./custom_dataset.yaml',
        epochs=10, 
        batch=4, 

        optimizer='SGD',
        momentum=0.9,
        weight_decay=0.0005,
        lr0=0.0001,
        lrf=0.000001,

        warmup_epochs=1,
        pretrained=False,
        save=True)

[glenn-jocher]

Hey there! 😊

It's great to see your initiative in customizing YOLOv8 by integrating custom blocks into the backbone and attempting to load pretrained weights creatively.

Based on the behavior you're observing (mAP close to zero), it appears some of the weights might not have been correctly assigned, or the custom blocks may have affected the model's stability in ways not initially anticipated.

When transferring weights, especially when modifying the architecture, it's crucial that the new weights align exactly with the corresponding parts of the model they are intended for. Errors in indexing or mismatches in layer configurations can easily lead to poor performance.

A few quick suggestions:

• Double-check the alignments of the weights: Ensure your modified keys in new_weights align with the respective modules in the updated model architecture.

• Initialize custom blocks more effectively: Instead of setting all initial weights of the custom blocks to zero, consider using a different initialization method, like Xavier or Kaiming, that might help mitigate initial training instability.

• Debugging step: After loading the weights, check for a proper load by comparing a few random weights from model.model.state_dict() and new_weights to ensure they match.

Here's how you can tweak weight initialization for custom blocks if you want to experiment:

for name, param in model.named_parameters():
    if 'custom_block_name' in name:  # replace 'custom_block_name' with the actual block name
        torch.nn.init.xavier_normal_(param)

Hope this helps clear up the situation! Let us know how it goes. Happy coding! 🚀

[JoanneLin168]

Hey there! 😊

It's great to see your initiative in customizing YOLOv8 by integrating custom blocks into the backbone and attempting to load pretrained weights creatively.

Based on the behavior you're observing (mAP close to zero), it appears some of the weights might not have been correctly assigned, or the custom blocks may have affected the model's stability in ways not initially anticipated.

When transferring weights, especially when modifying the architecture, it's crucial that the new weights align exactly with the corresponding parts of the model they are intended for. Errors in indexing or mismatches in layer configurations can easily lead to poor performance.

A few quick suggestions:

• Double-check the alignments of the weights: Ensure your modified keys in new_weights align with the respective modules in the updated model architecture.
• Initialize custom blocks more effectively: Instead of setting all initial weights of the custom blocks to zero, consider using a different initialization method, like Xavier or Kaiming, that might help mitigate initial training instability.
• Debugging step: After loading the weights, check for a proper load by comparing a few random weights from model.model.state_dict() and new_weights to ensure they match.

Here's how you can tweak weight initialization for custom blocks if you want to experiment:

for name, param in model.named_parameters():
    if 'custom_block_name' in name:  # replace 'custom_block_name' with the actual block name
        torch.nn.init.xavier_normal_(param)

Hope this helps clear up the situation! Let us know how it goes. Happy coding! 🚀

@glenn-jocher I checked the weights and they seem correct. I even did model.val('./custom_dataset.yaml') and it returned the same mAP results as the pretrained yolov8n-seg.pt model.

So I guess the weights are correctly in place, so what may be the reason for the mAP to go down to zero during the first epoch, with a high loss?

[JoanneLin168]

Hey there! 😊 It's great to see your initiative in customizing YOLOv8 by integrating custom blocks into the backbone and attempting to load pretrained weights creatively. Based on the behavior you're observing (mAP close to zero), it appears some of the weights might not have been correctly assigned, or the custom blocks may have affected the model's stability in ways not initially anticipated. When transferring weights, especially when modifying the architecture, it's crucial that the new weights align exactly with the corresponding parts of the model they are intended for. Errors in indexing or mismatches in layer configurations can easily lead to poor performance. A few quick suggestions:

• Double-check the alignments of the weights: Ensure your modified keys in new_weights align with the respective modules in the updated model architecture.
• Initialize custom blocks more effectively: Instead of setting all initial weights of the custom blocks to zero, consider using a different initialization method, like Xavier or Kaiming, that might help mitigate initial training instability.
• Debugging step: After loading the weights, check for a proper load by comparing a few random weights from model.model.state_dict() and new_weights to ensure they match.

Here's how you can tweak weight initialization for custom blocks if you want to experiment:

for name, param in model.named_parameters():
    if 'custom_block_name' in name:  # replace 'custom_block_name' with the actual block name
        torch.nn.init.xavier_normal_(param)

Hope this helps clear up the situation! Let us know how it goes. Happy coding! 🚀

@glenn-jocher I checked the weights and they seem correct. I even did model.val('./custom_dataset.yaml') and it returned the same mAP results as the pretrained yolov8n-seg.pt model.

So I guess the weights are correctly in place, so what may be the reason for the mAP to go down to zero during the first epoch, with a high loss?

Just to add to this, I printed the weights before model.train() and after model.train() for coco8-seg.yaml for 1 epoch and the weights were reset. However, this wasn't the issue when training the pretrained yolov8n-seg.pt model, so does model.train() reset all of the weights for custom architectures?