Pytorch Model Support?

[ambondarev]

I am just wondering if there will be support for .pt (Pytorch) format for models? I would like to give Blue Candle a serious try but the default yolov8 models are somewhat lacking. Mike Lud who provides the default models for CPAI and Deepstack has released yolov8 model here (https://github.com/MikeLud/CodeProject.AI-Custom-IPcam-Models/blob/main/YOLOv8%20Models/Custom%20Models/ipcam-general-v8.pt) and I think it would be great if Blue Candle will support loading them.

By default yolo models export in a number of formats and safetensors isn't one actively documented.

[xnorpx]

I did spend some time trying to convert to safetensors but with no success. I need to spend more time to figure it out.

[ambondarev]

I tried last night before I wrote this issue and had no luck as well. I think people who follow the yolov8 guides to train their custom models wont be able to use theirs easily either. I hope you either can include a converter or build support for those models so it would be easier to use.

[xnorpx]

Seems the translation needed is this one. Will try it out when schedule allows.

https://huggingface.co/lmz/candle-yolo-v8/discussions/1

import torch
from safetensors.torch import load_file
from safetensors.torch import save_file

def rename(name: str):
    name = name.replace("model.0.", "net.b1.0.")
    name = name.replace("model.1.", "net.b1.1.")
    name = name.replace("model.2.m.", "net.b2.0.bottleneck.")
    name = name.replace("model.2.", "net.b2.0.")
    name = name.replace("model.3.", "net.b2.1.")
    name = name.replace("model.3.", "net.b2.1.")
    name = name.replace("model.4.m.", "net.b2.2.bottleneck.")
    name = name.replace("model.4.", "net.b2.2.")
    name = name.replace("model.5.", "net.b3.0.")
    name = name.replace("model.6.m.", "net.b3.1.bottleneck.")
    name = name.replace("model.6.", "net.b3.1.")
    name = name.replace("model.7.", "net.b4.0.")
    name = name.replace("model.8.m.", "net.b4.1.bottleneck.")
    name = name.replace("model.8.", "net.b4.1.")
    name = name.replace("model.9.", "net.b5.0.")
    name = name.replace("model.12.m.", "fpn.n1.bottleneck.")
    name = name.replace("model.12.", "fpn.n1.")
    name = name.replace("model.15.m.", "fpn.n2.bottleneck.")
    name = name.replace("model.15.", "fpn.n2.")
    name = name.replace("model.16.", "fpn.n3.")
    name = name.replace("model.18.m.", "fpn.n4.bottleneck.")
    name = name.replace("model.18.", "fpn.n4.")
    name = name.replace("model.19.", "fpn.n5.")
    name = name.replace("model.21.m.", "fpn.n6.bottleneck.")
    name = name.replace("model.21.", "fpn.n6.")
    name = name.replace("model.22.", "head.")
    return name

data = torch.load("/tmp/yolov8s-pose.pt")
#data = torch.load("/tmp/yolov8s.pt")
print(data)
tensors = data['model'].state_dict().items()
tensors = dict(tensors)
tensors = {rename(k): t for k, t in tensors.items()}
print(data["model"])
save_file(tensors, "/tmp/model.safetensors")
for k, v in tensors.items():
    print(str(k), v.shape)

[ambondarev]

I tested, it does create a .safetensors file, but when I run the model through Blue Candle I get:

Error: Failed to parse multiples from model filename: "./models/ipcam-general-v8.safetensors"

Output log attached from the script you provided.

output.log

[xnorpx]

I was thinking of trying to convert the original yolo8 models first to see if I can reproduce the checked in safetensors. Once I verified I could reproduce the conversation then try to convert the custom trained one.

[xnorpx]

Ok, converting the model works with the above scripts but there is 2 remaining items that I need to solve.

Currently I just parse the multiples based on the filename, so if you rename the file to: yolov8s-ipcam-general.safetensors

Then it get passed that issue, now after that we have the classes issue. We need to know the number of classes. (COCO is 8) the ipcam-general is 2.

So I need add some support for this how to pass in the custom classes and number of them when we load the model. Also the labels need to be correct.

Here is the result when I hardcoded the num_classes to 2 with the general-model. As you can see it believes the Vehicle is Bicycle due to the coco classes.

blue_candle.exe --model .\models\yolov8s-ipcam-general.safetensors --test

[xnorpx]

I need to spend some time think how I can support non coco models in a decent way.

[jmh139]

Appreciate the work on this. I am using BlueCandle on two different Blue Iris installs and everything is working much better than CPAI with the exception of animals. We have a remote cabin and are not seeing any small animals picked up. We really only care about people and cars but it is nice to see the animals at times. Again really appreciate it.

[gabbymouth]

I just wanted to give this an upvote. I've been testing ipcam-general-v8.pt with my own python watchdog app (passing blue iris/other programs image files) and yolov8 on windows and it's very good! I am very impressed with it. It sounds like you have a plan and I'm all for it.

[ambondarev]

Could we as a placeholder maybe provide a file that outlines the classes and their index if the attributes are not available in safetensors format?

[xnorpx]

A little overloaded at work atm, but adding a cli option with a custom labels for a safetensor to unblock this I can try to do this weekend.

[jmh139]

Reading over this I am just curious if there is a way to use Mike Lud's custom models in any way? I have been using Blue Candle for months and it is working great but would loved to be able to get some of the animal alerts back.

[xnorpx]

Reading over this I am just curious if there is a way to use Mike Lud's custom models in any way? I have been using Blue Candle for months and it is working great but would loved to be able to get some of the animal alerts back.

It looks like Mike Luds haven't trained retrained animals on Yolo8 yet: https://github.com/MikeLud/CodeProject.AI-Custom-IPcam-Models/tree/main/YOLOv8%20Models/Custom%20Models

(I will get to this soon but work and life is busy atm)