[Feature-Request] How can I pass in the transforms e.g. Normalization before a frame hits the model

[franva]

Start with the why:

I trained a custom model, this model requires normalization with a specific mean and std. This is the image pre-processing which we normally do before feeding an image to a model for prediction. AND With the depthai_demo.py, we have access to handler.py. But when the method decode() is hit, we already lost the window to do image pre-processing before the video frame is fed to the model. Thus the data we received from packet is not correctly segmented.

Move to the what:

An easy interface or a place where user can specify the mean and std for the Normalization in an array of transformations.

Move to the how:

May be something like:

Assume the model called road_seg.blob:

road_seg.json

{
    "nn_config": {
        "output_format" : "raw",
        "input_size": "416x416",
        "transformations":[
                "normalization: {
                        "mean": [0.5, 0.5, 0.5],
                        "std": [0.5, 0.5, 0.5],
                }
        ]
    },
    "handler": "handler.py"
}

[Erol444]

Hello @franva , usually this is done before converting model into IR format (xml/bin) with model optimizer, see docs here (section When to Specify Mean and Scale Values). And we already have that on TODO for our firmware, so you will be able to normalize the preview frame from the color camera, CC @themarpe on that.

[franva]

hi @Erol444 thanks for the timely reply.

The link is broken:

https://discord.com/channels/@me/877120431801376799/877927215831089233

It points to @me which I think it means your own link and others won't be able to see that.

I have found that link, do you mean this one?

If it is the one, then my question is, normally the Normalization requires not only mean but also std which I cannot find in OpenVINO's doc. How are we going to handle the std?

My feeling is that it's better to have the 2 configs, mean and std together in one place for convenience.

[Erol444]

Sorry @franva , I thought I copied the link not the discord message! I updated the link and yes, it's the correct one. I thought you were talking about scaling and mean values (that model optimizer), why would standard deviation be useful in this case, could you elaborate please? Thanks, Erik

[franva]

hi @Erol444 sure. I have updated my question and highlighted the changed part.