TypeError in automatic annotations with tag

[TsykunovDmitriy]

I'm trying to add my classifier as a serverless function. But I get "TypeError: Cannot read properties of null (reading 'constructor')" in UI. I have simplified the handler function to the form:

def handler(context, event):
    context.logger.info("Run classifier...")

    results = [{
        "label": "class",
        "type": "tag",
    }]

    return context.Response(body=json.dumps(results), headers={},
        content_type='application/json', status_code=200)

But still getting the same error.

spec in yaml file looks like:

    spec: |
      [
        { 
          "id": 1, 
          "name": "class"
        }
      ]

[TsykunovDmitriy]

For reproducibility, you can modify this part of code: https://github.com/openvinotoolkit/cvat/blob/develop/serverless/openvino/omz/public/yolo-v3-tf/nuclio/model_handler.py#L153

results.append({
    "confidence": str(obj['confidence']),
    "label": self.labels.get(obj_class, "unknown"),
    # "points": [xtl, ytl, xbr, ybr],
    "type": "tag",
})

[bingo00]

I get the same problem, maybe it's caused by cvat_ui. Since the code of auto-annotation is updated without modifying the ui code. see cvat-ui/src/components/annotation-page/standard-workspace/controls-side-bar/tools-control.tsx, line 1035 when auto annotion completed, the result is still copied to objectType: ObjectType.SHAPE, and points return needed

                       const states = result.map(
                            (data: any): any => new core.classes.ObjectState({
                                shapeType: data.type,
                                label: jobInstance.labels.filter((label: any): boolean => label.name === data.label)[0],
                                points: data.points,
                                objectType: ObjectType.SHAPE,
                                frame,
                                occluded: false,
                                source: 'auto',
                                attributes: (data.attributes as { name: string, value: string }[])
                                    .reduce((mapping, attr) => {
                                        mapping[attrsMap[data.label][attr.name]] = attr.value;
                                        return mapping;
                                    }, {} as Record<number, string>),
                                zOrder: curZOrder,
                            }),
                        );

                        createAnnotations(jobInstance, frame, states);

Knowing how to modify this code may solve the problem

[RZetko]

Even if you send send points you'll still get error because type cannot be tag, it's supposed to be shape:

Could not create annotations
Error: Object shape must be one of: ["rectangle","polygon","polyline","points","ellipse","cuboid"]

One option could be to allow objectType be defined in response you send from serverless model. I'm not sure if it wouldn't break any other things, I haven't tested it and not sure that shapeType rectangle will work objectType tag, just a wild guess.

const states = result.map(
    (data: any): any => new core.classes.ObjectState({
        shapeType: data.type,
        label: jobInstance.labels.filter((label: any): boolean => label.name === data.label)[0],
        points: data.points,
        objectType: data.objectType ?? ObjectType.SHAPE,
        frame,
        occluded: false,
        source: 'auto',
        attributes: (data.attributes as { name: string, value: string }[])
            .reduce((mapping, attr) => {
                mapping[attrsMap[data.label][attr.name]] = attr.value;
                return mapping;
            }, {} as Record<number, string>),
        zOrder: curZOrder,
    }),
);

And response from serverless model would look like this:

{
    "confidence": "1.00",
    "label": "detection_label",
    "type": "rectangle",
    "objectType": "tag",
    "points": [0, 0, 100, 100],
    "attributes": [],
}

[SachidanandAlle]

I am looking for this feature support...

Basically there will some DeepLearning classification model which I would to run (possibly as detector) and tag the image to have all those classes...

Idea is to run the automation task using a classification model to tag all the frames or images.

Definitely want to use tag rather than hacking object shape to be rectangle etc.. so that annotators won't get confused and continue using Tags from CVAT

Any help on this? Any pointers to modify the code also works for now..

[SachidanandAlle]

I get the same problem, maybe it's caused by cvat_ui. Since the code of auto-annotation is updated without modifying the ui code. see cvat-ui/src/components/annotation-page/standard-workspace/controls-side-bar/tools-control.tsx, line 1035 when auto annotion completed, the result is still copied to objectType: ObjectType.SHAPE, and points return needed

                       const states = result.map(
                            (data: any): any => new core.classes.ObjectState({
                                shapeType: data.type,
                                label: jobInstance.labels.filter((label: any): boolean => label.name === data.label)[0],
                                points: data.points,
                                objectType: ObjectType.SHAPE,
                                frame,
                                occluded: false,
                                source: 'auto',
                                attributes: (data.attributes as { name: string, value: string }[])
                                    .reduce((mapping, attr) => {
                                        mapping[attrsMap[data.label][attr.name]] = attr.value;
                                        return mapping;
                                    }, {} as Record<number, string>),
                                zOrder: curZOrder,
                            }),
                        );

                        createAnnotations(jobInstance, frame, states);

Knowing how to modify this code may solve the problem

any luck on modifying the UI code to support creating Tags?

[bsekachev]

If you want to return tags from an annotation model, add objectType to model response and server response (need to check that cvat server also passes this to response). After that you can modify source code approximately this way to run detector on one frame: cvat-ui/src/components/annotation-page/standard-workspace/controls-side-bar/tools-control.tsx

return new core.classes.ObjectState({
    shapeType: data.type,
    label: jobLabel,
    points: data.points,
    objectType: data.objectType || ObjectType.SHAPE, 
    frame,
    occluded: data.occluded || false,
    source: 'auto',
    attributes: (data.attributes as { name: string, value: string }[])
        .reduce((acc, attr) => {
            const [modelAttr] = Object.entries(body.mapping[modelLabel].attributes)
                .find((value: string[]) => value[1] === attr.name) || [];
            const areCompatible = checkAttributesCompatibility(
                model.attributes[modelLabel].find((mAttr) => mAttr.name === modelAttr),
                jobLabel.attributes.find((jobAttr: Attribute) => (
                    jobAttr.name === attr.name
                )),
                attr.value,
            );

            if (areCompatible) {
                acc[attrsMap[data.label][attr.name]] = attr.value;
            }

            return acc;
        }, {} as Record<number, string>),
    zOrder: curZOrder,
});

When you are running detectors for the whole task, UI is not matter because all handling and object creating is on the server side. Probably need to modify this file cvat/apps/lambda_manager/views.py, LambdaFunction::invoke method.

Let me know if you need some more assistance, and of course, you are welcome to open a pull request :)

[bsekachev]

I will close the issue for now because I've answered the question, if somebody is able to contribute us the feature of automatic annotation of tags, it would be great.