Exception: input type is not supported.

[satyajithj]

Procedure: Opened an annotated image, removed two annotations and hit save. Note: Image is 512x512 and has 250-300 annotations.

Output:

[2019-03-29 13:25:25,166] ERROR in app: Exception on /api/annotator/data [POST]
flask_1     | Traceback (most recent call last):
flask_1     |   File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1813, in full_dispatch_request
flask_1     |     rv = self.dispatch_request()
flask_1     |   File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1799, in dispatch_request
flask_1     |     return self.view_functions[rule.endpoint](**req.view_args)
flask_1     |   File "/usr/local/lib/python3.6/site-packages/flask_restplus/api.py", line 325, in wrapper
flask_1     |     resp = resource(*args, **kwargs)
flask_1     |   File "/usr/local/lib/python3.6/site-packages/flask/views.py", line 88, in view
flask_1     |     return self.dispatch_request(*args, **kwargs)
flask_1     |   File "/usr/local/lib/python3.6/site-packages/flask_restplus/resource.py", line 44, in dispatch_request
flask_1     |     resp = meth(*args, **kwargs)
flask_1     |   File "/usr/local/lib/python3.6/site-packages/flask_login/utils.py", line 261, in decorated_view
flask_1     |     return func(*args, **kwargs)
flask_1     |   File "/workspace/app/api/annotator.py", line 90, in post
flask_1     |     paperjs_to_coco(width, height, paperjs_object)
flask_1     |   File "/workspace/app/util/coco_util.py", line 70, in paperjs_to_coco
flask_1     |     segments, image_height, image_width)
flask_1     |   File "/workspace/app/util/coco_util.py", line 77, in get_segmentation_area_and_bbox
flask_1     |     rles = mask.frPyObjects(segmentation, image_height, image_width)
flask_1     |   File "pycocotools/_mask.pyx", line 307, in pycocotools._mask.frPyObjects
flask_1     | Exception: input type is not supported.

[satyajithj]

Rolled back the delete operation, opened the image (the annotation is available again) and hit the save button. Results in the same error.

[jsbroks]

@fuzzyBatman Can you check the annotation in the database to see if the segmentation contains [256, 256] ?

[satyajithj]

Sorry, in the database?

[jsbroks]

The annotation you rolled back the delete on, does the segmentation contain [256, 256]

[satyajithj]

No. That particular annotation does not have [256, 256] appended at the end. Though another image has [256, 256] in at least one annotation, and the save operation works.

The vertex inside the red circle is at the centre of the image.

[jsbroks]

Any change you can share the annotation document stored in the database? You can us mongo compass to view the database

[satyajithj]

Upload the JSON file, right?

[jsbroks]

I would like the JSON of the annotations directly from the mongodb database, since not all data is exported in COCO

[satyajithj]

I'm sorry. I do not know how to do this. Do you mind giving me directions?

I installed mongodb-compass.

[jsbroks]

Open it and there should be an option to connect to the database. Assuming you are running this locally, you should be able to us the default settings to connect. Once connect go the the flask/annotation_models collection and type {_id: <ANNOTATION_ID>}. This should return the annotation

[satyajithj]

For

{"id":1903,"image_id":108,"category_id":1,"dataset_id":2,"segmentation":[[135.3,211,144.8,212.5,149.1,218.7,148.4,226.5,144.1,232,136.6,234.9,128.9,232.2,125.1,224.4,127.5,215.5],[256,256]],"area":429,"bbox":[125,211,24,24],"iscrowd":false,"creator":"fuzzybatman","width":512,"height":512,"color":"#d8a81e","metadata":{}}

The document from the database is _id_1903.txt

[jsbroks]

That particular annotation does not have [256, 256] appended at the end.

I see [256,256] appended to the end, which is what is cause the error to be thrown since it is not a valid polygon.

[satyajithj]

The document for the deleted annotation _id_1182.txt. This one does not contain a [256, 256].

Sorry for the confusion. The first annotation I uploaded (id: 1903) is for one that has a [256, 256]. There are 10 such instances in my annotation document.

[satyajithj]

Hey, just saw that there are 2 files that contain [256, 256] and they get saved. Not sure if that's what is causing the ERROR under consideration.

[chitraa]

Hi I have run into the same exception. Is there a solution to mitigate this problem? My app is hosted on a cloud service. I am inside the command line of mongodb-compass (inside the mongodb docker instance). What is the name of the db, and could you point out what I should be looking at?

[chitraa]

File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.6/site-packages/flask_restplus/api.py", line 325, in wrapper
    resp = resource(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/flask/views.py", line 88, in view
    return self.dispatch_request(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/flask_restplus/resource.py", line 44, in dispatch_request
    resp = meth(*args, **kwargs)
  File "/workspace/webserver/util/__init__.py", line 10, in wrap
    result = func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/flask_login/utils.py", line 261, in decorated_view
    return func(*args, **kwargs)
  File "/workspace/webserver/api/annotator.py", line 113, in post
    paperjs_to_coco(width, height, paperjs_object)
  File "/workspace/webserver/util/coco_util.py", line 75, in paperjs_to_coco
    segments, image_height, image_width)
  File "/workspace/webserver/util/coco_util.py", line 82, in get_segmentation_area_and_bbox
    rles = mask.frPyObjects(segmentation, image_height, image_width)
  File "pycocotools/_mask.pyx", line 307, in pycocotools._mask.frPyObjects
Exception: input type is not supported.

This is the same traceback @fuzzyBatman encountered. This occurs when I draw a polygon, and it allows me to draw another polygon before closing the last one.

[chitraa]

{"images": [{"id": 44, "dataset_id": 2, "path": "/datasets/1/1.png", "width": 55, "height": 50, "file_name": "1.png"}], "categories": [{"id": 1, "name": "color1", "supercategory": "", "color": "#cee43c", "metadata": {}}], "annotations": [{"id": 107, "image_id": 44, "category_id": 1, "segmentation": [[21.4, 11.7, 27.5, 15.2, 31.1, 20.5, 36.9, 25.2, 33.0, 31.7, 27.9, 35.9, 25.9, 30.8, 28.7, 27.1, 22.6, 27.3, 18.3, 22.3, 15.4, 21.6, 19.6, 16.9, 18.6, 12.5, 16.3, 11.2]], "area": 231, "bbox": [15.0, 11.0, 22.0, 24.0], "iscrowd": false, "color": "#406afd", "metadata": {}}]} meanwhile posting the exported json to make my point. These are two independent lines (unclosed polygons). Is this desired behavior?

[chitraa]

The image I annotated was of the resolution 56x50 pixels. I can reproduce the bug easily on such small resolutions. I copied the data in annotation_model of the db in the file annotation_model.txt and the exported json the file coco-1567160216.9040387.txt

[rbavery]

I'm also experiencing this bug specified above, "Procedure: Opened an annotated image, removed two annotations and hit save." I don't have [256, 256] appended to my segmentations. Here's an example:

{"id": 31, "image_id": 550, "category_id": 1, "iscrowd": 0, "area": 452, "bbox": [361.0, 66.0, 24.0, 24.0], "segmentation": [[377.0, 89.5, 367.0, 88.5, 360.5, 79.0, 363.5, 70.0, 371.0, 65.5, 378.0, 66.5, 384.5, 74.0, 384.5, 82.0, 377.0, 89.5]], "width": 512, "height": 512}, {"id": 32, "image_id": 550, "category_id": 1, "iscrowd": 0, "area": 473, "bbox": [307.0, 68.0, 25.0, 24.0], "segmentation": [[323.0, 91.5, 315.0, 91.5, 307.5, 84.0, 308.5, 73.0, 316.0, 67.5, 325.0, 68.5, 331.5, 76.0, 329.5, 87.0, 323.0, 91.5]], "width": 512, "height": 512}

and here is the traceback

annotator_webclient | [2019-09-06 02:47:24 +0000] [11] [INFO] rbavery has started annotating image 3
annotator_webclient | [2019-09-06 02:47:24 +0000] [11] [INFO] Socket connection has been disconnected with rbavery
annotator_webclient | [2019-09-06 02:47:24 +0000] [11] [INFO] Socket connection has been disconnected with rbavery
annotator_webclient | [2019-09-06 02:47:24 +0000] [11] [INFO] Socket connection has been disconnected with rbavery
annotator_webclient | [2019-09-06 02:47:24 +0000] [11] [INFO] Socket connection has been disconnected with rbavery
annotator_webclient | [2019-09-06 02:47:24 +0000] [11] [DEBUG] GET /api/annotator/data/3
annotator_webclient | [2019-09-06 02:47:24 +0000] [11] [INFO] Socket connection has been disconnected with rbavery
annotator_webclient | [2019-09-06 02:48:41 +0000] [11] [DEBUG] POST /api/annotator/data
annotator_webclient | [2019-09-06 02:48:43 +0000] [11] [ERROR] Exception on /api/annotator/data [POST]
annotator_webclient | Traceback (most recent call last):
annotator_webclient |   File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1813, in full_dispatch_request
annotator_webclient |     rv = self.dispatch_request()
annotator_webclient |   File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1799, in dispatch_request
annotator_webclient |     return self.view_functions[rule.endpoint](**req.view_args)
annotator_webclient |   File "/usr/local/lib/python3.6/site-packages/flask_restplus/api.py", line 325, in wrapper
annotator_webclient |     resp = resource(*args, **kwargs)
annotator_webclient |   File "/usr/local/lib/python3.6/site-packages/flask/views.py", line 88, in view
annotator_webclient |     return self.dispatch_request(*args, **kwargs)
annotator_webclient |   File "/usr/local/lib/python3.6/site-packages/flask_restplus/resource.py", line 44, in dispatch_request
annotator_webclient |     resp = meth(*args, **kwargs)
annotator_webclient |   File "/workspace/webserver/util/__init__.py", line 10, in wrap
annotator_webclient |     result = func(*args, **kwargs)
annotator_webclient |   File "/usr/local/lib/python3.6/site-packages/flask_login/utils.py", line 261, in decorated_view
annotator_webclient |     return func(*args, **kwargs)
annotator_webclient |   File "/workspace/webserver/api/annotator.py", line 113, in post
annotator_webclient |     paperjs_to_coco(width, height, paperjs_object)
annotator_webclient |   File "/workspace/webserver/util/coco_util.py", line 75, in paperjs_to_coco
annotator_webclient |     segments, image_height, image_width)
annotator_webclient |   File "/workspace/webserver/util/coco_util.py", line 82, in get_segmentation_area_and_bbox
annotator_webclient |     rles = mask.frPyObjects(segmentation, image_height, image_width)
annotator_webclient |   File "pycocotools/_mask.pyx", line 307, in pycocotools._mask.frPyObjects
annotator_webclient | Exception: input type is not supported.

[rbavery]

Open it and there should be an option to connect to the database. Assuming you are running this locally, you should be able to us the default settings to connect. Once connect go the the flask/annotation_models collection and type {_id: <ANNOTATION_ID>}. This should return the annotation

I tried these instructions to provide more info on why it's failing, but the default connection settings aren't working with mongodb compass (I'm running this locally on he production build). Do I need to supply authentication? My hostname is "localhost" and the port is 27019.

[jsbroks]

You shouldn't been authentication. Any errors?

[rbavery]

The error was ERR CONNECTION REFUSED when I used the defaults. and when I tried to use different ports and specify different ip addresses I saw from the prompt after running docker-compose up I was getting a different error, connection closed

[rbavery]

I've installed the latest version of mongodb compass on ubuntu 18.04 and I'm running the stable production build of coco-annotator, a fresh install from github. Here's what mongodb compass looks like when I try to connect with the defaults. Thanks for the help!

[rbavery]

my hunch is that something about my failing coco format is causing one of these failing conditions below, from coco_util.py. For example, maybe pyobj[0] is a dict without a counts or size key. Do I need a counts and/or size key for each segmentation or image? To what do these keys refer to exactly? What's the best way to inspect the segments return value of paperjs_to_coco https://github.com/jsbroks/coco-annotator/blob/master/backend/webserver/util/coco_util.py#L12?

def frPyObjects(pyobj, h, w):
    # encode rle from a list of python objects
    if type(pyobj) == np.ndarray:
        objs = frBbox(pyobj, h, w)
    elif type(pyobj) == list and len(pyobj[0]) == 4:
        objs = frBbox(pyobj, h, w)
    elif type(pyobj) == list and len(pyobj[0]) > 4:
        objs = frPoly(pyobj, h, w)
    elif type(pyobj) == list and type(pyobj[0]) == dict \
        and 'counts' in pyobj[0] and 'size' in pyobj[0]:
        objs = frUncompressedRLE(pyobj, h, w)
    # encode rle from single python object
    elif type(pyobj) == list and len(pyobj) == 4:
        objs = frBbox([pyobj], h, w)[0]
    elif type(pyobj) == list and len(pyobj) > 4:
        objs = frPoly([pyobj], h, w)[0]
    elif type(pyobj) == dict and 'counts' in pyobj and 'size' in pyobj:
        objs = frUncompressedRLE([pyobj], h, w)[0]
    else:
        raise Exception('input type is not supported.')
    return objs

If it helps to diagnose the issue, I looked at the format of coco json annotations I exported from scratch and compared them to the structure of the failing coco json that I created with pycocotools (converting from binary masks to cococ json). I can import these failing annotations to the annotator and edit them, but I cannot save due to the error above.

The top level keys of the working annotations are dict_keys(['images', 'categories', 'annotations']) vs the failing annotations dict_keys(['info', 'licenses', 'categories', 'images', 'annotations'])

and the keys for a single annotation for the working annotations are

good_coco_json.keys()
dict_keys(['id', 'image_id', 'category_id', 'segmentation', 'area', 'bbox', 'iscrowd', 'color', 'metadata'])

and for my failing annotations

bad_coco_json['annotations'][0].keys()
dict_keys(['id', 'image_id', 'category_id', 'iscrowd', 'area', 'bbox', 'segmentation', 'width', 'height'])

[rbavery]

The database connection error I was getting was because instead of localhost, I needed to supply the ip address of the mongodb container, which I got with → docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' <container id>

This is the annotation that I imported to coco:

I can add new annotations and save it, but I can't delete annotations and save it. When I delete an annotation, it disappears from the editor, but when I click the save button, I get the input type is not supported error and I can't leave the editor window. Also, the json entry in the mongdodb database is not updated with the change to the annotations, only "SessionEvent" items are added:

{
    "_id": 2,
    "dataset_id": 1,
    "category_ids": [1],
    "path": "/datasets/nebraska/LT050290312005031601T1-SC20190818204935_tile_1024_2560/image/LT050290312005031601T1-SC20190818204935_tile_1024_2560.jpeg",
    "width": 512,
    "height": 512,
    "file_name": "LT050290312005031601T1-SC20190818204935_tile_1024_2560.jpeg",
    "annotated": true,
    "annotating": [],
    "num_annotations": 27,
    "metadata": {},
    "deleted": false,
    "milliseconds": 181269,
    "events": [{
        "_cls": "SessionEvent",
        "user": "rbavery",
        "milliseconds": 27391,
        "tools_used": []
    }, {
        "_cls": "SessionEvent",
        "user": "rbavery",
        "milliseconds": 28673,
        "tools_used": []
    }, {
        "_cls": "SessionEvent",
        "user": "rbavery",
        "milliseconds": 29740,
        "tools_used": []
    }, {
        "_cls": "SessionEvent",
        "user": "rbavery",
        "milliseconds": 30952,
        "tools_used": []
    }, {
        "_cls": "SessionEvent",
        "user": "rbavery",
        "milliseconds": 31641,
        "tools_used": []
    }, {
        "_cls": "SessionEvent",
        "user": "rbavery",
        "milliseconds": 32872,
        "tools_used": []
    }],
    "regenerate_thumbnail": false,
    "is_modified": false
}

[rbavery]

I think I've found the root of the issue. There were line segments in my dataset at the image edges that were preventing me from saving and exporting all the edited annotations. also, when I would delete annotations with the eraser tool, sometimes a point would be leftover, causing their to be an invalid polygon that would also cause annotations to not be saved and would prevent leaving the annotation editor. The single point appears sometimes in the exact middle of the image after using the eraser tool, and it can't be deleted by the eraser tool, only by going to the annotation pane on the right hand side and clicking the delete icon after identifying the id of the point.

See below for an example. The center point causing the error is at [256,256], and this is created by the editor when deleting shapes with the eraser tool instead of the side panel's delete annotation icons.

[rbavery]

In the GIF above, there is also a mismatch between the hover box id of the newly created shape and the id on the right side panel.

[jsbroks]

Thank you for the great detail. Much appreciated

[chitraa]

I am not sure if the best way to do this is as below, so I didn't create a PR. But for now, I use this workaround to ignore exceptions to be raised when line segments or a single point is tried to be inserted into the database. The changes are in the file backend\webserver\util\coco_util.py

def get_segmentation_area_and_bbox(segmentation, image_height, image_width):
    # Convert into rle
    rles = mask.frPyObjects(segmentation, image_height, image_width)
    try:
        rles = mask.frPyObjects(segmentation, image_height, image_width)
    except Exception:
        print("Exception: input type is not supported.")
        print(segmentation)
    rle = mask.merge(rles)

    return mask.area(rle), mask.toBbox(rle)

I also noticed that small lines are created without closing the polygon while annotating really low resolution images (50x50 px). I avoided this by reducing the completeDistance & minDistance hyperparameters in PolygonTool.vue

Here is a patch file of the changes.

[zeeshanalipanhwar]

This error can also occur if the segmentation mask has only two values, ie if it is only one pixel / point.

[ArkAung]

The error occurs for annotations which have additional one pixel annotation that the user does not draw. This point annotation usually occurs in the middle of the image. If you remove this annotation, your issue will be solved. However, I am not sure what is the cause of this problem and how we can properly solve it yet. Maybe this PR #250 would have solved the problem. The way we figured out the problem was to look for the bad behaving annotation_id (by logging the annotation_id if there is an exception in api/annotator.py at this point https://github.com/jsbroks/coco-annotator/blob/c35dba220535e8552f60db310efc7e70e601d32e/backend/webserver/api/annotator.py#L118 Then checking the logs of web_client docker container) and removing it in the image.

[LuisFelipeLeivaH]

I solved the issue in my case. As I transformed a dataset with bitmap masks into COCO (instance segmentation), I found that on the segmentations there were empty lists of the form "segmentation" : [[something], []]. Sometimes there were more than one empty list. I removed all the "[]" and then I could run my evaluation using mmdetection.