search

microsoft / aerial_wildlife_detection

Tools for detecting wildlife in aerial images using active learning
MIT License
226 stars 58 forks source link

extracting/interpreting annotations from AIDE #17

Closed mikeyEcology closed 4 years ago

mikeyEcology commented 4 years ago

This software is great and I'm excited to get using it! I have a few questions about extracting data; I want to ensure that I get this sorted out before having the data labeled.

  1. Is there documentation in how to understand data that are downloaded from AIDE? I don't know how to interpret what I get in the download file aide_quiery_[date/time] (pasted below).
  2. In my semantic segmentation project, the download included segmentation masks, but when I tried to view them they appeared to be blank (even though I annotated the images). Although it's possible that I'm not viewing them properly.
  3. Is it possible to control how the segmentation masks will be downloaded? For example, if I have images named image1.jpg, image2.jpg, would it be possible to have the masks downloaded as image1_mask.jpg, image2_mask.jpg. Also is it possible to save the masks as jpeg files instead of tif files?

Contents of aide_query file:

id; image; meta; label; username; autoConverted; timeCreated; timeRequired; unsure; username; viewcount; last_checked; last_time_required c604558b-ab27-47e9-a532-56726a0a821c; ed79e933-a83a-4c40-9269-9f75c6cf0b03; {"browser": {"vendorSub": "", "productSub": "20030107", "vendor": "Google Inc.", "maxTouchPoints": 0, "hardwareConcurrency": 16, "cookieEnabled": true, "appCodeName": "Mozilla", "appName": "Netscape", "appVersion": "5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36", "platform": "Win32", "product": "Gecko", "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36", "language": "en-US", "languages": ["en-US", "en"], "onLine": true, "doNotTrack": null, "geolocation": {}, "mediaCapabilities": {}, "connection": {}, "plugins": {"0": {"0": {}}, "1": {"0": {}}, "2": {"0": {}, "1": {}}}, "mimeTypes": {"0": {}, "1": {}, "2": {}, "3": {}}, "webkitTemporaryStorage": {}, "webkitPersistentStorage": {}, "userActivation": {}, "mediaSession": {}, "permissions": {}}, "windowSize": [2560, 1281], "uiControls": {"burstModeEnabled": false}}; 1ab5e9a8-c08c-11ea-bad2-0242ac160002; admin; False; 2020-07-08 16:57:31.985000+01:00; 493071; True; admin; 6; 2020-07-08 16:57:31.985000+01:00; 19283;

bkellenb commented 4 years ago

Hi Mikey,

Thank you very much for the questions! I will try and answer them below:

  1. The main file in the data download ("aide_query") is a text file with user annotations; one per line and with tokens separated by a semicolon (basically like a CSV file, but as said ";"-separated). The first line is the header and contains the field names for all lines to follow. In your example, these are in order:

    • id: unique annotation identifier (UUID)
    • image: unique image identifier (UUID)
    • meta: JSON-encoded web browser metadata
    • label: unique label class identifier (UUID)
    • username: creator of the annotation (in your case: "admin")
    • autoConverted: whether the annotation was originally an AI model prediction that got converted into an annotation (False in your case)
    • timeCreated: date and time when the annotation was made by the user (in your case: 2020-07-08 16:57:31.985000+01:00)
    • timeRequired: number of milliseconds from first seeing the image on screen until creating the annotation (493071 ms in your case)
    • unsure: whether the annotation was flagged as "unsure" by the user (True in your case)
    • viewcount: number of times the image was viewed by the user (six times in your case)
    • last_checked: date and time when the image was last viewed (2020-07-08 16:57:31.985000+01:00)
    • last_time_required: number of milliseconds the image was on screen the last time the user checked (19283 ms)

  2. The segmentation masks are stored as TIFF files with indexed color: for example, if the mask contains pixels annotated as class with index number 4, those pixels will contain the actual value 4, and not the color assigned to the class. I will add a flag that allows users to decide whether they want indexed or real colors for simplicity.

  3. The segmentation mask name is a good suggestion, thank you! I might add a text field that allows users to add a default prefix (or suffix) to the file name. Regarding the file format, I advise against using JPEG. The reason is that JPEG uses a lossy compression format (i.e., it reduces the file size, but alters the pixel values to do so). The result is artifacts on contrasting edges (transitions between label classes), such as blurriness or even interpolation values between label class colors, which has detrimental effects on the quality of the segmentation mask. For this reason, AIDE exports all masks as TIFF files. If you wish to still convert them to JPEGs, the easiest solution at the moment is to employ a third-party batch conversion tool.

Thanks a lot once more for your comments and suggestions! The data download function is still work in progress and will improve in functionality and also documentation over time.

mikeyEcology commented 4 years ago

Thank you so much for your detailed explanations! I have a few followup questions regarding the aide_query file. Sorry if the answers are obvious, but maybe others will have the same questions. I'll go back through the points by number:

  1. Thank you for all of the details. These categories make sense, but there are still a couple I don't understand. For image, how does link up with an image in my dataset? For my example, how would I find what filename is associated with this image "ed79e933-a83a-4c40-9269-9f75c6cf0b03"? Likewise, how do I find what is meant by the label "1ab5e9a8-c08c-11ea-bad2-0242ac160002"? How is this label related to an image class (e.g., animal species)?

  2. This makes perfect sense. I was at first concerned because I've never used TIF for this type of stuff and my image viewing software didn't pick anything up in the image. But I was able to open the segmentation masks in Python and they work well. I think it makes sense to use numbers for the masks. One utility that might help is getting to choose which number is used for each class. This could help users who are combining with another dataset.

  3. Thank you. If it's easier I'm less concerned about specifying the name of the mask as I am about ensuring that I understand what image it is associated with.

bkellenb commented 4 years ago

Hello Mikey,

The latest version of AIDE now features the following improvements and new functionalities:

  1. More comprehensive querying: in addition to the bare image and label class UUIDs, AIDE now also appends the label class names to each exported annotation, resp. prediction. In the case of segmentation masks, it now exports a dedicated comma-separated file ("labelclasses.csv"), which contains the label class IDs ("id"), names ("name"), assigned colors ("color"), parent group ("labelclassgroup") and, importantly, the index value in the segmentation mask TIFF files ("labelclass_index"). So if your label class "elephant" has a "labelclass_index" value of 5, all pixels marked as "elephant" will contain the value 5.

  2. Customizable formatting: it is now possible to exclude bulky query fields (the browser metadata) by unticking a checkbox in the "Data Download" page. For segmentation masks, AIDE now offers an option to save the TIFF files with the same name (and folder structure) as the original images instead of their UUIDs. Segmentation mask names can further be customized with user-specified prefix and suffix strings.

In sum, you now get the plain label class name as a token for every annotation/prediction, and you can now export segmentation masks with the same (or similar) name like their associated original images.

mikeyEcology commented 4 years ago

Thank you so much for making these modifications! It will make it a lot easier for me to work with it.