Several jobs for one segment

nmanovic commented 6 years ago

In some tasks it is necessary to annotate the same segment several times. In general it can be important for "subjective" tasks (age, race, etc...). Thus for the same object we will have several annotations and it will be possible to vote for the best variant.

[ ] It should be possible to create a task with several jobs for one segment. Zero job is master. It is used to dump annotations always.
[ ] It should be possible to update the master job using annotations from other jobs for the same segment (automatic mode). Internally attributes should be "voted", shapes should be merged (for bounding boxes - weighting, for polygons - join points and smooth shape, for points - weighting). All ambiguous should be resolved randomly.
[ ] Manual review mode with ability to resolve all conflicts manually. For example, if in one job an object is absent it should have a specific color and hint; if in a job an object has different shapes it should be highlighted somehow with ability to choose a correct variant or draw a new one.
[ ] Think how to extend the filter feature to support manual review mode (e.g. show only ambiguous).

bsekachev commented 5 years ago

There are abilities:

Add jobs after creating
Cut jobs after creating

Also: Master branch is used for download annotation, upload annotation and auto annotation.

martin-etchart commented 4 years ago

I think a simplified version of this feature could prove to be useful. Without the annotation merging and without the manual review, just by identifying which annotation comes from which job (or who was the assigned job annotator) for a given image. Annotation merging or consensus is sometimes specific for the labeling problem itself and just plain voting or shape merging is not enough or can be inadequate. I would prefer to have the raw annotations and post process them to a custom consensus or merged label by myself.

I did try some version of this a while ago with a task with multiple jobs with the same segment by hardcoding that tasks be created with 3 jobs using the same (full) segment: https://github.com/martin-etchart/cvat/tree/feature/same_segment_jobs

It worked in practice but on dump, annotations from the 3 jobs where not properly identified. If I labeled an image on the 3 jobs with 1 box each in each job for example, the dump would give me:

  <image id="9" name="00009.png" width="128" height="128">
    <box label="Label" occluded="0" xtl="0.00" ytl="0.00" xbr="128.00" ybr="128.00">
      <attribute name="Attribute">Value0</attribute>
    </box>
    <box label="Label" occluded="0" xtl="0.00" ytl="0.00" xbr="128.00" ybr="128.00">
      <attribute name="Attribute">Value1</attribute>
    </box>
    <box label="Label" occluded="0" xtl="0.00" ytl="0.00" xbr="128.00" ybr="128.00">
      <attribute name="Attribute">Value2</attribute>
    </box>
  </image>

One possibility would be that you could dump the xml of a job by itself and then have them merged by your own custom script. The other possibility would be to identify the annotations in the xml by job and/or job assignee, something like,

<box label="Label" job="<job_id>" occluded="0" xtl="0.00" ytl="0.00" xbr="128.00" ybr="128.00">
      <attribute name="Attribute">Value1</attribute>
</box>

Maybe this kid of tasks with multiple same segment jobs should dump annotations differently or have an option for raw annotations or, in case it gets implemented in the future, with the automatic annotation merging and/or manual review of labels. In this case the job 0 or master could be the result of this.

nmanovic commented 4 years ago

@martin-etchart , why tasks don't work in your case? You can create several tasks with the same images and labels, annotate them and dump 3 independent annotations.

martin-etchart commented 4 years ago

Yes they do work, we are doing exactly that at the moment. Just my two cents if it where to be implemented as an option when creating a task.

An argument against that would be storage, in case of a highly subjective task for example you would need to create as many tasks as annotators and duplicate images. But that's not an issue for us now.

(Is there a way to link images directly to a task without the upload/copy and format conversion? We are usually using the share option via de CLI)

faustorbit commented 3 years ago

@martin-etchart , why tasks don't work in your case? You can create several tasks with the same images and labels, annotate them and dump 3 independent annotations.

This is not a solution.

cvat-ai / cvat

Several jobs for one segment #125