[FR] load annotations with specific status

[mmoollllee]

Proposal Summary

Adding an annotations job with segement_size to CVAT, makes me want to load those segments with e.g. CVAT Job Stage "validation" back into FiftyOne to validate the progress.

Motivation

• As I add annotation jobs with predictions to CVAT, where those predictions become ground_truth annotations, I don't want to load the partial progress into the new dest_field. It helps me to validate progress in FiftyOne instead of CVAT, and maybe use this progress for training already.
• There's currently no option to filter load_annotations

What areas of FiftyOne does this feature affect?

• [ ] App: FiftyOne application
• [x] Core: Core fiftyone Python library
• [ ] Server: FiftyOne server

Details

I'm adding annotation jobs to CVAT like this:

view.annotate(
    anno_key,
    label_field="bad_predictions",
    segment_size=50
)

and I'd like to have a filter_jobs key for load_annotations, e.g.:

dataset.load_annotations(
    anno_key, 
    dest_field="ground_truth", 
    unexpected="keep",
    filter_jobs="validation" # only import CVAT jobs with "stage: validation"
)

Willingness to contribute

The FiftyOne Community welcomes contributions! Would you or another member of your organization be willing to contribute an implementation of this feature?

• [ ] Yes. I can contribute this feature independently
• [x] Yes. I would be willing to contribute this feature with guidance from the FiftyOne community
• [ ] No. I cannot contribute this feature at this time

[brimoor]

@mmoollllee this feature request makes sense to me 👍

In order to implement this, you'd need to update the download_annotations() interface to allow for the possibility of passing backend-specific **kwargs to customize how the annotations are downloaded.

Then, since the load_annotations() method is generic (supports multiple backends), you'd need to check its **kwargs and pull out any download parameters for the specific backend being used.

We have an extract_kwargs_for_function() utility that can do that 👍

Let me know if this makes sense and happy to provide additional guidance if I can be of assistance 🤗

[mmoollllee]

Happy to hear the feature makes sense :))

Puhh I digged around the provided code parts and need to accept that I'm totally lost in this advanced coding architecture... I guess this is beyond my abilities :( Maybe I'll give it another try and a lot of testing... Otherwise it seems like somebody else needs to implement this. Sorry :(

[brimoor]

I believe in you! If you're able to add your parameter only in this location, then we can help complete the feature for you 🤗

https://github.com/voxel51/fiftyone/blob/7853d085f398d9d2d47e356da822f65f725d98db/fiftyone/utils/cvat.py#L4562

[mmoollllee]

Thanks for your pushing me!! :)

See my current status in this commit

• [ ] Get argument job_status. No progress with this, but I tried this:

  fou.extract_kwargs_for_function(foua.load_annotations, self.kwargs)

  but when I add a argument to .load_annotations I get TypeError: load_credentials() got an unexpected keyword argument 'job_status'. Don't know what to do :(

• [x] If job_status is one of CVAT job_status build a list of frames to include:

              # If job_status is set and is one of CVAT's status'
              frames_to_download = []
              if job_status in ["annotation", "validation", "acceptance"]:
                  for job_id in job_ids[task_id]:
                      # Get job_status
                      job_url = self.taskless_job_url(job_id)
                      try:
                          job_resp = self.get(job_url).json()
                      except:
                          logger.warning(
                              "Couldn't fetch job information. Skipping"
                          )
                          continue
  
                      if not job_resp["status"] == job_status:
                          logger.info(
                              "Skipping job %s because of status mismatch",
                              job_id,
                          )
                          continue
  
                      frames_to_download += range(job_resp["start_frame"], job_resp["stop_frame"])

• [x] After fetching data_resp filter shapes, tags & tracks to match only frames_to_download:

              if frames_to_download:
                  all_shapes = [x for x in all_shapes if x['frame'] in frames_to_download]
                  all_tags = [x for x in all_tags if x['frame'] in frames_to_download]
                  all_tracks = [x for x in all_tracks if x['frame'] in frames_to_download]

I have to say I'm not really confident in what I did. For example I never worked with videos in CVAT. Will this still work for videos?

I'd be happy to get feedback and another hint on how to pass job_status from dataset.load_annotations!