Cannot export task (AssertionError) Due to invalid tracks

pktiuk commented 1 year ago

My actions before raising this issue

[x] Read/searched the docs
[x] Searched past issues

Steps to Reproduce (for bugs)

Some of our tasks cannot be exported,

Open Job
Press export
Select format (CVAT, for video, yolo etc, it doesn't matter)
Accept

Expected Behaviour

Export

Current Behaviour

when we try to export them we get an error 500: AssertionError

Possible Solution

🤷

Context

Some of affected tasks could be downloaded 2 days ago, but now they cannot be exported.
I can still open them and watch their content.

Your Environment

Git hash commit (git log -1):
Docker version docker version (e.g. Docker 17.0.05):
Are you using Docker Swarm or Kubernetes?

Selfhosted CVAT version 2.6.1

Logs:

2023-09-01 07:47:00,461 DEBG 'rqworker-export-0' stderr output:
ERROR:rq.worker:Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/rq/worker.py", line 936, in perform_job
    rv = job.perform()
  File "/opt/venv/lib/python3.10/site-packages/rq/job.py", line 684, in perform
    self._result = self._execute()
  File "/opt/venv/lib/python3.10/site-packages/rq/job.py", line 690, in _execute
    return self.func(*self.args, **self.kwargs)
  File "/home/django/cvat/apps/dataset_manager/views.py", line 110, in export_job_annotations
    return export(dst_format,job_id=job_id, server_url=server_url, save_images=False)
  File "/home/django/cvat/apps/dataset_manager/views.py", line 83, in export
    export_fn(db_instance.id, temp_file, dst_format,
  File "/home/django/cvat/apps/dataset_manager/task.py", line 855, in export_job
    job.export(f, exporter, host=server_url, save_images=save_images)
  File "/home/django/cvat/apps/dataset_manager/task.py", line 685, in export
    exporter(dst_file, temp_dir, job_data, **options)
  File "/home/django/cvat/apps/dataset_manager/formats/registry.py", line 36, in __call__
    f_or_cls(*args, **kwargs)
  File "/home/django/cvat/apps/dataset_manager/formats/cvat.py", line 1422, in _export_video
    _export_task_or_job(dst_file, temp_dir, instance_data,
  File "/home/django/cvat/apps/dataset_manager/formats/cvat.py", line 1394, in _export_task_or_job
    dump_task_or_job_anno(f, instance_data, anno_callback)
  File "/home/django/cvat/apps/dataset_manager/formats/cvat.py", line 1362, in dump_task_or_job_anno
    callback(dumper, instance_data)
  File "/home/django/cvat/apps/dataset_manager/formats/cvat.py", line 1049, in dump_as_cvat_interpolation
    for track in annotations.tracks:
  File "/home/django/cvat/apps/dataset_manager/bindings.py", line 484, in tracks
    yield self._export_track(track, idx)
  File "/home/django/cvat/apps/dataset_manager/bindings.py", line 369, in _export_track
    tracked_shapes = TrackManager.get_interpolated_shapes(
  File "/home/django/cvat/apps/dataset_manager/annotation.py", line 912, in get_interpolated_shapes
    assert curr_frame > prev_shape["frame"] # Catch invalid tracks
AssertionError
Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/rq/worker.py", line 936, in perform_job
    rv = job.perform()
  File "/opt/venv/lib/python3.10/site-packages/rq/job.py", line 684, in perform
    self._result = self._execute()
  File "/opt/venv/lib/python3.10/site-packages/rq/job.py", line 690, in _execute
    return self.func(*self.args, **self.kwargs)
  File "/home/django/cvat/apps/dataset_manager/views.py", line 110, in export_job_annotations
    return export(dst_format,job_id=job_id, server_url=server_url, save_images=False)
  File "/home/django/cvat/apps/dataset_manager/views.py", line 83, in export
    export_fn(db_instance.id, temp_file, dst_format,
  File "/home/django/cvat/apps/dataset_manager/task.py", line 855, in export_job
    job.export(f, exporter, host=server_url, save_images=save_images)
  File "/home/django/cvat/apps/dataset_manager/task.py", line 685, in export
    exporter(dst_file, temp_dir, job_data, **options)
  File "/home/django/cvat/apps/dataset_manager/formats/registry.py", line 36, in __call__
    f_or_cls(*args, **kwargs)
  File "/home/django/cvat/apps/dataset_manager/formats/cvat.py", line 1422, in _export_video
    _export_task_or_job(dst_file, temp_dir, instance_data,
  File "/home/django/cvat/apps/dataset_manager/formats/cvat.py", line 1394, in _export_task_or_job
    dump_task_or_job_anno(f, instance_data, anno_callback)
  File "/home/django/cvat/apps/dataset_manager/formats/cvat.py", line 1362, in dump_task_or_job_anno
    callback(dumper, instance_data)
  File "/home/django/cvat/apps/dataset_manager/formats/cvat.py", line 1049, in dump_as_cvat_interpolation
    for track in annotations.tracks:
  File "/home/django/cvat/apps/dataset_manager/bindings.py", line 484, in tracks
    yield self._export_track(track, idx)
  File "/home/django/cvat/apps/dataset_manager/bindings.py", line 369, in _export_track
    tracked_shapes = TrackManager.get_interpolated_shapes(
  File "/home/django/cvat/apps/dataset_manager/annotation.py", line 912, in get_interpolated_shapes
    assert curr_frame > prev_shape["frame"] # Catch invalid tracks
AssertionError

zhiltsov-max commented 1 year ago

Hi, could you please share a backup of the task? Please remove any images and other personal info from the archive.

user-log1n commented 1 year ago

Hi @zhiltsov-max I work together with @pktiuk
Unluckily, I am not sure whether we can share full backup of selected task, because it may contain some sensitive informations for my company.
Could you tell us what we should focus on to debug this issue?

zhiltsov-max commented 1 year ago

As you see, the check that fails has been added specifically to catch invalid tracks. It's one of the problems we're trying to reproduce, but could not do it reliably. I suppose there can be some unexpected frame sequence in the track annotations, so you should check the frame ids and the outside property.

user-log1n commented 1 year ago

Thank you for your suggestion. I will try to prepare an full anonymized task (without sensitive informations) that reproduces the error. Indeed in our annotation tasks we use often outside property and we merge objects that changes object ids.

user-log1n commented 1 year ago

I was unable to reproduce the bug in the new task, so I will try to anonymize one of the existing tasks causing the error.

user-log1n commented 1 year ago

I am sending an example of anonymized two tasks in which an export error occurred.

Example 1: When you export annotations from a task to an xml file and then load them unchanged into the same task or new task, an error occurs when you try to export annotations. In the original task before the export of the annotations, the number of tracked objects is 540. Info -> Annotations Statistics -> Rectangle -> Total = 540 After reloading unchanged xml file into the task the number of tracked objects is 538. Info -> Annotations Statistics -> Rectangle -> Total = 538

Example 2: In this case, before exporting and after reloading the annotation to the task, the number of objects is the same 19. Info -> Annotations Statistics -> Rectangle -> Total = 19

example-1.zip example-2.zip

pktiuk commented 1 year ago

@zhiltsov-max
Could you look at these backups? I think this issue should also get a bug label.

user-log1n commented 1 year ago

Hi,

Here's one way to reproduce an AssertionError in any cvat object tracking task:

select any frame (e.g. frame number 3) in a task in which there are already some marked objects
create a new tracked object in the selected frame (track option)
(this step can be skipped) we click forward a few frames and switch the outside property for this object to the state so that the object is not visible
go back to the frame where the object was tagged for the first time (here the 3rd frame) and also set the outside property for this object to the state so that the object is not visible
save the task - download annotations in the format "Cvat for video 1.1" (annotations.xml file)
upload the downloaded annotations into the task from which they were previously downloaded
when you try to export the annotation again, you will get an AssertionError

The reason for the AssertionError when trying to export annotations is that the cvat in the originally exported file places tracks for objects where the outside property is checked on the first frame on which the object first appeared (or on all frames for a given object). After removing all such tracks (the object on the first frame, where it was created, is invisible, i.e. outside="1") from the annotations and uploading them into cvat, the problem with AssertionError disappears.

If anyone needs it, they can fix their xml annotations (format xml "Cvat for video 1.1") with a python3 script that I wrote. The script checks the annotations.xml input file for problematic tracks and removes them. It then reindex the tracks that are left and writes the results to the annotations-mod.xml file.

Cvat-AssertionError-XML-Validate.zip

zhiltsov-max commented 6 months ago

Hi, probably I've got some understanding about possible causes of the problem. There is the function _slice_track(), which got several edits over time as the project evolved, and currently (for quite a time, though) it can potentially produce several shapes on the same frame, as it first just copies shapes related to the current segments, and then adds some interpolated shapes without checking if the frame of the added scoped shape is already included in segment_shapes. Probably, in most cases the shapes are just duplicated this way (unless they were actually different in the saved/imported annotations). Later, this can lead to the failure in the interpolation code.

The problem can be obtained during annotation saving, import or export.

I think, the proper solution would be:

to avoid adding the shapes on the same frame in this function and add a check on the annotation saving for track shapes with the same frame. If the shape is a duplicate - just skip it, otherwise raise an error. This will prevent the problem for benign cases in the future. Malignant cases will report an error, which can be fixed on the user side
to ignore the duplicated shapes in the interpolation function (but still fail if the shapes are different). There can already be some duplicated track shapes in the CVAT DB, so this will just fix the reading for benign cases. Malignant cases will require need some investigation.

PMazarovich commented 3 weeks ago

Hello, @Eldies , @zhiltsov-max , just wanted to clarify are there any estimations on when this is expected to be resolved? Thanks!

cvat-ai / cvat