Could not export dataset (Request Status Code 500 "StopIteration") when including images.

luisbc92 commented 2 years ago

My actions before raising this issue

[X] Read/searched the docs
[X] Searched past issues

Expected Behaviour

Exporting a dataset using "Export task dataset" or "Export project dataset" with images should work.

Current Behaviour

Using either "Export task dataset" or "Export project dataset" on any format and checking "Save images" throws the following error:

Exporting the dataset without images does work properly. Exporting the task also works.

Context

The images are selected from a "Connected file share". Annotation works properly and all images show up correctly. I have been able to export datasets in the past. This particular task is the one that fails.

Let me know if I'm missing information!

Your Environment

Git hash commit (git log -1): 483e4f6a586471d4a64c6d2f579f7d9e50f1908c
Docker version docker version (e.g. Docker 17.0.05): Docker 20.10.7
Are you using Docker Swarm or Kubernetes? No
Operating System and version (e.g. Linux, Windows, MacOS): Linux

nmanovic commented 2 years ago

@luisbc92 , do you have any robust way to reproduce the problem?

luisbc92 commented 2 years ago

I think I may have found the problem. It occurs whenever a subset of images is selected with "Start Frame" and "End Frame":

Interestingly, in some instances it throws "StopIteration" while in some others it throws "No Image Found".

EDIT: By instances I mean repeated attempts at reproducing the problem the same way.

nmanovic commented 2 years ago

@luisbc92 , thanks. We will try to reproduce the issue and fix in future releases.

yasakova-anastasia commented 1 year ago

It's been fixed by https://github.com/cvat-ai/cvat-old/pull/17

pktiuk commented 1 year ago

I think this issue should be reopened.

It still happens with CVAT 3.4.5 But maybe the root cause is different.

yasakova-anastasia commented 1 year ago

@pktiuk, could you please provide steps to reproduce this issue in your case?

pktiuk commented 1 year ago

Open the task site https://mycvatinstance.com/tasks/817
Press Actions-> Export task dataset
Switch ON save images switch
Wait 2-4 minutes
See the error

Note: It happens mainly with big tasks (1920x1080 over 3000 segment size)

pktiuk commented 1 year ago

Let me know if you need any additional logs or help from my side.

user-log1n commented 11 months ago

Hi @nmanovic, @yasakova-anastasia

The problem (inability to export annotations with images for some tasks) is still unresolved: cvat version 2.7.6.

In my case, it's a task created from a video in mp4 format (video size ~ 305MB, frame count ~ 2500, video resolution: 1920x1080).

A few minutes after generating a request to export annotations in the "Cvat for video 1.1" format with images (Menu -> Export job dataset -> Export format: Cvat for video 1.1 + checked: Save images) cvat displays an error message:

Could not export dataset for the job 1543
Error: Request failed with status code 500. 
"RuntimeError: generator raised StopIteration\n".

The Chrome browser console error messages:

cvat-ui.7193fe59a9cb22db14c8.min.js:2 Error: Request failed with status code 500. "Traceback (most recent call last):\n  File \"/home/django/cvat/apps/engine/frame_provider.py\", line 205, in get_frames\n    yield self.get_frame(idx, quality=quality, out_type=out_type)\n  File \"/home/django/cvat/apps/engine/frame_provider.py\", line 196, in get_frame\n    frame, frame_name, _ = chunk_reader[frame_offset]\n  File \"/home/django/cvat/apps/engine/frame_provider.py\", line 40, in __getitem__\n    v = next(self.iterator)\nStopIteration\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"/opt/venv/lib/python3.10/site-packages/rq/worker.py\", line 936, in perform_job\n    rv = job.perform()\n  File \"/opt/venv/lib/python3.10/site-packages/rq/job.py\", line 684, in perform\n    self._result = self._execute()\n  File \"/opt/venv/lib/python3.10/site-packages/rq/job.py\", line 690, in _execute\n    return self.func(*self.args, **self.kwargs)\n  File \"/home/django/cvat/apps/dataset_manager/views.py\", line 115, in export_job_as_dataset\n    return export(dst_format, job_id=job_id, server_url=server_url, save_images=True)\n  File \"/home/django/cvat/apps/dataset_manager/views.py\", line 85, in export\n    export_fn(db_instance.id, temp_file, dst_format,\n  File \"/home/django/cvat/apps/dataset_manager/task.py\", line 855, in export_job\n    job.export(f, exporter, host=server_url, save_images=save_images)\n  File \"/home/django/cvat/apps/dataset_manager/task.py\", line 685, in export\n    exporter(dst_file, temp_dir, job_data, **options)\n  File \"/home/django/cvat/apps/dataset_manager/formats/registry.py\", line 36, in __call__\n    f_or_cls(*args, **kwargs)\n  File \"/home/django/cvat/apps/dataset_manager/formats/cvat.py\", line 1422, in _export_video\n    _export_task_or_job(dst_file, temp_dir, instance_data,\n  File \"/home/django/cvat/apps/dataset_manager/formats/cvat.py\", line 1397, in _export_task_or_job\n    dump_media_files(instance_data, osp.join(temp_dir, 'images'))\n  File \"/home/django/cvat/apps/dataset_manager/formats/cvat.py\", line 1381, in dump_media_files\n    for frame_id, (frame_data, _) in zip(instance_data.rel_range, frames):\nRuntimeError: generator raised StopIteration\n".

CVAT cvat_worker_export container logs:

2023-10-19 10:44:50,759 DEBG 'rqworker-export-1' stderr output:
[2023-10-19 10:44:50,759] INFO cvat.apps.engine.cache: Ending to prepare chunk: key 1078_34_Quality.ORIGINAL

2023-10-19 10:44:52,650 DEBG 'rqworker-export-1' stderr output:
[2023-10-19 10:44:52,650] ERROR cvat.apps.dataset_manager.views: [Job.id=1543] [cvat.apps.dataset_manager.views @ export]: exception occurred
Traceback (most recent call last):
  File "/home/django/cvat/apps/engine/frame_provider.py", line 205, in get_frames
    yield self.get_frame(idx, quality=quality, out_type=out_type)
  File "/home/django/cvat/apps/engine/frame_provider.py", line 196, in get_frame
    frame, frame_name, _ = chunk_reader[frame_offset]
  File "/home/django/cvat/apps/engine/frame_provider.py", line 40, in __getitem__
    v = next(self.iterator)
StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/django/cvat/apps/dataset_manager/views.py", line 85, in export
    export_fn(db_instance.id, temp_file, dst_format,
  File "/home/django/cvat/apps/dataset_manager/task.py", line 855, in export_job
    job.export(f, exporter, host=server_url, save_images=save_images)
  File "/home/django/cvat/apps/dataset_manager/task.py", line 685, in export
    exporter(dst_file, temp_dir, job_data, **options)
  File "/home/django/cvat/apps/dataset_manager/formats/registry.py", line 36, in __call__
    f_or_cls(*args, **kwargs)
  File "/home/django/cvat/apps/dataset_manager/formats/cvat.py", line 1422, in _export_video
    _export_task_or_job(dst_file, temp_dir, instance_data,
  File "/home/django/cvat/apps/dataset_manager/formats/cvat.py", line 1397, in _export_task_or_job
    dump_media_files(instance_data, osp.join(temp_dir, 'images'))
  File "/home/django/cvat/apps/dataset_manager/formats/cvat.py", line 1381, in dump_media_files
    for frame_id, (frame_data, _) in zip(instance_data.rel_range, frames):
RuntimeError: generator raised StopIteration

2023-10-19 10:44:52,651 DEBG 'rqworker-export-1' stderr output:
[2023-10-19 10:44:52,651] DEBUG rq.worker: Handling failed execution of job export:annotations-for-job.id1543-in-CVAT_for_video_1.1-format

2023-10-19 10:44:54,981 DEBG 'rqworker-export-1' stderr output:
[2023-10-19 10:44:54,981] ERROR rq.worker: Traceback (most recent call last):
  File "/home/django/cvat/apps/engine/frame_provider.py", line 205, in get_frames
    yield self.get_frame(idx, quality=quality, out_type=out_type)
  File "/home/django/cvat/apps/engine/frame_provider.py", line 196, in get_frame
    frame, frame_name, _ = chunk_reader[frame_offset]
  File "/home/django/cvat/apps/engine/frame_provider.py", line 40, in __getitem__
    v = next(self.iterator)
StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/rq/worker.py", line 936, in perform_job
    rv = job.perform()
  File "/opt/venv/lib/python3.10/site-packages/rq/job.py", line 684, in perform
    self._result = self._execute()
  File "/opt/venv/lib/python3.10/site-packages/rq/job.py", line 690, in _execute
    return self.func(*self.args, **self.kwargs)
  File "/home/django/cvat/apps/dataset_manager/views.py", line 115, in export_job_as_dataset
    return export(dst_format, job_id=job_id, server_url=server_url, save_images=True)
  File "/home/django/cvat/apps/dataset_manager/views.py", line 85, in export
    export_fn(db_instance.id, temp_file, dst_format,
  File "/home/django/cvat/apps/dataset_manager/task.py", line 855, in export_job
    job.export(f, exporter, host=server_url, save_images=save_images)
  File "/home/django/cvat/apps/dataset_manager/task.py", line 685, in export
    exporter(dst_file, temp_dir, job_data, **options)
  File "/home/django/cvat/apps/dataset_manager/formats/registry.py", line 36, in __call__
    f_or_cls(*args, **kwargs)
  File "/home/django/cvat/apps/dataset_manager/formats/cvat.py", line 1422, in _export_video
    _export_task_or_job(dst_file, temp_dir, instance_data,
  File "/home/django/cvat/apps/dataset_manager/formats/cvat.py", line 1397, in _export_task_or_job
    dump_media_files(instance_data, osp.join(temp_dir, 'images'))
  File "/home/django/cvat/apps/dataset_manager/formats/cvat.py", line 1381, in dump_media_files
    for frame_id, (frame_data, _) in zip(instance_data.rel_range, frames):
RuntimeError: generator raised StopIteration
Traceback (most recent call last):
  File "/home/django/cvat/apps/engine/frame_provider.py", line 205, in get_frames
    yield self.get_frame(idx, quality=quality, out_type=out_type)
  File "/home/django/cvat/apps/engine/frame_provider.py", line 196, in get_frame
    frame, frame_name, _ = chunk_reader[frame_offset]
  File "/home/django/cvat/apps/engine/frame_provider.py", line 40, in __getitem__
    v = next(self.iterator)
StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/rq/worker.py", line 936, in perform_job
    rv = job.perform()
  File "/opt/venv/lib/python3.10/site-packages/rq/job.py", line 684, in perform
    self._result = self._execute()
  File "/opt/venv/lib/python3.10/site-packages/rq/job.py", line 690, in _execute
    return self.func(*self.args, **self.kwargs)
  File "/home/django/cvat/apps/dataset_manager/views.py", line 115, in export_job_as_dataset
    return export(dst_format, job_id=job_id, server_url=server_url, save_images=True)
  File "/home/django/cvat/apps/dataset_manager/views.py", line 85, in export
    export_fn(db_instance.id, temp_file, dst_format,
  File "/home/django/cvat/apps/dataset_manager/task.py", line 855, in export_job
    job.export(f, exporter, host=server_url, save_images=save_images)
  File "/home/django/cvat/apps/dataset_manager/task.py", line 685, in export
    exporter(dst_file, temp_dir, job_data, **options)
  File "/home/django/cvat/apps/dataset_manager/formats/registry.py", line 36, in __call__
    f_or_cls(*args, **kwargs)
  File "/home/django/cvat/apps/dataset_manager/formats/cvat.py", line 1422, in _export_video
    _export_task_or_job(dst_file, temp_dir, instance_data,
  File "/home/django/cvat/apps/dataset_manager/formats/cvat.py", line 1397, in _export_task_or_job
    dump_media_files(instance_data, osp.join(temp_dir, 'images'))
  File "/home/django/cvat/apps/dataset_manager/formats/cvat.py", line 1381, in dump_media_files
    for frame_id, (frame_data, _) in zip(instance_data.rel_range, frames):
RuntimeError: generator raised StopIteration

The mp4 video file associated with the task can be easily exported via Actions -> Backup Task

Downloaded video can be easily split into individual PNG images/frames when using ffmpeg on a local computer, which excludes the possibility that the cause of the StopIteration error in cvat on server is corrupted video file.

There is no problem with retrieving only the xml annotations without images (unchecked option Save images) for this task.

pktiuk commented 10 months ago

@nmanovic @yasakova-anastasia
Have you reproduced this?

cvat-ai / cvat