search

cvat-ai / cvat

Annotate better with CVAT, the industry-leading data engine for machine learning. Used and trusted by teams at any scale, for data of any scale.
https://cvat.ai
MIT License
11.75k stars 2.88k forks source link

Server run out of disk space when cloud storage is used. #8009

Closed Tom1204 closed 5 days ago

Tom1204 commented 2 weeks ago

Actions before raising this issue

Steps to Reproduce

  1. Deploy CVAT with docker
  2. Create project and attach cloud storage
  3. Create task from cloud storage images(where image quality is maximum 70%)

Disk space 300GB

Expected Behavior

Now I am facing issue where CVAT is crashing due to health check for disk space > 90%. So, when I create task it download original images and use them for annotations. When I have look for directories I have seen that it saves original and compressed images.

image image

I have around 1TB data on cloud. So, I can't create task for rest of them. Or even use it.

Possible Solution

It should clean images when disk space runs out or after compressing. Or after defined period it should clean downloaded cloud images.

Context

image image image

Environment

No response

bsekachev commented 2 weeks ago

Hello, Please, try to reproduce the problem in the latest version of CVAT before submitting the issue.

Tom1204 commented 2 weeks ago

I have updated to 2.14.3, however it did not clean the disk. CVAT did not launch. I have cleaned it by myself. I wanted to ask have you thought about automatic cleaning disk, if it is using cloud?

What should I do if I have TB data in cloud, but have 300GB in server which is installed CVAT? What is best practice for such situation? @bsekachev

bsekachev commented 1 week ago
  1. If you create tasks from a cloud storage with enabled "Use cache" option, it will not copy all the data from the cloud storage to local disk (but this is supported only for images, not for videos).
  2. But when you request frames from the browser, it creates data chunks in cache. Default lifetime of this cache is 24 hours. This cache is stored on the local disk.
  3. Upgrading CVAT will not remove something automatically.