search

cvat-ai / cvat

Annotate better with CVAT, the industry-leading data engine for machine learning. Used and trusted by teams at any scale, for data of any scale.
https://cvat.ai
MIT License
12.61k stars 3.01k forks source link

GUI crashes hard when task repos are broken #4360

Closed MartinHanewald closed 8 months ago

MartinHanewald commented 2 years ago

My actions before raising this issue

I ran automatic annotation with a custom model over the night. All annotations were inserted into the task successfully. But now all other tasks cannot be opened in the UI and I get a React error message.

Expected Behaviour

I should be able to open tasks in the project without UI error message.

Current Behaviour

cvatbug

Error
Minified React error #31; visit https://reactjs.org/docs/error-decoder.html?invariant=31&args[]=object%20with%20keys%20%7Bformat%7D&args[]= for the full message or use the non-minified dev environment for full errors and additional helpful warnings.

Error: Minified React error #31; visit https://reactjs.org/docs/error-decoder.html?invariant=31&args[]=object%20with%20keys%20%7Bformat%7D&args[]= for the full message or use the non-minified dev environment for full errors and additional helpful warnings.
    at Eo (https://tg-annotation.hanewald.ai/cvat-ui.580642541947bec0314e.min.js:346:51489)
    at f (https://tg-annotation.hanewald.ai/cvat-ui.580642541947bec0314e.min.js:346:53127)
    at m (https://tg-annotation.hanewald.ai/cvat-ui.580642541947bec0314e.min.js:346:54236)
    at https://tg-annotation.hanewald.ai/cvat-ui.580642541947bec0314e.min.js:346:56319
    at Li (https://tg-annotation.hanewald.ai/cvat-ui.580642541947bec0314e.min.js:346:65790)
    at vs (https://tg-annotation.hanewald.ai/cvat-ui.580642541947bec0314e.min.js:346:106176)
    at cl (https://tg-annotation.hanewald.ai/cvat-ui.580642541947bec0314e.min.js:346:96717)
    at sl (https://tg-annotation.hanewald.ai/cvat-ui.580642541947bec0314e.min.js:346:96642)
    at Qs (https://tg-annotation.hanewald.ai/cvat-ui.580642541947bec0314e.min.js:346:93672)
    at https://tg-annotation.hanewald.ai/cvat-ui.580642541947bec0314e.min.js:346:45314

That error message decodes to Objects are not valid as a React child (found: object with keys {format}). If you meant to render a collection of children, use an array instead.

I can however access the tasks and all their information via the REST API endpoint, so it does not seem to be a backend issue. I already tried restarting all containers to no avail.

Possible Solution

I have no clue whatever caused this, so I have no suggestion.

Steps to Reproduce (for bugs)

I created a custom model in a nuclio function and ran automatic annotation on a large task with > 2000 images. The tasks creation I also did via REST API. I did not touch the other tasks. But every single one of them (apart from the automatic annotated one) throws that error in the UI.

Context

...

Your Environment

GCP compute engine, 150GB SSD, 16 GB RAM, 4 Cores, Nvidia T4 GPU CVAT runs with docker-compose

CVAT version Server: 1.7 Core: 3.17.0 Canvas: 2.8.0 UI: 1.25.0

MartinHanewald commented 2 years ago

Now it comes to my mind, that during the annotation process I ran into diskspace issues and had to restart the service. The backend server had IO errors in its log. Could it be, that this corrupted something? However, as I stated I can access the tasks via the API without problems. Could there be something frontend specific in the database which needs to be re-initialized to fix this error?

On another note: I could also use some hints on how to debug this further...

MartinHanewald commented 2 years ago

Ok, I am one step further with this...actually it does not seem to have to do with the automatic annotation, but with the connected GIT repo of the other tasks. The error which is displayed via toast shows

image

And one of the requests to https://cvat-host/git/repository/get/44 shows

{"url": {"value": "https://github.com/MartinHanewald/tg-poc-training [annotation/Erdaushub_(Set_130)_v2.zip]"}, "status": {"value": null, "error": "Local repository is failed"}, "format": {"format": null}}

So how can I fix or remove a broken local repository?

MartinHanewald commented 2 years ago

Ok, so I actually solved my acute problem, but it is still not very desirable behaviour. Seems like somehow the remotes from all git repos in the subfolders /home/django/data/tasks/<id>/artifacts/repos got deleted, hence they were reported as "broken" and the GUI crashed hard. I have no idea, how that happened in the first place. My fix was to login manually to the PostgreSQL DB and delete all entries from dataset_repo_gitdata. Now my tasks of course are not connected to git anymore, but I can access them again, which is more important. I guess there is an API endpoint for adding a git repo after task creation. As this is not documented and I have only seen it in the utils/CLI code I have no time right now to try it out.

Anyways, I hesitate to close this issue, since actually the GUI should not fail hard when repos are broken. So this should be fixed.

Keep up the good work :)

nmanovic commented 2 years ago

@MartinHanewald , thanks for your investigation. We will try to address the issue in our future releases.

bsekachev commented 8 months ago

Functionality removed