search

cvat-ai / cvat

Annotate better with CVAT, the industry-leading data engine for machine learning. Used and trusted by teams at any scale, for data of any scale.
https://cvat.ai
MIT License
12.49k stars 2.99k forks source link

Could not create the task - 504 (Gateway Timeout) #5885

Closed maecky closed 9 months ago

maecky commented 1 year ago

My actions before raising this issue

Steps to Reproduce (for bugs)

  1. Create Azure Blob Store according to documentation
  2. Upload image data (50k Images) and manifest file to blob storage
  3. Create an empty Project
  4. Create a new Task
    • Select previously created cloud storage
    • Select manifest file
    • Tick "select all" and choose all Images there
    • Advanced Conifguration:
      • untick "use cache"
      • untick "Use project source storage" -> select Cloud Storage
      • untick "Use project target storage" -> select Cloud Storage
  5. Submit & Continue

Expected Behaviour

Task is created and I can start labeling the dataset.

Current Behaviour

Popup in the right upper corner: "Could not create the task Open the Browser Console to get details:

From the Browser Console:

xhr.js:220          POST http://localhost:8080/api/tasks/7/data?org=test504 (Gateway Timeout)
xhr.js:220          DELETE http://localhost:8080/api/tasks/7?org=test500 (Internal Server Error)
cvat-app.tsx:354 Error: Request failed with status code 504. "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>504 Gateway Timeout</title>\n</head><body>\n<h1>Gateway Timeout</h1>\n<p>The gateway did not receive a timely response\nfrom the upstream server or application.</p>\n</body></html>\n".

Your Environment

Latest updates from repository pulled and deployed with docker-compose.

I tried several combinations to create the task (with cache / without cache, with Project source storage enabled / disabled, etc) Any ideas how I can solve this issue without mounting a local file store into the container are welcome :)

Thx!

bsekachev commented 1 year ago

Hi, @maecky

Please, provide some logs from cvat_server container (docker logs cvat_server)

nmanovic commented 1 year ago

@maecky , we need more information from you. Do you use the latest CVAT v2.4.0? Please attach logs as Boris asked. It will help us to identify the issue.

maecky commented 1 year ago

I'm on a business trip currently - will provide them at the end of the week when I have access to my machine again.

maecky commented 1 year ago

I wanted to reproduce the error but my cloud storage is not working anymore. So I tried to create a new cloud storage connection, but I get the same error in the browser now:

xhr.js:220          POST http://localhost:8080/api/cloudstorages?org=test 504 (Gateway Timeout)
cvat-app.tsx:354 Error: Request failed with status code 504. "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>504 Gateway Timeout</title>\n</head><body>\n<h1>Gateway Timeout</h1>\n<p>The gateway did not receive a timely response\nfrom the upstream server or application.</p>\n</body></html>\n".

This is the cvat server log:

2023-03-28 11:17:20 2023-03-28 09:17:20,672 DEBG 'runserver' stderr output:
2023-03-28 11:17:20 [Tue Mar 28 09:17:20.672609 2023] [wsgi:error] [pid 104:tid 139812239501056] [client 172.20.0.7:59152] Timeout when reading response headers from daemon process 'localhost:8080': /tmp/cvat-server/htdocs/api, referer: http://localhost:8080/cloudstorages/create
2023-03-28 11:17:20 
2023-03-28 11:17:48 2023-03-28 09:17:48,176 DEBG 'runserver' stderr output:
2023-03-28 11:17:48 [Tue Mar 28 09:17:48.176409 2023] [wsgi:error] [pid 70:tid 139811715544832] [remote 172.20.0.7:59152] [2023-03-28 09:17:48,176] WARNING django.request: Bad Request: /api/cloudstorages
2023-03-28 11:17:48 
2023-03-28 11:17:48 2023-03-28 09:17:48,176 DEBG 'runserver' stderr output:
2023-03-28 11:17:48 [Tue Mar 28 09:17:48.176494 2023] [wsgi:error] [pid 70:tid 139811715544832] [remote 172.20.0.7:59152] WARNING:django.request:Bad Request: /api/cloudstorages
2023-03-28 11:17:48
josiahls commented 1 year ago

I think I have a similar issue with AWS. I have 2.4.2 installed based on the changelog.

This worked for my validation set (which is smaller). I think depending on the cloud storage, its not waiting long enough if there are lots of cloud storage files.

josiahls commented 1 year ago

With the help of our almighty overlord gpt-4, I was able to solve this.

Issue: Timeout when reading response headers from daemon process while creating a task with a large number of images.

Temporary Solution: We increased various timeout settings in the command section of [program:runserver] in supervisord/server.conf. Additionally, we mounted the modified supervisord.conf into the cvat_server service in the docker-compose.yml file. Here are the changes made:

  1. Added the following options to the command in the [program:runserver] section:

    --request-timeout 1200
--connect-timeout 1200
--socket-timeout 1200
--queue-timeout 1200
--header-timeout 1200
--header-max-timeout 2400
--header-min-rate 500

  2. Added a volume mount in the cvat_server service of the docker-compose.yml file:

    volumes:
 - ./custom_supervisord.conf:/app/supervisord/server.conf

Recommendation: You can monitor the data download progress using PostgreSQL by running a query like:

SELECT count(*) FROM engine_serverfile WHERE data_id=58;

Replace 58 with the relevant data_id for your task.

As well as via docker:

docker compose logs cvat_server -f

and

docker compose logs cvat_worker_import -f

Note: This is still a bug that needs to be addressed. The task creation takes a long time to download the images, and CVAT should not timeout when the download is actively in progress.

I now and also dealing with a similar error: https://github.com/opencv/cvat/issues/5887

handcreate commented 1 year ago

I have a similar issue with [Connected file share]. This error occurs when it takes more than 60 seconds to import images. My cvat version: 2.4.3

Browser's console output: xhr.js:220 POST http://192.168.17.17:8080/api/tasks/73/data?org= 504 (Gateway Timeout) (anonymous) @ xhr.js:220 e.exports @ xhr.js:16 e.exports @ dispatchRequest.js:56 Promise.then (async) d.request @ Axios.js:89 (anonymous) @ Axios.js:144 (anonymous) @ bind.js:9 create @ server-proxy.ts:1181 await in create (async) Br.e.save.implementation @ session-implementation.ts:484 apiWrapper @ plugins.ts:27 await in apiWrapper (async) save @ session.ts:1064 (anonymous) @ tasks-actions.ts:218 (anonymous) @ index.js:16 onCreate @ create-task-page.tsx:24 onCreate @ create-task-page.tsx:37 (anonymous) @ create-task-content.tsx:456 Promise.then (async) (anonymous) @ create-task-content.tsx:431 Q @ button.js:241 s @ react-dom.production.min.js:14 f @ react-dom.production.min.js:14 (anonymous) @ react-dom.production.min.js:14 v @ react-dom.production.min.js:15 ot @ react-dom.production.min.js:52 rt @ react-dom.production.min.js:51 it @ react-dom.production.min.js:52 pt @ react-dom.production.min.js:56 R @ react-dom.production.min.js:287 H @ react-dom.production.min.js:19 Xt @ react-dom.production.min.js:70 $t @ react-dom.production.min.js:69 t.unstable_runWithPriority @ scheduler.production.min.js:19 Ua @ react-dom.production.min.js:122 j @ react-dom.production.min.js:287 qt @ react-dom.production.min.js:68 xhr.js:220 DELETE http://192.168.17.17:8080/api/tasks/73?org= 500 (Internal Server Error) (anonymous) @ xhr.js:220 e.exports @ xhr.js:16 e.exports @ dispatchRequest.js:56 Promise.then (async) d.request @ Axios.js:89 r.forEach.d. @ Axios.js:131 (anonymous) @ bind.js:9 Ye @ server-proxy.ts:662 create @ server-proxy.ts:1188 await in create (async) Br.e.save.implementation @ session-implementation.ts:484 apiWrapper @ plugins.ts:27 await in apiWrapper (async) save @ session.ts:1064 (anonymous) @ tasks-actions.ts:218 (anonymous) @ index.js:16 onCreate @ create-task-page.tsx:24 onCreate @ create-task-page.tsx:37 (anonymous) @ create-task-content.tsx:456 Promise.then (async) (anonymous) @ create-task-content.tsx:431 Q @ button.js:241 s @ react-dom.production.min.js:14 f @ react-dom.production.min.js:14 (anonymous) @ react-dom.production.min.js:14 v @ react-dom.production.min.js:15 ot @ react-dom.production.min.js:52 rt @ react-dom.production.min.js:51 it @ react-dom.production.min.js:52 pt @ react-dom.production.min.js:56 R @ react-dom.production.min.js:287 H @ react-dom.production.min.js:19 Xt @ react-dom.production.min.js:70 $t @ react-dom.production.min.js:69 t.unstable_runWithPriority @ scheduler.production.min.js:19 Ua @ react-dom.production.min.js:122 j @ react-dom.production.min.js:287 qt @ react-dom.production.min.js:68 cvat-app.tsx:354 Error: Request failed with status code 504. "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n\n504 Gateway Timeout\n\n

Gateway Timeout

\n

The gateway did not receive a timely response\nfrom the upstream server or application.

\n\n".

cvat_server logs: 2023-06-02 07:19:55,009 DEBG 'runserver' stderr output: [Fri Jun 02 07:19:55.009110 2023] [wsgi:error] [pid 568:tid 139990556902976] [client 172.21.0.8:53454] Timeout when reading response headers from daemon process 'localhost:8080': /tmp/cvat-server/htdocs/api, referer: http://192.168.17.17:8080/tasks/create?projectId=15

2023-06-02 07:19:55,340 DEBG 'runserver' stderr output: [Fri Jun 02 07:19:55.339741 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] [2023-06-02 07:19:55,338] ERROR django.request: Internal Server Error: /api/tasks/73 [Fri Jun 02 07:19:55.339772 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] Traceback (most recent call last): [Fri Jun 02 07:19:55.339779 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/opt/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 242, in _commit [Fri Jun 02 07:19:55.339786 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] return self.connection.commit() [Fri Jun 02 07:19:55.339793 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] psycopg2.errors.ForeignKeyViolation: update or delete on table "engine_data" violates foreign key constraint "engine_serverfile_data_id_2364110a_fk_engine_data_id" on table "engine_serverfile" [Fri Jun 02 07:19:55.339801 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] DETAIL: Key (id)=(73) is still referenced from table "engine_serverfile". [Fri Jun 02 07:19:55.339808 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] [Fri Jun 02 07:19:55.339813 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] [Fri Jun 02 07:19:55.339819 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] The above exception was the direct cause of the following exception: [Fri Jun 02 07:19:55.339826 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] [Fri Jun 02 07:19:55.339831 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] Traceback (most recent call last): [Fri Jun 02 07:19:55.339837 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner [Fri Jun 02 07:19:55.339844 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] response = get_response(request) [Fri Jun 02 07:19:55.339850 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/opt/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 181, in _get_response [Fri Jun 02 07:19:55.339856 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] response = wrapped_callback(request, *callback_args, *callback_kwargs) [Fri Jun 02 07:19:55.339862 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/opt/venv/lib/python3.10/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view [Fri Jun 02 07:19:55.339869 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] return view_func(args, **kwargs) [Fri Jun 02 07:19:55.339875 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/opt/venv/lib/python3.10/site-packages/rest_framework/viewsets.py", line 125, in view

2023-06-02 07:19:55,340 DEBG 'runserver' stderr output: [Fri Jun 02 07:19:55.339881 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] return self.dispatch(request, *args, kwargs) [Fri Jun 02 07:19:55.339887 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/opt/venv/lib/python3.10/site-packages/rest_framework/views.py", line 509, in dispatch [Fri Jun 02 07:19:55.339894 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] response = self.handle_exception(exc) [Fri Jun 02 07:19:55.339900 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/opt/venv/lib/python3.10/site-packages/rest_framework/views.py", line 469, in handle_exception [Fri Jun 02 07:19:55.339906 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] self.raise_uncaught_exception(exc) [Fri Jun 02 07:19:55.339912 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/opt/venv/lib/python3.10/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception [Fri Jun 02 07:19:55.339918 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] raise exc [Fri Jun 02 07:19:55.339924 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/opt/venv/lib/python3.10/site-packages/rest_framework/views.py", line 506, in dispatch [Fri Jun 02 07:19:55.339930 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] response = handler(request, *args, *kwargs) [Fri Jun 02 07:19:55.339936 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/opt/venv/lib/python3.10/site-packages/drf_spectacular/drainage.py", line 156, in wrapped_method [Fri Jun 02 07:19:55.339943 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] return method(self, request, args, kwargs) [Fri Jun 02 07:19:55.339949 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/opt/venv/lib/python3.10/site-packages/rest_framework/mixins.py", line 91, in destroy [Fri Jun 02 07:19:55.339955 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] self.perform_destroy(instance) [Fri Jun 02 07:19:55.339961 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/opt/venv/lib/python3.10/site-packages/rest_framework/mixins.py", line 95, in perform_destroy [Fri Jun 02 07:19:55.339967 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] instance.delete() [Fri Jun 02 07:19:55.339973 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/home/django/cvat/apps/events/utils.py", line 62, in wrap [Fri Jun 02 07:19:55.339979 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] method(self, *args, *kwargs) [Fri Jun 02 07:19:55.339985 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/home/django/cvat/apps/engine/models.py", line 400, in delete [Fri Jun 02 07:19:55.339991 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] super().delete(using, keep_parents) [Fri Jun 02 07:19:55.339997 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/opt/venv/lib/python3.10/site-packages/django/db/models/base.py", line 967, in delete [Fri Jun 02 07:19:55.340004 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] return collector.delete() [Fri Jun 02 07:19:55.340010 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/opt/venv/lib/python3.10/site-packages/django/db/models/deletion.py", line 400, in delete [Fri Jun 02 07:19:55.340016 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] with transaction.atomic(using=self.using, savepoint=False): [Fri Jun 02 07:19:55.340022 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/opt/venv/lib/python3.10/site-packages/django/db/transaction.py", line 246, in exit [Fri Jun 02 07:19:55.340028 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] connection.commit() [Fri Jun 02 07:19:55.340036 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/opt/venv/lib/python3.10/site-packages/django/utils/asyncio.py", line 33, in inner [Fri Jun 02 07:19:55.340043 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] return func(args, **kwargs) [Fri Jun 02 07:19:55.340049 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/opt/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 266, in commit [Fri Jun 02 07:19:55.340055 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] self._commit() [Fri Jun 02 07:19:55.340061 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/opt/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 241, in _commit [Fri Jun 02 07:19:55.340067 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] with self.wrap_database_errors: [Fri Jun 02 07:19:55.340073 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/opt/venv/lib/python3.10/site-packages/django/db/utils.py", line 90, in exit [Fri Jun 02 07:19:55.340079 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] raise dj_exc_value.with_traceback(traceback) from exc_value [Fri Jun 02 07:19:55.340086 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/opt/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 242, in _commit [Fri Jun 02 07:19:55.340092 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] return self.connection.commit() [Fri Jun 02 07:19:55.340098 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] django.db.utils.IntegrityError: update or delete on table "engine_data" violates foreign key constraint "engine_serverfile_data_id_2364110a_fk_engine_data_id" on table "engine_serverfile" [Fri Jun 02 07:19:55.340105 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] DETAIL: Key (id)=(73) is still referenced from table "engine_serverfile". [Fri Jun 02 07:19:55.340111 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454]

2023-06-02 07:19:55,340 DEBG 'runserver' stderr output: [Fri Jun 02 07:19:55.340349 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] ERROR:django.request:Internal Server Error: /api/tasks/73 [Fri Jun 02 07:19:55.340360 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] Traceback (most recent call last): [Fri Jun 02 07:19:55.340366 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/opt/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 242, in _commit [Fri Jun 02 07:19:55.340373 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] return self.connection.commit() [Fri Jun 02 07:19:55.340379 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] psycopg2.errors.ForeignKeyViolation: update or delete on table "engine_data" violates foreign key constraint "engine_serverfile_data_id_2364110a_fk_engine_data_id" on table "engine_serverfile" [Fri Jun 02 07:19:55.340386 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] DETAIL: Key (id)=(73) is still referenced from table "engine_serverfile". [Fri Jun 02 07:19:55.340392 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] [Fri Jun 02 07:19:55.340398 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] [Fri Jun 02 07:19:55.340404 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] The above exception was the direct cause of the following exception: [Fri Jun 02 07:19:55.340410 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] [Fri Jun 02 07:19:55.340416 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] Traceback (most recent call last): [Fri Jun 02 07:19:55.340421 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner [Fri Jun 02 07:19:55.340428 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] response = get_response(request) [Fri Jun 02 07:19:55.340434 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/opt/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 181, in _get_response [Fri Jun 02 07:19:55.340440 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] response = wrapped_callback(request, *callback_args, callback_kwargs) [Fri Jun 02 07:19:55.340446 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/opt/venv/lib/python3.10/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view [Fri Jun 02 07:19:55.340452 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] return view_func(*args, *kwargs) [Fri Jun 02 07:19:55.340458 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/opt/venv/lib/python3.10/site-packages/rest_framework/viewsets.py", line 125, in view [Fri Jun 02 07:19:55.340465 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] return self.dispatch(request, args, kwargs) [Fri Jun 02 07:19:55.340471 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/opt/venv/lib/python3.10/site-packages/rest_framework/views.py", line 509, in dispatch [Fri Jun 02 07:19:55.340477 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] response = self.handle_exception(exc) [Fri Jun 02 07:19:55.340483 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/opt/venv/lib/python3.10/site-packages/rest_framework/views.py", line 469, in handle_exception [Fri Jun 02 07:19:55.340490 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] self.raise_uncaught_exception(exc) [Fri Jun 02 07:19:55.340496 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/opt/venv/lib/python3.10/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception [Fri Jun 02 07:19:55.340504 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] raise exc [Fri Jun 02 07:19:55.340510 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/opt/venv/lib/python3.10/site-packages/rest_framework/views.py", line 506, in dispatch [Fri Jun 02 07:19:55.340517 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] response = handler(request, *args, kwargs) [Fri Jun 02 07:19:55.340523 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/opt/venv/lib/python3.10/site-packages/drf_spectacular/drainage.py", line 156, in wrapped_method [Fri Jun 02 07:19:55.340530 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] return method(self, request, *args, *kwargs) [Fri Jun 02 07:19:55.340536 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/opt/venv/lib/python3.10/site-packages/rest_framework/mixins.py", line 91, in destroy [Fri Jun 02 07:19:55.340542 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] self.perform_destroy(instance) [Fri Jun 02 07:19:55.340548 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/opt/venv/lib/python3.10/site-packages/rest_framework/mixins.py", line 95, in perform_destroy [Fri Jun 02 07:19:55.340555 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] instance.delete() [Fri Jun 02 07:19:55.340561 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/home/django/cvat/apps/events/utils.py", line 62, in wrap [Fri Jun 02 07:19:55.340567 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] method(self, args, kwargs) [Fri Jun 02 07:19:55.340573 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/home/django/cvat/apps/engine/models.py", line 400, in delete [Fri Jun 02 07:19:55.340579 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] super().delete(using, keep_parents) [Fri Jun 02 07:19:55.340585 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/opt/venv/lib/python3.10/site-packages/django/db/models/base.py", line 967, in delete [Fri Jun 02 07:19:55.340591 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] return collector.delete() [Fri Jun 02 07:19:55.340597 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/opt/venv/lib/python3.10/site-packages/django/db/models/deletion.py", line 400, in delete [Fri Jun 02 07:19:55.340603 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] with transaction.atomic(using=self.using, savepoint=False): [Fri Jun 02 07:19:55.340623 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/opt/venv/lib/python3.10/site-packages/django/db/transaction.py", line 246, in exit [Fri Jun 02 07:19:55.340629 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] connection.commit()

2023-06-02 07:19:55,340 DEBG 'runserver' stderr output: [Fri Jun 02 07:19:55.340635 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/opt/venv/lib/python3.10/site-packages/django/utils/asyncio.py", line 33, in inner [Fri Jun 02 07:19:55.340643 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] return func(*args, **kwargs) [Fri Jun 02 07:19:55.340649 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/opt/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 266, in commit [Fri Jun 02 07:19:55.340655 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] self._commit() [Fri Jun 02 07:19:55.340661 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/opt/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 241, in _commit [Fri Jun 02 07:19:55.340668 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] with self.wrap_database_errors: [Fri Jun 02 07:19:55.340674 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/opt/venv/lib/python3.10/site-packages/django/db/utils.py", line 90, in exit [Fri Jun 02 07:19:55.340680 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] raise dj_exc_value.with_traceback(traceback) from exc_value [Fri Jun 02 07:19:55.340686 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] File "/opt/venv/lib/python3.10/site-packages/django/db/backends/base/base.py", line 242, in _commit [Fri Jun 02 07:19:55.340692 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] return self.connection.commit() [Fri Jun 02 07:19:55.340698 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] django.db.utils.IntegrityError: update or delete on table "engine_data" violates foreign key constraint "engine_serverfile_data_id_2364110a_fk_engine_data_id" on table "engine_serverfile" [Fri Jun 02 07:19:55.340705 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454] DETAIL: Key (id)=(73) is still referenced from table "engine_serverfile". [Fri Jun 02 07:19:55.340711 2023] [wsgi:error] [pid 1633:tid 139987796457024] [remote 172.21.0.8:53454]

geometrikal commented 1 year ago

Worked for me, just needed to change /app/supervisord/server.conf to /home/django/supervisord/server.conf for version v242

arehmer commented 1 year ago

I get the same Exception when using the Python Implementiation of cvat_sdk. I call client.tasks.create_from_data() providing the path to a folder that contains roughly 5000 images. Then I get the following error message and exception

line 127, in create_task_from_data    task = client.tasks.create_from_data(spec=task_spec,...
line 372, in create_from_data    task.upload_data(...
line 125, in upload_data    DataUploader(self._client).upload_files(...
line 371, in upload_files response = self._client.api_client.rest_client.POST(
line 256, in POST    return self.request("POST", url,...
line 209, in request    raise ServiceException(http_resp=r)

ServiceException: Status Code: 504
Reason: Gateway Timeout
HTTP response headers: HTTPHeaderDict({'Content-Length': '176', 'Content-Type': 'text/html', 'Date': 'Fri, 01 Sep 2023 08:53:57 GMT', 'Server': 'nginx/1.18.0 (Ubuntu)'})
HTTP response body: b'<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n<hr><center>nginx/1.18.0 (Ubuntu)</center>\r\n</body>\r\n</html>\r\n'

I can upload about 1400 images no problem. Problems occured when uploading 2800 images or more. But I don't know where excatly the limit is.

geometrikal commented 1 year ago

@nmanovic solved this by increasing the server timeouts but it looks like the .conf file format has changed in latest versions (compared to v2.4.2).

Is it possible to turn these task API endpoints that can potentially take a long time and time out into background tasks where you poll for status etc?