search

cvat-ai / cvat

Annotate better with CVAT, the industry-leading data engine for machine learning. Used and trusted by teams at any scale, for data of any scale.
https://cvat.ai
MIT License
12.34k stars 2.97k forks source link

500 Server Error: Internal Server Error for url: http://host.docker.internal:32769/" #7267

Open hoangdung3498 opened 9 months ago

hoangdung3498 commented 9 months ago

Actions before raising this issue

Steps to Reproduce

I ran the following instructions to get the serverless function siammask on CVAT: B1: docker compose -f docker-compose.yml -f docker-compose.dev.yml -f components/serverless/docker-compose.serverless.yml up -d --build B2: ./serverless/deploy_gpu.sh serverless/pytorch/foolwood/siammask Everything worked fine with 416x416 images, but when I ran the model to track with 640x640 photos, I got an error. I'm definitely at fault for the size of the image because when I resize the image to a smaller size it works properly.

Expected Behavior

tracking well like image size 640x640

Possible Solution

-I've read a few errors that seem similar to me, doubling the default value of maxRequestBodySize, from 32Mb to 64Mb, but still failing with 640x640 images.

Context

The error here is :

Tracker initialization error Request failed with status code 500. "500 Server Error: Internal Server Error for url: http://host.docker.internal:32769/".

Environment

No response

bsekachev commented 9 months ago

Be sure you have enough GPU memory available, try to get logs from the container: docker logs nuclio-nuclio-<container_function_name>

Try running on CPU.

hoangdung3498 commented 9 months ago

docker logs nuclio-nuclio-

when i > docker logs nuclio-nuclio-

when i use docker ps, it show:

(base) huy:~/cvat$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 065db9df4ee9 gcr.io/iguazio/alpine:3.15 "/bin/sh -c '/bin/sl…" 3 hours ago Up 3 hours nuclio-local-storage-reader ae6d30d25df4 cvat/ui:dev "/docker-entrypoint.…" 22 hours ago Up 22 hours 80/tcp cvat_ui 377e33b0b2f1 cvat/server:dev "/home/django/backen…" 22 hours ago Up 22 hours 8080/tcp, 0.0.0.0:9090->9090/tcp, :::9090->9090/tcp cvat_server f99f42c2b4b2 cvat/server:dev "/usr/bin/supervisor…" 22 hours ago Up 22 hours 8080/tcp, 0.0.0.0:9091->9091/tcp, :::9091->9091/tcp cvat_worker_annotation 8f2c331d8472 cvat/server:dev "/usr/bin/supervisor…" 22 hours ago Up 22 hours 8080/tcp, 0.0.0.0:9094->9094/tcp, :::9094->9094/tcp cvat_worker_quality_reports 6ef780ba855b cvat/server:dev "/usr/bin/supervisor…" 22 hours ago Up 22 hours 8080/tcp, 0.0.0.0:9093->9093/tcp, :::9093->9093/tcp cvat_worker_import edd7fecabb82 cvat/server:dev "/usr/bin/supervisor…" 22 hours ago Up 22 hours 8080/tcp, 0.0.0.0:9092->9092/tcp, :::9092->9092/tcp cvat_worker_export 25dc27d006f4 cvat/server:dev "/usr/bin/supervisor…" 22 hours ago Up 22 hours 8080/tcp cvat_utils 53cde481affa cvat/server:dev "/usr/bin/supervisor…" 22 hours ago Up 22 hours 8080/tcp cvat_worker_webhooks 6bbb0ed9f369 timberio/vector:0.26.0-alpine "/usr/local/bin/vect…" 22 hours ago Up 22 hours cvat_vector 69712c1977c5 postgres:15-alpine "docker-entrypoint.s…" 22 hours ago Up 22 hours 0.0.0.0:5432->5432/tcp, :::5432->5432/tcp cvat_db 47f4810f329c grafana/grafana-oss:9.3.6 "sh -euc 'mkdir -p /…" 22 hours ago Up 22 hours 3000/tcp cvat_grafana 3a1962413e49 quay.io/nuclio/dashboard:1.8.14-amd64 "/docker-entrypoint.…" 22 hours ago Up 22 hours (healthy) 80/tcp, 0.0.0.0:8070->8070/tcp, :::8070->8070/tcp nuclio d6277f1df473 clickhouse/clickhouse-server:22.3-alpine "/entrypoint.sh" 22 hours ago Up 22 hours 8123/tcp, 9000/tcp, 9009/tcp cvat_clickhouse 2ef9935a8648 openpolicyagent/opa:0.45.0-rootless "/opa run --server -…" 22 hours ago Up 22 hours 0.0.0.0:8181->8181/tcp, :::8181->8181/tcp cvat_opa 2d7c28e0f0ab traefik:v2.9 "/entrypoint.sh --pr…" 22 hours ago Up 22 hours 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp, 80/tcp, 0.0.0.0:8090->8090/tcp, :::8090->8090/tcp traefik f7ee33a54af5 redis:7.0-alpine "docker-entrypoint.s…" 22 hours ago Up 22 hours 6379/tcp cvat_redis 1ed2f6772869 cvat.pth.foolwood.siammask:latest "conda run -n siamma…" 26 hours ago Up 26 hours (healthy) 0.0.0.0:32772->8080/tcp, :::32772->8080/tcp nuclio-nuclio-pth-foolwood-siammask 83e7a6c92148 cvat.openvino.omz.public.yolo-v3-tf:latest "processor" 47 hours ago Up 47 hours (healthy) 0.0.0.0:32770->8080/tcp, :::32770->8080/tcp nuclio-nuclio-openvino-omz-public-yolo-v3-tf f5356cd11695 cvat.openvino.dextr:latest "processor" 47 hours ago Up 47 hours (healthy) 0.0.0.0:32768->8080/tcp, :::32768->8080/tcp nuclio-nuclio-openvino-dextr 68318c343954 dat.local:5001/milvusdb/milvus:1.1.1-cuda118-4b031c-04571b "/app/docker-entrypo…" 5 months ago Up 2 days (healthy) 0.0.0.0:19121->19121/tcp, :::19121->19121/tcp, 0.0.0.0:19530->19530/tcp, :::19530->19530/tcp huy-vectordbmilvus-1 710940132dda mysql:5.7 "docker-entrypoint.s…" 5 months ago Up 2 days (healthy) 3306/tcp, 33060/tcp huy-vectordbmysql-1use docker ps, it show: (base) huy:~/cvat$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 065db9df4ee9 gcr.io/iguazio/alpine:3.15 "/bin/sh -c '/bin/sl…" 3 hours ago Up 3 hours nuclio-local-storage-reader ae6d30d25df4 cvat/ui:dev "/docker-entrypoint.…" 22 hours ago Up 22 hours 80/tcp cvat_ui 377e33b0b2f1 cvat/server:dev "/home/django/backen…" 22 hours ago Up 22 hours 8080/tcp, 0.0.0.0:9090->9090/tcp, :::9090->9090/tcp cvat_server f99f42c2b4b2 cvat/server:dev "/usr/bin/supervisor…" 22 hours ago Up 22 hours 8080/tcp, 0.0.0.0:9091->9091/tcp, :::9091->9091/tcp cvat_worker_annotation 8f2c331d8472 cvat/server:dev "/usr/bin/supervisor…" 22 hours ago Up 22 hours 8080/tcp, 0.0.0.0:9094->9094/tcp, :::9094->9094/tcp cvat_worker_quality_reports 6ef780ba855b cvat/server:dev "/usr/bin/supervisor…" 22 hours ago Up 22 hours 8080/tcp, 0.0.0.0:9093->9093/tcp, :::9093->9093/tcp cvat_worker_import edd7fecabb82 cvat/server:dev "/usr/bin/supervisor…" 22 hours ago Up 22 hours 8080/tcp, 0.0.0.0:9092->9092/tcp, :::9092->9092/tcp cvat_worker_export 25dc27d006f4 cvat/server:dev "/usr/bin/supervisor…" 22 hours ago Up 22 hours 8080/tcp cvat_utils 53cde481affa cvat/server:dev "/usr/bin/supervisor…" 22 hours ago Up 22 hours 8080/tcp cvat_worker_webhooks 6bbb0ed9f369 timberio/vector:0.26.0-alpine "/usr/local/bin/vect…" 22 hours ago Up 22 hours cvat_vector 69712c1977c5 postgres:15-alpine "docker-entrypoint.s…" 22 hours ago Up 22 hours 0.0.0.0:5432->5432/tcp, :::5432->5432/tcp cvat_db 47f4810f329c grafana/grafana-oss:9.3.6 "sh -euc 'mkdir -p /…" 22 hours ago Up 22 hours 3000/tcp cvat_grafana 3a1962413e49 quay.io/nuclio/dashboard:1.8.14-amd64 "/docker-entrypoint.…" 22 hours ago Up 22 hours (healthy) 80/tcp, 0.0.0.0:8070->8070/tcp, :::8070->8070/tcp nuclio d6277f1df473 clickhouse/clickhouse-server:22.3-alpine "/entrypoint.sh" 22 hours ago Up 22 hours 8123/tcp, 9000/tcp, 9009/tcp cvat_clickhouse 2ef9935a8648 openpolicyagent/opa:0.45.0-rootless "/opa run --server -…" 22 hours ago Up 22 hours 0.0.0.0:8181->8181/tcp, :::8181->8181/tcp cvat_opa 2d7c28e0f0ab traefik:v2.9 "/entrypoint.sh --pr…" 22 hours ago Up 22 hours 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp, 80/tcp, 0.0.0.0:8090->8090/tcp, :::8090->8090/tcp traefik f7ee33a54af5 redis:7.0-alpine "docker-entrypoint.s…" 22 hours ago Up 22 hours 6379/tcp cvat_redis 1ed2f6772869 cvat.pth.foolwood.siammask:latest "conda run -n siamma…" 26 hours ago Up 26 hours (healthy) 0.0.0.0:32772->8080/tcp, :::32772->8080/tcp nuclio-nuclio-pth-foolwood-siammask 83e7a6c92148 cvat.openvino.omz.public.yolo-v3-tf:latest "processor" 47 hours ago Up 47 hours (healthy) 0.0.0.0:32770->8080/tcp, :::32770->8080/tcp nuclio-nuclio-openvino-omz-public-yolo-v3-tf f5356cd11695 cvat.openvino.dextr:latest "processor" 47 hours ago Up 47 hours (healthy) 0.0.0.0:32768->8080/tcp, :::32768->8080/tcp nuclio-nuclio-openvino-dextr 68318c343954 dat.local:5001/milvusdb/milvus:1.1.1-cuda118-4b031c-04571b "/app/docker-entrypo…" 5 months ago Up 2 days (healthy) 0.0.0.0:19121->19121/tcp, :::19121->19121/tcp, 0.0.0.0:19530->19530/tcp, :::19530->19530/tcp huy-vectordbmilvus-1 710940132dda mysql:5.7 "docker-entrypoint.s…" 5 months ago Up 2 days (healthy) 3306/tcp, 33060/tcp huy-vectordbmysql-1

i try to log container cvat.pth.foolwood.siammask:latest by docker logs 1ed2f6772869 but it don't show anything. It's really strange. I also monitor GPU vRam when running image track size 640, vRam does not overflow Screenshot from 2023-12-15 13-52-58

hoangdung3498 commented 9 months ago

Be sure you have enough GPU memory available, try to get logs from the container: docker logs nuclio-nuclio-<container_function_name>

Try running on CPU.

i will try with CPU ,but I really can't understand what the problem is

bsekachev commented 9 months ago

Is you can see in docker ps it was deployed on port 32770. It tries to access :32769

So, try just redeploy the function, it should work.

hoangdung3498 commented 9 months ago

Is you can see in docker ps it was deployed on port 32770. It tries to access :32769

So, try just redeploy the function, it should work.

So sorry, I have copied the old error, in fact I have redeployed the function and the port has changed. I make sure the port of the container, the port on the nuclio as well as the port on the error are the same. Another thing is that I tried on CPU and the same error still occurs, I don't know if there is an image size limit in siammask's deploy code

bsekachev commented 9 months ago

In this case, as I told before, look at nuclio dashboard logs and at nuclio function container logs.