search

cvat-ai / cvat

Annotate better with CVAT, the industry-leading data engine for machine learning. Used and trusted by teams at any scale, for data of any scale.
https://cvat.ai
MIT License
12.62k stars 3.01k forks source link

Inference Error from Automatics Annotation Tools #2280

Closed GoGoPen closed 1 year ago

GoGoPen commented 4 years ago

I am encountering the problem in the automatics annotation tools.

I initialized the CVAT with the following command 

docker-compose -f docker-compose.yml -f components/analytics/docker-compose.analytics.yml -f components/serverless/docker-compose.serverless.yml up -d --build

Then, I have used nuctl-1.4.17-linux-amd64 and nuctl-1.1.37-linux-amd64 to build the automatic annotation tool. For example, 

./nuctl-1.4.17-linux-amd64 deploy --project-name cvat \
    --path serverless/openvino/dextr/nuclio \
    --volume `pwd`/serverless/openvino/common:/opt/nuclio/common \
    --platform local

It is properly built and running in nuclio. However, when I try to run the inference.

The following error message showed in the CVAT, 

Fetching inference status for the task 2
Error: Inference status for the task 2 is failed. requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://nuclio:8070/api/function_invocations

Here are the relevant informations.

docker ps
CONTAINER ID        IMAGE                                          COMMAND                  CREATED             STATUS                 PORTS                                        NAMES
7b015a0e8731        zhang0jhon/demo:ocr                            "bash"                   33 minutes ago      Up 33 minutes          6006/tcp, 0.0.0.0:5000->5000/tcp, 8888/tcp   stupefied_wright
68d3e5792d11        alpine:3.11                                    "/bin/sh -c '/bin/sl…"   3 hours ago         Up 3 hours                                                          nuclio-local-storage-reader
5a0c2e4f3ca0        cvat/openvino.dextr:latest                     "processor"              21 hours ago        Up 3 hours (healthy)   0.0.0.0:39665->8080/tcp                      nuclio-nuclio-openvino.dextr
31f13f0ee6c8        cvat/tf.faster_rcnn_inception_v2_coco:latest   "processor"              21 hours ago        Up 3 hours (healthy)   0.0.0.0:40485->8080/tcp                      nuclio-nuclio-tf.faster_rcnn_inception_v2_coco
8628400c703a        nginx:stable-alpine                            "/docker-entrypoint.…"   22 hours ago        Up 3 hours             0.0.0.0:8080->80/tcp                         cvat_proxy
279675fafd74        cvat/ui                                        "/docker-entrypoint.…"   22 hours ago        Up 3 hours             80/tcp                                       cvat_ui
8f8afb9182d9        cvat/server                                    "/usr/bin/supervisord"   22 hours ago        Up 3 hours             8080/tcp, 8443/tcp                           cvat
d956a8ca7b39        cvat_logstash                                  "/usr/local/bin/dock…"   22 hours ago        Up 3 hours             5000/tcp, 5044/tcp, 9600/tcp                 cvat_logstash
32d90bcb7d5a        cvat_kibana                                    "/usr/local/bin/kiba…"   22 hours ago        Up 3 hours             5601/tcp                                     cvat_kibana
066b9f1ec338        quay.io/nuclio/dashboard:1.4.8-amd64           "sh -c ./runner.sh"      22 hours ago        Up 3 hours             80/tcp, 0.0.0.0:8070->8070/tcp               nuclio
861330f37659        postgres:10-alpine                             "docker-entrypoint.s…"   22 hours ago        Up 3 hours             5432/tcp                                     cvat_db
ccfc973213bc        cvat_elasticsearch                             "/usr/local/bin/dock…"   22 hours ago        Up 3 hours             9200/tcp, 9300/tcp                           cvat_elasticsearch
94db6b883589        redis:4.0-alpine                               "docker-entrypoint.s…"   22 hours ago        Up 3 hours             6379/tcp                                     cvat_redis

nuctl get functions

  NAMESPACE |               NAME               | PROJECT | STATE | NODE PORT | REPLICAS  
  nuclio    | openvino.dextr                   | cvat    | ready |     39665 | 1/1       
  nuclio    | tf.faster_rcnn_inception_v2_coco | cvat    | ready |     40485 | 1/1

Log of the cvat/openvino.dextr:latest (5a0c2e4f3ca0)

docker logs 5a0c2e4f3ca0
20.10.07 09:45:05.356                 processor (I) Starting processor {"version": "Label: 1.4.17, Git commit: 278c7a4fb23a93973d16d87dbaaad87823e9644f, OS: linux, Arch: amd64, Go version: go1.14.3"}
20.10.07 09:45:05.356                 processor (D) Read configuration {"config": "{\n    \"metadata\": {\n        \"name\": \"openvino.dextr\",\n        \"namespace\": \"nuclio\",\n        \"labels\": {\n            \"nuclio.io/project-name\": \"cvat\"\n        },\n        \"annotations\": {\n            \"framework\": \"openvino\",\n            \"min_pos_points\": \"4\",\n            \"name\": \"DEXTR\",\n            \"spec\": \"\",\n            \"type\": \"interactor\"\n        }\n    },\n    \"spec\": {\n        \"description\": \"Deep Extreme Cut\",\n        \"handler\": \"main:handler\",\n        \"runtime\": \"python:3.6\",\n        \"env\": [\n            {\n                \"name\": \"NUCLIO_PYTHON_EXE_PATH\",\n                \"value\": \"/opt/nuclio/common/python3\"\n            }\n        ],\n        \"resources\": {},\n        \"image\": \"cvat/openvino.dextr:latest\",\n        \"targetCPU\": 75,\n        \"triggers\": {\n            \"myHttpTrigger\": {\n                \"class\": \"\",\n                \"kind\": \"http\",\n                \"name\": \"\",\n                \"maxWorkers\": 2,\n                \"workerAvailabilityTimeoutMilliseconds\": 10000,\n                \"attributes\": {\n                    \"maxRequestBodySize\": 33554432\n                }\n            }\n        },\n        \"volumes\": [\n            {\n                \"volume\": {\n                    \"name\": \"volume-1\",\n                    \"hostPath\": {\n                        \"path\": \"/home/corsair/cvat/serverless/openvino/common\"\n                    }\n                },\n                \"volumeMount\": {\n                    \"name\": \"volume-1\",\n                    \"mountPath\": \"/opt/nuclio/common\"\n                }\n            }\n        ],\n        \"build\": {\n            \"image\": \"cvat/openvino.dextr\",\n            \"baseImage\": \"openvino/ubuntu18_runtime:2020.2\",\n            \"directives\": {\n                \"postCopy\": [\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"curl -O https://download.01.org/openvinotoolkit/models_contrib/cvat/dextr_model_v1.zip\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"unzip dextr_model_v1.zip\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"pip3 install Pillow\"\n                    }\n                ],\n                \"preCopy\": [\n                    {\n                        \"kind\": \"USER\",\n                        \"value\": \"root\"\n                    },\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"ln -s /usr/bin/pip3 /usr/bin/pip\"\n                    }\n                ]\n            },\n            \"codeEntryType\": \"image\",\n            \"timestamp\": 1602063904\n        },\n        \"platform\": {\n            \"attributes\": {\n                \"restartPolicy\": {\n                    \"maximumRetryCount\": 3,\n                    \"name\": \"always\"\n                }\n            }\n        },\n        \"readinessTimeoutSeconds\": 60,\n        \"eventTimeout\": \"30s\"\n    },\n    \"PlatformConfig\": null\n}", "platformConfig": "{\n    \"kind\": \"local\",\n    \"webAdmin\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8081\"\n    },\n    \"healthCheck\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8082\"\n    },\n    \"logger\": {\n        \"sinks\": {\n            \"stdout\": {\n                \"kind\": \"stdout\"\n            }\n        },\n        \"system\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ],\n        \"functions\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ]\n    },\n    \"metrics\": {},\n    \"scaleToZero\": {},\n    \"autoScale\": {},\n    \"cronTriggerCreationMode\": \"processor\"\n}"}
20.10.07 09:45:05.356 cessor.healthcheck.server (I) Listening {"listenAddress": ":8082"}
20.10.07 09:45:05.356            processor.http (D) Creating worker pool {"num": 2}
20.10.07 09:45:05.356 sor.http.w1.python.logger (D) Creating listener socket {"path": "/tmp/nuclio-rpc-btuos8astpvpgt99odeg.sock"}
20.10.07 09:45:05.356 sor.http.w0.python.logger (D) Creating listener socket {"path": "/tmp/nuclio-rpc-btuos8astpvpgt99odf0.sock"}
20.10.07 09:45:05.357 sor.http.w1.python.logger (D) Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
20.10.07 09:45:05.357 sor.http.w1.python.logger (D) Using Python handler {"handler": "main:handler"}
20.10.07 09:45:05.357 sor.http.w1.python.logger (D) Using Python executable {"path": "/opt/nuclio/common/python3"}
20.10.07 09:45:05.357 sor.http.w1.python.logger (D) Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio"}
20.10.07 09:45:05.357 sor.http.w1.python.logger (D) Running wrapper {"command": "/opt/nuclio/common/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-btuos8astpvpgt99odeg.sock --platform-kind local --namespace nuclio --worker-id 1 --trigger-kind http --trigger-name myHttpTrigger"}
20.10.07 09:45:05.357 sor.http.w0.python.logger (D) Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
20.10.07 09:45:05.357 sor.http.w0.python.logger (D) Using Python handler {"handler": "main:handler"}
20.10.07 09:45:05.357 sor.http.w0.python.logger (D) Using Python executable {"path": "/opt/nuclio/common/python3"}
20.10.07 09:45:05.357 sor.http.w0.python.logger (D) Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio"}
20.10.07 09:45:05.357 sor.http.w0.python.logger (D) Running wrapper {"command": "/opt/nuclio/common/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-btuos8astpvpgt99odf0.sock --platform-kind local --namespace nuclio --worker-id 0 --trigger-kind http --trigger-name myHttpTrigger"}
[setupvars.sh] OpenVINO environment initialized
[setupvars.sh] OpenVINO environment initialized
20.10.07 09:45:05.627 sor.http.w1.python.logger (I) Wrapper connected {"wid": 1}
20.10.07 09:45:05.627 sor.http.w1.python.logger (D) Waiting for start
l{"datetime": "2020-10-07 09:45:05,627", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "1"}}
20.10.07 09:45:05.647 sor.http.w0.python.logger (I) Wrapper connected {"wid": 0}
20.10.07 09:45:05.647 sor.http.w0.python.logger (D) Waiting for start
l{"datetime": "2020-10-07 09:45:05,647", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "0"}}
l{"datetime": "2020-10-07 09:45:07,829", "level": "info", "message": "Init context...100%", "with": {"worker_id": "1"}}
l{"datetime": "2020-10-07 09:45:07,829", "level": "info", "message": "Replacing logger output", "with": {"handler_name": "default", "worker_id": "1"}}
20.10.07 09:45:07.830 sor.http.w1.python.logger (D) Started
l{"datetime": "2020-10-07 09:45:07,856", "level": "info", "message": "Init context...100%", "with": {"worker_id": "0"}}
l{"datetime": "2020-10-07 09:45:07,856", "level": "info", "message": "Replacing logger output", "with": {"handler_name": "default", "worker_id": "0"}}
20.10.07 09:45:07.856 sor.http.w0.python.logger (D) Started
20.10.07 09:45:07.857                 processor (I) Starting event timeout watcher {"timeout": "30s"}
20.10.07 09:45:07.857 .webadmin.server.triggers (D) Registered custom route {"pattern": "/{id}/stats", "method": "GET"}
20.10.07 09:45:07.857 processor.webadmin.server (D) Registered resource {"name": "triggers"}
20.10.07 09:45:07.857                 processor (W) No metric sinks configured, metrics will not be published
20.10.07 09:45:07.857                 processor (D) Starting triggers {"triggers": [{"ID":"myHttpTrigger","Logger":{},"WorkerAllocator":{},"Class":"sync","Kind":"http","Name":"myHttpTrigger","Statistics":{"EventsHandledSuccessTotal":0,"EventsHandledFailureTotal":0,"WorkerAllocatorStatistics":{"WorkerAllocationCount":0,"WorkerAllocationSuccessImmediateTotal":0,"WorkerAllocationSuccessAfterWaitTotal":0,"WorkerAllocationTimeoutTotal":0,"WorkerAllocationWaitDurationMilliSecondsSum":0,"WorkerAllocationWorkersAvailablePercentage":0}},"Namespace":"nuclio","FunctionName":"openvino.dextr"}]}
20.10.07 09:45:07.857            processor.http (I) Starting {"listenAddress": ":8080", "readBufferSize": 16384, "maxRequestBodySize": 33554432, "cors": null}
20.10.07 09:45:07.857 processor.webadmin.server (I) Listening {"listenAddress": ":8081"}
20.10.07 09:45:07.857                 processor (D) Processor started
20.10.08 03:35:31.422                 processor (I) Starting processor {"version": "Label: 1.4.17, Git commit: 278c7a4fb23a93973d16d87dbaaad87823e9644f, OS: linux, Arch: amd64, Go version: go1.14.3"}
20.10.08 03:35:31.422                 processor (D) Read configuration {"config": "{\n    \"metadata\": {\n        \"name\": \"openvino.dextr\",\n        \"namespace\": \"nuclio\",\n        \"labels\": {\n            \"nuclio.io/project-name\": \"cvat\"\n        },\n        \"annotations\": {\n            \"framework\": \"openvino\",\n            \"min_pos_points\": \"4\",\n            \"name\": \"DEXTR\",\n            \"spec\": \"\",\n            \"type\": \"interactor\"\n        }\n    },\n    \"spec\": {\n        \"description\": \"Deep Extreme Cut\",\n        \"handler\": \"main:handler\",\n        \"runtime\": \"python:3.6\",\n        \"env\": [\n            {\n                \"name\": \"NUCLIO_PYTHON_EXE_PATH\",\n                \"value\": \"/opt/nuclio/common/python3\"\n            }\n        ],\n        \"resources\": {},\n        \"image\": \"cvat/openvino.dextr:latest\",\n        \"targetCPU\": 75,\n        \"triggers\": {\n            \"myHttpTrigger\": {\n                \"class\": \"\",\n                \"kind\": \"http\",\n                \"name\": \"\",\n                \"maxWorkers\": 2,\n                \"workerAvailabilityTimeoutMilliseconds\": 10000,\n                \"attributes\": {\n                    \"maxRequestBodySize\": 33554432\n                }\n            }\n        },\n        \"volumes\": [\n            {\n                \"volume\": {\n                    \"name\": \"volume-1\",\n                    \"hostPath\": {\n                        \"path\": \"/home/corsair/cvat/serverless/openvino/common\"\n                    }\n                },\n                \"volumeMount\": {\n                    \"name\": \"volume-1\",\n                    \"mountPath\": \"/opt/nuclio/common\"\n                }\n            }\n        ],\n        \"build\": {\n            \"image\": \"cvat/openvino.dextr\",\n            \"baseImage\": \"openvino/ubuntu18_runtime:2020.2\",\n            \"directives\": {\n                \"postCopy\": [\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"curl -O https://download.01.org/openvinotoolkit/models_contrib/cvat/dextr_model_v1.zip\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"unzip dextr_model_v1.zip\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"pip3 install Pillow\"\n                    }\n                ],\n                \"preCopy\": [\n                    {\n                        \"kind\": \"USER\",\n                        \"value\": \"root\"\n                    },\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"ln -s /usr/bin/pip3 /usr/bin/pip\"\n                    }\n                ]\n            },\n            \"codeEntryType\": \"image\",\n            \"timestamp\": 1602063904\n        },\n        \"platform\": {\n            \"attributes\": {\n                \"restartPolicy\": {\n                    \"maximumRetryCount\": 3,\n                    \"name\": \"always\"\n                }\n            }\n        },\n        \"readinessTimeoutSeconds\": 60,\n        \"eventTimeout\": \"30s\"\n    },\n    \"PlatformConfig\": null\n}", "platformConfig": "{\n    \"kind\": \"local\",\n    \"webAdmin\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8081\"\n    },\n    \"healthCheck\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8082\"\n    },\n    \"logger\": {\n        \"sinks\": {\n            \"stdout\": {\n                \"kind\": \"stdout\"\n            }\n        },\n        \"system\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ],\n        \"functions\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ]\n    },\n    \"metrics\": {},\n    \"scaleToZero\": {},\n    \"autoScale\": {},\n    \"cronTriggerCreationMode\": \"processor\"\n}"}
20.10.08 03:35:31.422 cessor.healthcheck.server (I) Listening {"listenAddress": ":8082"}
20.10.08 03:35:31.422            processor.http (D) Creating worker pool {"num": 2}
20.10.08 03:35:31.424 sor.http.w1.python.logger (D) Creating listener socket {"path": "/tmp/nuclio-rpc-btv8i0qstpvpgtfqqar0.sock"}
20.10.08 03:35:31.424 sor.http.w0.python.logger (D) Creating listener socket {"path": "/tmp/nuclio-rpc-btv8i0qstpvpgtfqqarg.sock"}
20.10.08 03:35:31.424 sor.http.w1.python.logger (D) Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
20.10.08 03:35:31.424 sor.http.w1.python.logger (D) Using Python handler {"handler": "main:handler"}
20.10.08 03:35:31.424 sor.http.w0.python.logger (D) Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
20.10.08 03:35:31.424 sor.http.w1.python.logger (D) Using Python executable {"path": "/opt/nuclio/common/python3"}
20.10.08 03:35:31.424 sor.http.w1.python.logger (D) Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio"}
20.10.08 03:35:31.424 sor.http.w0.python.logger (D) Using Python handler {"handler": "main:handler"}
20.10.08 03:35:31.424 sor.http.w1.python.logger (D) Running wrapper {"command": "/opt/nuclio/common/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-btv8i0qstpvpgtfqqar0.sock --platform-kind local --namespace nuclio --worker-id 1 --trigger-kind http --trigger-name myHttpTrigger"}
20.10.08 03:35:31.424 sor.http.w0.python.logger (D) Using Python executable {"path": "/opt/nuclio/common/python3"}
20.10.08 03:35:31.424 sor.http.w0.python.logger (D) Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio"}
20.10.08 03:35:31.424 sor.http.w0.python.logger (D) Running wrapper {"command": "/opt/nuclio/common/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-btv8i0qstpvpgtfqqarg.sock --platform-kind local --namespace nuclio --worker-id 0 --trigger-kind http --trigger-name myHttpTrigger"}
[setupvars.sh] OpenVINO environment initialized
[setupvars.sh] OpenVINO environment initialized
20.10.08 03:35:31.815 sor.http.w1.python.logger (I) Wrapper connected {"wid": 1}
20.10.08 03:35:31.815 sor.http.w1.python.logger (D) Waiting for start
l{"datetime": "2020-10-08 03:35:31,815", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "1"}}
20.10.08 03:35:31.841 sor.http.w0.python.logger (I) Wrapper connected {"wid": 0}
l{"datetime": "2020-10-08 03:35:31,841", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "0"}}
20.10.08 03:35:31.841 sor.http.w0.python.logger (D) Waiting for start
l{"datetime": "2020-10-08 03:35:34,554", "level": "info", "message": "Init context...100%", "with": {"worker_id": "0"}}
l{"datetime": "2020-10-08 03:35:34,554", "level": "info", "message": "Replacing logger output", "with": {"handler_name": "default", "worker_id": "0"}}
20.10.08 03:35:34.554 sor.http.w0.python.logger (D) Started
l{"datetime": "2020-10-08 03:35:34,557", "level": "info", "message": "Init context...100%", "with": {"worker_id": "1"}}
l{"datetime": "2020-10-08 03:35:34,557", "level": "info", "message": "Replacing logger output", "with": {"handler_name": "default", "worker_id": "1"}}
20.10.08 03:35:34.557 sor.http.w1.python.logger (D) Started
20.10.08 03:35:34.557                 processor (I) Starting event timeout watcher {"timeout": "30s"}
20.10.08 03:35:34.557 .webadmin.server.triggers (D) Registered custom route {"pattern": "/{id}/stats", "method": "GET"}
20.10.08 03:35:34.557 processor.webadmin.server (D) Registered resource {"name": "triggers"}
20.10.08 03:35:34.557                 processor (W) No metric sinks configured, metrics will not be published
20.10.08 03:35:34.557                 processor (D) Starting triggers {"triggers": [{"ID":"myHttpTrigger","Logger":{},"WorkerAllocator":{},"Class":"sync","Kind":"http","Name":"myHttpTrigger","Statistics":{"EventsHandledSuccessTotal":0,"EventsHandledFailureTotal":0,"WorkerAllocatorStatistics":{"WorkerAllocationCount":0,"WorkerAllocationSuccessImmediateTotal":0,"WorkerAllocationSuccessAfterWaitTotal":0,"WorkerAllocationTimeoutTotal":0,"WorkerAllocationWaitDurationMilliSecondsSum":0,"WorkerAllocationWorkersAvailablePercentage":0}},"Namespace":"nuclio","FunctionName":"openvino.dextr"}]}
20.10.08 03:35:34.558            processor.http (I) Starting {"listenAddress": ":8080", "readBufferSize": 16384, "maxRequestBodySize": 33554432, "cors": null}
20.10.08 03:35:34.558 processor.webadmin.server (I) Listening {"listenAddress": ":8081"}
20.10.08 03:35:34.558                 processor (D) Processor started

Thank you very much for your help.

charlescho64 commented 4 years ago

Hi @GoGoPen

Did you get model(you installed) in model list window? If you can't see your model, refer to "https://github.com/openvinotoolkit/cvat/issues/2245" Bye

GoGoPen commented 4 years ago

Hi @GoGoPen

Did you get model(you installed) in model list window? If you can't see your model, refer to "#2245" Bye

The issue you mentioned is not related to mine. The first docker command I wrote already included serverless function. I could find the models installed are shown in the Nuclio with "Running" status. But the inference failed.

azhavoro commented 4 years ago

@GoGoPen Hi, could you also attach the logs from the cvat container?

GoGoPen commented 4 years ago

@GoGoPen Hi, could you also attach the logs from the cvat container?

Thank you very much for your reply and help. Here are all the logs of all container from CVAT. For nginx:stable-alpine , it only gives empty log.

cvat_elastic.txt cvat_elasticsearch cvat_kibana.txt cvat_kibana cvat_logstash.txt cvat_logstash cvat_postgres.txt postgres:10-alpine cvat_quay.txt quay.io/nuclio/dashboard:1.4.8-amd64 cvat_redis.txt redis:4.0-alpine cvat_server.txt cvat/server cvat_ui.txt cvat/ui nginx.txt nginx:stable-alpine
cvat_tf.txt cvat/tf.faster_rcnn_inception_v2_coco:latest

nmanovic commented 4 years ago

@GoGoPen , thanks for the report.

Just some thoughts and observation:

``` 20.10.09 08:43:18.781 .api/function_invocations (W) Failed to invoke function {"err": "Failed to send HTTP request", "errVerbose": "\nError - Post \"http://172.17.0.1:33783\": dial tcp 172.17.0.1:33783: i/o timeout\n /nuclio/pkg/platform/abstract/invoker.go:119\n\nCall stack:\nFailed to send HTTP request\n /nuclio/pkg/platform/abstract/invoker.go:119\nFailed to send HTTP request", "errCauses": [{"error": "Post \"http://172.17.0.1:33783\": dial tcp 172.17.0.1:33783: i/o timeout"}]} 20.10.09 08:43:18.782  dashboard.server (D) Handled request {"requestMethod": "POST", "requestPath": "/api/function_invocations", "requestHeaders": {"Accept":["*/*"],"Accept-Encoding":["gzip, deflate"],"Connection":["close"],"Content-Length":["2325689"],"Content-Type":["application/json"],"User-Agent":["python-requests/2.24.0"],"X-Nuclio-Function-Name":["tf.faster_rcnn_inception_v2_coco"],"X-Nuclio-Function-Namespace":["nuclio"],"X-Nuclio-Log-Level":[""],"X-Nuclio-Path":["/"],"X-Nuclio-Project-Name":["cvat"],"X-Nuclio-Target":["tf.faster_rcnn_inception_v2_coco"]}, "requestBody": "{\"image\": \"/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAIBAQEBAQIBAQECAgICAgQDAgICAgUEBAMEBgUGBgYFBgYGBwkIBgcJBwYGCAsICQoKCgoKBggLDAsKDAkKCgr/2wBDAQICAgICAgUDAwUKBwYHCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgr/wAARCAhwDwADASIAAhEBAxEB/ ```

A couple of questions:

GoGoPen commented 4 years ago

@GoGoPen , thanks for the report.

Just some thoughts and observation:

  • From cvat_tf.txt I can see that the function is up and running but it has not invoked by a reason (no logs about that).
  • From cvat_server.txt I see that the server tries to call a function but it gets 500 error by a reason.
  • From cvat_quay.txt I see that we have two functions: dextr and faster. First one isn't healthy but faster is fine. When we try to call it, CVAT gets I/O timeout.
20.10.09 08:43:18.781 �[37m.api/function_invocations�[0m �[33m(W)�[0m Failed to invoke function {"err": "Failed to send HTTP request", "errVerbose": "\nError - Post \"http://172.17.0.1:33783\": dial tcp 172.17.0.1:33783: i/o timeout\n    /nuclio/pkg/platform/abstract/invoker.go:119\n\nCall stack:\nFailed to send HTTP request\n    /nuclio/pkg/platform/abstract/invoker.go:119\nFailed to send HTTP request", "errCauses": [{"error": "Post \"http://172.17.0.1:33783\": dial tcp 172.17.0.1:33783: i/o timeout"}]}
20.10.09 08:43:18.782 �[37m         dashboard.server�[0m �[32m(D)�[0m Handled request {"requestMethod": "POST", "requestPath": "/api/function_invocations", "requestHeaders": {"Accept":["*/*"],"Accept-Encoding":["gzip, deflate"],"Connection":["close"],"Content-Length":["2325689"],"Content-Type":["application/json"],"User-Agent":["python-requests/2.24.0"],"X-Nuclio-Function-Name":["tf.faster_rcnn_inception_v2_coco"],"X-Nuclio-Function-Namespace":["nuclio"],"X-Nuclio-Log-Level":[""],"X-Nuclio-Path":["/"],"X-Nuclio-Project-Name":["cvat"],"X-Nuclio-Target":["tf.faster_rcnn_inception_v2_coco"]}, "requestBody": "{\"image\": \"/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAIBAQEBAQIBAQECAgICAgQDAgICAgUEBAMEBgUGBgYFBgYGBwkIBgcJBwYGCAsICQoKCgoKBggLDAsKDAkKCgr/2wBDAQICAgICAgUDAwUKBwYHCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgr/wAARCAhwDwADASIAAhEBAxEB/

A couple of questions:

What OS do you use? I am using Ubuntu 18.04.

Did automatic annotation work for you in the past? I have started to use CVAT recently (Oct-1) . When I first use CVAT, it is not working for me already.

Please try these steps: https://github.com/openvinotoolkit/cvat/blob/develop/CONTRIBUTING.md#dl-models-as-serverless-functions and report your results

I used nuctl-1.4.18 as suggested to build to annotation models. Both examples from installation were built properly and showed "Running" Status in the nuclio.

Then I tried to use the DEXRT segmentation annotation tools. It gives 

Interaction error occured
Error: Request failed with status code 500. "500 Server Error: Internal Server Error for url: http://nuclio:8070/api/function_invocations".

I have also tried to use Yolo V3 TF, it gives

Error: Inference status for the task 5 is failed. requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://nuclio:8070/api/function_invocations

cvat_server_log.txt cvat_ui_log.txt dashboard_log.txt

nuctl get function 
  NAMESPACE |      NAME      | PROJECT | STATE | NODE PORT | REPLICAS  
  nuclio    | deploy         | cvat    | ready |     32769 | 1/1       
  nuclio    | openvino.dextr | cvat    | ready |     32768 | 1/1

Thank you very much.

borgarpa commented 4 years ago

Hi, I'm having the same issue when using matterport's Mask RCNN and f-BRS, although I don't have Mask RCNN deployed now. Interestingly, DEXTR causes no error whatsoever.

When using f-BRS, the following error prompts:

Interaction error occured
Error: Request failed with status code 500.
"500 Server Error: Internal Server Error for url: http://nuclio:8070/api/function_invocations".

I'm using Ubuntu 20.04 distro in WSL2.

Some info below:

docker_logs_ps.txt nuctl_get_functions.txt nuctl_version.txt docker_logs_cvat.txt docker_logs_cvat_db.txt docker_logs_cvat_proxy.txt docker_logs_cvat_redis.txt docker_logs_cvat_ui.txt docker_logs_nuclio-nuclio-openvino_dextr.txt docker_logs_nuclio-nuclio-pth_saic-vul_fbrs.txt

GehadG commented 3 years ago

@borgarpa did you manage to solve it for f-BRS , i am having the exact same problem

bsekachev commented 1 year ago

I will close the issue as outdated. Please, reopen if still relevant.