search

cvat-ai / cvat

Annotate better with CVAT, the industry-leading data engine for machine learning. Used and trusted by teams at any scale, for data of any scale.
https://cvat.ai
MIT License
12.38k stars 2.97k forks source link

Automatic annotation 500 ServerError #2189

Closed EldarKurbanov closed 8 months ago

EldarKurbanov commented 4 years ago

My actions before raising this issue

Expected Behaviour

Automatic annotation for the task

Current Behaviour

Failed.requests.exceptions.HTTPError: 500 Server Error: Internal server error for url: http://nuclio:8070/api/function_invocations for all openvino and tensorflow example models from deploy.sh (for example YOLO v3) when click to the automatic annotation button.

Possible Solution

Fix import models to the nuclio

Steps to Reproduce (for bugs)

  1. Installed Ubuntu 18.04 on Amazon (also I tried on Ubuntu 20.04 on new virtual machine);
  2. Installed CVAT by this instructions;
  3. cd ~/cvat
  4. ./deploy.sh

Context

Can't use automatic annotation but I installed cvat and nuctl according to the instructions.

Your Environment

Next steps

Waiting for any help.

bsekachev commented 4 years ago

I am not sure what is a reason, but each model on the screenshot https://prnt.sc/uih1qq has error state. Do these models work immediately after the script ./deploy.sh is performed? I get similar error after rebooting a PC, maybe your instance did reboot too..?

EldarKurbanov commented 4 years ago

I tried on virtual machine and models doesn't work without rebooting too.

bugfloyd commented 4 years ago

I'm experiencing the same problem. I've deployed some functions using nuctl as described in the docs.

nuctl get function reports function status as ready:

  NAMESPACE |               NAME               | PROJECT | STATE | NODE PORT | REPLICAS  
  nuclio    | openvino.dextr                   | cvat    | ready |     42503 | 1/1       
  nuclio    | openvino.omz.public.yolo-v3-tf   | cvat    | ready |     36835 | 1/1       
  nuclio    | tf.faster_rcnn_inception_v2_coco | cvat    | ready |     35819 | 1/1

When I try auto annotating a task via CVAT dashboard, This error appears:

Fetching inference status for the task 1
Error: Inference status for the task 1 is failed. requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://nuclio:8070/api/function_invocations

I get this log in nuclio container:

20.09.26 11:20:09.356 dashboard.platform.docker (D) Container is healthy {"containerID": "4b7f5cc22a8f00b4d81fe2e6bd5dc60a46d6bda8e02b8c0ff0d807fe743952b1"}
20.09.26 11:20:11.133 .api/function_invocations (W) Failed to invoke function {"err": "Failed to send HTTP request", "errVerbose": "\nError - Post \"http://172.17.0.1:35819\": dial tcp 172.17.0.1:35819: i/o timeout\n    /nuclio/pkg/platform/abstract/invoker.go:119\n\nCall stack:\nFailed to send HTTP request\n    /nuclio/pkg/platform/abstract/invoker.go:119\nFailed to send HTTP request", "errCauses": [{"error": "Post \"http://172.17.0.1:35819\": dial tcp 172.17.0.1:35819: i/o timeout"}]}
20.09.26 11:20:11.133          dashboard.server (D) Handled request {"requestMethod": "POST", "requestPath": "/api/function_invocations", "requestHeaders": {"Accept":["*/*"],"Accept-Encoding":["gzip, deflate"],"Connection":["close"],"Content-Length":["177225"],"Content-Type":["application/json"],"User-Agent":["python-requests/2.24.0"],"X-Nuclio-Function-Name":["tf.faster_rcnn_inception_v2_coco"],"X-Nuclio-Function-Namespace":["nuclio"],"X-Nuclio-Log-Level":[""],"X-Nuclio-Path":["/"],"X-Nuclio-Project-Name":["cvat"],"X-Nuclio-Target":["tf.faster_rcnn_inception_v2_coco"]}, "requestBody": "{\"image\": \"iVBOgoA..........4AkJggg==\"}", "responseStatus": 500, "responseBody": "{\"error\": \"Failed to invoke function\"}", "responseTime": "30.174545501s"}

cvat container logs:

2020-09-26 13:09:37,674 DEBG 'rqworker_low' stderr output:
DEBUG:rq.worker:Handling failed execution of job 1ed2e996-87fc-4a8f-8445-0e0a61aef933

2020-09-26 13:09:37,676 DEBG 'rqworker_low' stderr output:
ERROR:rq.worker:Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/rq/worker.py", line 936, in perform_job
    rv = job.perform()
  File "/usr/local/lib/python3.5/dist-packages/rq/job.py", line 684, in perform
    self._result = self._execute()
  File "/usr/local/lib/python3.5/dist-packages/rq/job.py", line 690, in _execute
    return self.func(*self.args, **self.kwargs)
  File "/home/django/cvat/apps/lambda_manager/views.py", line 473, in __call__
    kwargs.get("threshold"), kwargs.get("mapping"))
  File "/home/django/cvat/apps/lambda_manager/views.py", line 342, in _call_detector
    "threshold": threshold})
  File "/home/django/cvat/apps/lambda_manager/views.py", line 175, in invoke
    response = self.gateway.invoke(self, payload)
  File "/home/django/cvat/apps/lambda_manager/views.py", line 81, in invoke
    'x-nuclio-path': '/'
  File "/home/django/cvat/apps/lambda_manager/views.py", line 56, in _http
    reply.raise_for_status()
  File "/usr/local/lib/python3.5/dist-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://nuclio:8070/api/function_invocations
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/rq/worker.py", line 936, in perform_job
    rv = job.perform()
  File "/usr/local/lib/python3.5/dist-packages/rq/job.py", line 684, in perform
    self._result = self._execute()
  File "/usr/local/lib/python3.5/dist-packages/rq/job.py", line 690, in _execute
    return self.func(*self.args, **self.kwargs)
  File "/home/django/cvat/apps/lambda_manager/views.py", line 473, in __call__
    kwargs.get("threshold"), kwargs.get("mapping"))
  File "/home/django/cvat/apps/lambda_manager/views.py", line 342, in _call_detector
    "threshold": threshold})
  File "/home/django/cvat/apps/lambda_manager/views.py", line 175, in invoke
    response = self.gateway.invoke(self, payload)
  File "/home/django/cvat/apps/lambda_manager/views.py", line 81, in invoke
    'x-nuclio-path': '/'
  File "/home/django/cvat/apps/lambda_manager/views.py", line 56, in _http
    reply.raise_for_status()
  File "/usr/local/lib/python3.5/dist-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://nuclio:8070/api/function_invocations

2020-09-26 13:09:37,677 DEBG 'rqworker_low' stderr output:
DEBUG:rq.worker:Invoking exception handler <function rq_handler at 0x7f3ddca79e18>

Also there is no log for nuclio-nuclio-tf.faster_rcnn_inception_v2_coco container during the process.

All three functions fail to be invoked and I get same error message.

My CVAT env details:

Server version: 1.1
Core version: 3.5.0
Canvas version: 2.0.2
UI version: 1.8.4
nmanovic commented 4 years ago

@EldarKurbanov , I will look at the issue today. Sorry for the delay with my reply.

cstamatopoulos commented 4 years ago

I'm also experiencing the same problem.

Becktor commented 4 years ago

I'm experiencing the same problem.

prateek09101996 commented 3 years ago

I am facing the same problem. My system is windows 10. I had been using cvat for a couple of months and had been using auto annotation. Now, the auto annotation stopped working while using a video annotation with a mask. Now, it completely stopped working showing the above-mentioned results. I had been using auto annotation and now it completely stopped working with any new or past task. Can anyone help me?

nmanovic commented 3 years ago

@prateek09101996 , could you please post output for a couple of commands (it is better to attach output of these commands as text files):

Let's start the investigation process together.

GoGoPen commented 3 years ago

@prateek09101996 , could you please post output for a couple of commands (it is better to attach output of these commands as text files):

  • docker logs ps
  • nuctl get function
  • docker logs cvat
  • nuctl version

Let's start the investigation process together.

@nmanovic I encountered the same error and created an issue here : https://github.com/openvinotoolkit/cvat/issues/2280#issue-717078553. I have tried to use the same version of nuctl as the docker (1.4.8) and the same error occurred.

ilouzl commented 3 years ago

I also experience the same issue, but in my case it happens only with certain models.

  NAMESPACE |                     NAME                     | PROJECT | STATE | NODE PORT | REPLICAS  
  nuclio    | openvino.omz.public.yolo-v3-tf               | cvat    | ready |     32768 | 1/1       
  nuclio    | openvino.omz.semantic-segmentation-adas-0001 | cvat    | ready |     32808 | 1/1

yolo-v3-tf works great but semantic-segmentation-adas-0001 raises that invoke error.

prateek09101996 commented 3 years ago

@prateek09101996 , could you please post output for a couple of commands (it is better to attach output of these commands as text files):

  • docker logs ps
  • nuctl get function
  • docker logs cvat
  • nuctl version

Let's start the investigation process together.

@nmanovic I encountered the same error and created an issue here : #2280 (comment). I have tried to use the same version of nuctl as the docker (1.4.8) and the same error occurred.

While trying to fix the problem, I somehow tried updating the docker and lost every image I had on the machine. Therefore, I could not provide you all with the necessary logs. While reinstalling from scratch, I found the models is missing in my new id.

belorenz commented 3 years ago

@prateek09101996 , could you please post output for a couple of commands (it is better to attach output of these commands as text files):

* `docker logs ps`

* `nuctl get function`

* `docker logs cvat`

* `nuctl version`

Having the same issue on both ubuntu 16 and 18. Here are my logs:

https://pastebin.com/btHkaWSp

borgarpa commented 3 years ago

Hi!

I left a comment in a related issue with some info regarding the same error: 2280#issuecomment

This error prompted when using f-BRS:

Interaction error occured
Error: Request failed with status code 500.
"500 Server Error: Internal Server Error for url: http://nuclio:8070/api/function_invocations".
belorenz commented 3 years ago

This is still an issue. Is there any way to use an older release in which automatic annotation is working?

fkreh commented 3 years ago

I got the same issue.

Setup:

Cvat and nuclio is running on system 1. The annotations should be done from several machines in the network.

  NAMESPACE |       NAME        | PROJECT | STATE | NODE PORT | REPLICAS
  nuclio    | pth-saic-vul-fbrs | cvat    | error |     32768 | 1/1

services: cvat_proxy: environment: CVAT_HOST: ubuntu-ki


When I try `ping nucleo`, I get the message `ping: nuclio: The name or service is not known`. It doesn't matter if i try to ping from the cvat-machine or any other system in the network.

The nucleo dashboard is reachable with the same url like the cvat dashboard. For my setup it's `ubuntu-ki`.
turowicz commented 3 years ago

I'm having this problem with CVAT but the discussion is being held here:

https://github.com/nuclio/nuclio/issues/2031

gachiemchiep commented 3 years ago

I also have this issue. I tried "develop" and "v1.2.0" branch but both have the same error.

image

The annotation for the first image is generated, but it is failed for the rest. I guess some code after draw annotation to image failed to execute.

ZahraAnam commented 3 years ago

Response to instructed commands to debug

Error500_Debug_Logs.txt

ZahraAnam commented 3 years ago

I can upload model but I don't get inferences But I didn't install or get the Cvat working via deploy.sh also am using Ubuntu 20.04

ZahraAnam commented 3 years ago

docker ps -a results in following :

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES dd73debf5caa nginx:stable-alpine "/docker-entrypoint.…" 15 minutes ago Up 15 minutes 0.0.0.0:8080->80/tcp cvat_proxy 03a7025a733b openvino/cvat_ui "/docker-entrypoint.…" 15 minutes ago Up 15 minutes 80/tcp cvat_ui 546b27475b12 openvino/cvat_server "/usr/bin/supervisord" 15 minutes ago Up 15 minutes 8080/tcp cvat 97e321b000e5 postgres:10-alpine "docker-entrypoint.s…" 16 minutes ago Up 15 minutes 5432/tcp cvat_db d393a04795b8 redis:4.0-alpine "docker-entrypoint.s…" 16 minutes ago Up 15 minutes 6379/tcp cvat_redis ecb25738ee11 quay.io/nuclio/dashboard:1.5.16-amd64 "/docker-entrypoint.…" 16 minutes ago Up 15 minutes (healthy) 80/tcp, 0.0.0.0:8070->8070/tcp nuclio 15d24c1a3cda alpine:3.11 "/bin/sh -c '/bin/sl…" 25 minutes ago Up 25 minutes nuclio-local-storage-reader 16eafa9c88a6 cvat/openvino.omz.public.yolo-v3-tf:latest "processor" 19 hours ago Exited (137) 17 minutes ago nuclio-nuclio-openvino-omz-public-yolo-v3-tf f8d4291ebfdc cvat/openvino.dextr:latest "processor" 19 hours ago Exited (137) 17 minutes ago nuclio-nuclio-openvino-dextr 6df6ac7d1552 quay.io/nuclio/dashboard:stable-amd64 "/docker-entrypoint.…" 19 hours ago Created nuclio-dashboard e5552f482d78 ubuntu "/bin/bash" 3 months ago Exited (0) 3 months ago wonderful_gauss 3460e4bb009b nuclio/processor-dates "processor" 3 months ago Exited (1) 3 months ago pedantic_greider 07d2fff97ac4 ubuntu "bash" 3 months ago Exited (129) 3 months ago practical_meitner ef3d348008c1 ubuntu "basg" 3 months ago Created adoring_roentgen 2f49d9dd987b hello-world "/hello" 3 months ago Exited (0) 3 months ago modest_galileo 6ba717e6d07e hello-world "/hello" 3 months ago Exited (0) 3 months ago dreamy_diffie

duanyao commented 3 years ago

I experienced the same issue, and managed to workaround it. In short: make sure the interface docker0 has address 172.17.0.1.

I also got "i/o timeout" in nuclio logs:

21.09.03 08:45:50.191 .api/function_invocations (W) Failed to invoke function {"err": "Failed to send HTTP request", "errVerbose": "\n
Error - Post \"http://172.17.0.1:49158\": dial tcp 172.17.0.1:49158: i/o timeout\n
    /nuclio/pkg/platform/abstract/invoker.go:119\n\nCall stack:\nFailed to send HTTP request\n
    /nuclio/pkg/platform/abstract/invoker.go:119\n
Failed to send HTTP request", "errCauses": [{"error": "Post \"http://172.17.0.1:49158\": dial tcp 172.17.0.1:49158: i/o timeout"}]}

172.17.0.1 is the default IP address of interface docker0. However, docker0 in my machine had 172.18.0.1 instead:

# ifconfig
br-380064f8c183: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.20.0.1  netmask 255.255.0.0  broadcast 172.20.255.255
...

docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.18.0.1  netmask 255.255.0.0  broadcast 172.18.255.255
...

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.191.220  netmask 255.255.240.0  broadcast 172.17.191.255
...

This was because my physical NIC eth0 was already in 172.17.0.0/16 so docker used 172.18.0.0/16. However CVAT or nuclio insisted to access 172.17.0.1 , so timeout happened.

I changed IP address of eth0 to 172.29.0.183, then docker0 to 172.17.0.1, and restarted CVAT, and this issue was resolved.

rohitsaluja22 commented 3 years ago

I experienced the same issue, and managed to workaround it. In short: make sure the interface docker0 has address 172.17.0.1.

I also got "i/o timeout" in nuclio logs:

21.09.03 08:45:50.191 .api/function_invocations (W) Failed to invoke function {"err": "Failed to send HTTP request", "errVerbose": "\n
Error - Post \"http://172.17.0.1:49158\": dial tcp 172.17.0.1:49158: i/o timeout\n
    /nuclio/pkg/platform/abstract/invoker.go:119\n\nCall stack:\nFailed to send HTTP request\n
    /nuclio/pkg/platform/abstract/invoker.go:119\n
Failed to send HTTP request", "errCauses": [{"error": "Post \"http://172.17.0.1:49158\": dial tcp 172.17.0.1:49158: i/o timeout"}]}

172.17.0.1 is the default IP address of interface docker0. However, docker0 in my machine had 172.18.0.1 instead:

# ifconfig
br-380064f8c183: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.20.0.1  netmask 255.255.0.0  broadcast 172.20.255.255
...

docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.18.0.1  netmask 255.255.0.0  broadcast 172.18.255.255
...

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.191.220  netmask 255.255.240.0  broadcast 172.17.191.255
...

This was because my physical NIC eth0 was already in 172.17.0.0/16 so docker used 172.18.0.0/16. However CVAT or nuclio insisted to access 172.17.0.1 , so timeout happened.

I changed IP address of eth0 to 172.29.0.183, then docker0 to 172.17.0.1, and restarted CVAT, and this issue was resolved.

I used this commend $ /sbin/ifconfig to get ip of docker0, Its already 172.17.0.1 in my case. Still getting the following error while using automatic annotations:-

Tracking error occured Error: Request failed with status code 500. "\n<!doctype html>\n<html lang=\"en\">\n\n Server Error (500)\n\n\n

Server Error (500)

\n\n\n".

duanyao commented 3 years ago

I used this commend $ /sbin/ifconfig to get ip of docker0, Its already 172.17.0.1 in my case. Still getting the following error while using automatic annotations:-

Tracking error occured Error: Request failed with status code 500. "\n<!doctype html>\n\n\n Server Error (500)\n\n\n

Server Error (500)

\n\n\n".

It seems it was caused by another problem. You may follow the logs in container nuclio:

docker logs --follow nuclio --tail=1

and use automatic annotation again.

c00lcoder commented 3 years ago

Any solution for this yet?

KiirkH commented 2 years ago

Faced same issue recently. After digging around investigated that processors are deployed not to the cvat_cvat network (where main 'staff' is located) and UFW 'kindly' blocked all requests from nuclio container to them. My dirty hotfix with room for security improvement is below:

# Add docker0 network adapter address to the UFW exceptions
$ sudo ufw allow proto tcp from any to 172.17.0.1
efcy commented 2 years ago

Another thing you can do is to change the function.yaml files to explicitely use the correct network. In this case cvat_cvat:

Change the following code:

  platform:
    attributes:
      restartPolicy:
        name: always
        maximumRetryCount: 3
      mountMode: volume

to

  platform:
    attributes:
      restartPolicy:
        name: always
        maximumRetryCount: 3
      mountMode: volume
      network: cvat_cvat

After this i could invoke the function correctly from the dashboard. But from a cvat task I still got the same 500 error as before. That because using the nuclio like cvat does somehow still leads to the post request being send to 172.17.0.1.

I am not sure why. I will further debug this and add logs to this thread.

f4z3k4s commented 2 years ago

I've been debugging this exact same issue for like 10 hours, just happened to solve it. The strange thing is that it used to work, but after a docker system prune, it stopped working with the same configurations. I am using an external network, and for me, specifying the name of that external network in platform.attributes.network like in the above comment solved this issue.

ZahraAnam commented 2 years ago

We just switched domain and changed fileshare.

On Tue, Oct 26, 2021, 8:09 PM Fazekas Dániel @.***> wrote:

I've been debugging this exact same issue for like 10 hours, just happened to solve it. The strange thing is that it used to work, but after a docker system prune, it stopped working with the same configurations. I am using an external network, and for me, specifying the name of that external network in platform.attributes.network like in the above comment solved this issue.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/openvinotoolkit/cvat/issues/2189#issuecomment-952185796, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIIJTCSMOS5632LKEPLNRA3UI34E5ANCNFSM4RPH4PNQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

f4z3k4s commented 2 years ago

We just switched domain and changed fileshare. … On Tue, Oct 26, 2021, 8:09 PM Fazekas Dániel @.***> wrote: I've been debugging this exact same issue for like 10 hours, just happened to solve it. The strange thing is that it used to work, but after a docker system prune, it stopped working with the same configurations. I am using an external network, and for me, specifying the name of that external network in platform.attributes.network like in the above comment solved this issue. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2189 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIIJTCSMOS5632LKEPLNRA3UI34E5ANCNFSM4RPH4PNQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

@ZahraAnam That doesn't say much to me, can this behaviour be related to that?

efcy commented 2 years ago

@f4z3k4s does setting the platform.attributes.network solved the problem of running auto annotation from the task page for you? If so can you give me some details about your network and which nuclio version you are using?

f4z3k4s commented 2 years ago

@StellaASchlotter Yes, it works from the task page as well. I am using the default quay.io/nuclio/dashboard:1.5.16-amd64 image and the corresponding cli version. My network is an external network, created with docker network create mynetwork. With docker inspect, I was able to see that the deployed function was on the default bridge network. I initially tried to solve the issue by connecting my deployed function to mynetwork with docker network connect but that didn't solve the problem. Specifying platform.attributes.network did in function.yaml.

Please note that nuclio and all other cvat services belong to mynetwork in my setup, I specified this in my docker-compose.serverless.yml:

networks:
  mynetwork:
    external:
      name: mynetwork

and added: 

networks:
  - mynetwork # (replace cvat with your network)
qkrwlgns0605 commented 2 years ago

@f4z3k4s Thank you for your solution! But I tried your solution, I still get 500 error. The source code that I changed after cloning is as follows.

I did 'docker network create mynetwork'

- docker-compose.serverless.yml

version: '3.3' services: nuclio: container_name: nuclio image: quay.io/nuclio/dashboard:1.5.16-amd64 restart: always networks:

volumes: cvat_events:

networks: mynetwork: external: name: mynetwork

And the result of running docker-compose -f docker-compose.yml -f components/serverless/docker-compose.serverless.yml up -d --build , models are also captured in localhost:8080/models, and the running state is also displayed in localhost:8070. But in web server and cli test, function invoke still fails. Are there any problems with my process? Thank you for your reply.

f4z3k4s commented 2 years ago

@qkrwlgns0605 your setup can be a bit different than mine, but for sure, what I see already wrong in docker-compose.serverless.yml is this:

networks:
  - mynetwork # (replace cvat with your network)

And also, in my environment, all other cvat services run on mynetwork.

qkrwlgns0605 commented 2 years ago

@f4z3k4s Hi. Thank you for your reply. I'm sorry I keep calling you. I modified docker-compose.serverless.yml and function.yaml as you said, and also modified docker-compose.yml as follows.

version: '3.3'

services: cvat_db: container_name: cvat_db image: postgres:10-alpine restart: always environment: POSTGRES_USER: root POSTGRES_DB: cvat POSTGRES_HOST_AUTH_METHOD: trust volumes:

volumes: cvat_db: cvat_data: cvat_keys: cvat_logs:

networks: mynetwork: external: name: mynetwork

When I executed the 'docker inspect mynetwork' command, I checked that they were all in one network, but I still get a 500 internal server error when I request annotations from the web. Are there any other issues that need to be fixed, such as firewall or host settings?

f4z3k4s commented 2 years ago

@qkrwlgns0605 The above detailed steps solved the issue for me, I configured no firewall or host settings. I have no clue if it is related, but I created my network with docker network create mynetwork --subnet=192.168.48.0/20 --gateway=192.168.48.1 with a specified subnet and gateway, the default is 172.18.x.x. Maybe give this a try and put everything in this network? Sorry, I am no author of this repository, I just shared what helped me solved this issue in my specific case, I am afraid I can help no further.

qkrwlgns0605 commented 2 years ago

@f4z3k4s Thank you. Your solution didn't work, but I did it the other way around. Perhaps, if only Linux is used, 'host.docker.internal' is not supported, so communication does not seem to work properly, so I installed Linux on Windows, set the port of the function and deployed it in Linux, and it worked properly. I refer to the tutorial part of the following url. https://github.com/openvinotoolkit/cvat/pull/3124 Thanks for your reply!!

jenhaoyang commented 2 years ago

Faced same issue recently. After digging around investigated that processors are deployed not to the cvat_cvat network (where main 'staff' is located) and UFW 'kindly' blocked all requests from nuclio container to them. My dirty hotfix with room for security improvement is below:

# Add docker0 network adapter address to the UFW exceptions
$ sudo ufw allow proto tcp from any to 172.17.0.1

This work for me thank you!!

PallaviBhargava commented 2 years ago

any suggestion for this issue?? I am also facing same issue

ZahraAnam commented 2 years ago

It mostly happens if there is some problem with your annotations file For instance missing a class or extra classes or type issue

On Thu, Dec 23, 2021, 1:04 PM PallaviBhargava @.***> wrote:

any suggestion for this issue?? I am also facing same issue

— Reply to this email directly, view it on GitHub https://github.com/openvinotoolkit/cvat/issues/2189#issuecomment-1000258448, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIIJTCUIW4NUS5PKSSLWQVTUSMF3XANCNFSM4RPH4PNQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

PallaviBhargava commented 2 years ago

It mostly happens if there is some problem with your annotations file

I tried everything but still I am getting this issue:

Error: Inference status for the task 4 is failed. requests.exceptions. HTTPError: 500 Server Error: Internal Server Error for url: http://nuclio:8070/api/function_invocations

ZahraAnam commented 2 years ago

What are you trying to do?

On Fri., Dec. 24, 2021, 9:16 a.m. PallaviBhargava @.***> wrote:

It mostly happens if there is some problem with your annotations file

I tried everything but still I am getting this issue:

Error: Inference status for the task 4 is failed. requests.exceptions. HTTPError: 500 Server Error: Internal Server Error for url: http://nuclio:8070/api/function_invocations

— Reply to this email directly, view it on GitHub https://github.com/openvinotoolkit/cvat/issues/2189#issuecomment-1000716983, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIIJTCWS6Q3QG3G5FJS53B3USQT4VANCNFSM4RPH4PNQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

PallaviBhargava commented 2 years ago

I tried everything but still I am getting this issue:

I am trying different models for auto-annotation in my local pc in linux. I built cvat using instruction provided in manual. Manual labelling is working fine but not auto-annotation. I keep getting this error : Error: Inference status for the task 4 is failed. requests.exceptions. HTTPError: 500 Server Error: Internal Server Error for url: http://nuclio:8070/api/function_invocations

ZahraAnam commented 2 years ago

Can you share files with me? Plus can you direct message me please On Fri., Dec. 24, 2021, 10:42 a.m. PallaviBhargava @.***> wrote:

I tried everything but still I am getting this issue:

I am trying different models for auto-annotation in my local pc in linux. I built cvat using instruction provided in manual. Manual labelling is working fine but not auto-annotation. I keep getting this error : Error: Inference status for the task 4 is failed. requests.exceptions. HTTPError: 500 Server Error: Internal Server Error for url: http://nuclio:8070/api/function_invocations

— Reply to this email directly, view it on GitHub https://github.com/openvinotoolkit/cvat/issues/2189#issuecomment-1000755449, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIIJTCR4KNEHWW7JHZWYIPLUSQ57FANCNFSM4RPH4PNQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

jlcd commented 2 years ago

Unfortunately I'm going through the same issue and have no clue on how to fix it.

image

ZahraAnam commented 2 years ago

Generally it happens coz of problem in annotation file. Either file is incomplete or the coding of file is not appropriate. Sent from my Huawei phone-------- Original message --------From: João Drummond @.>Date: Tue., Jan. 4, 2022, 8:41 a.m.To: openvinotoolkit/cvat @.>Cc: ZahraAnam @.>, Mention @.>Subject: Re: [openvinotoolkit/cvat] Automatic annotation 500 ServerError (#2189) Unfortunately I'm going through the same issue and have no clue on how to fix it.

—Reply to this email directly, view it on GitHub, or unsubscribe.Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you were mentioned.Message ID: @.***>

jlcd commented 2 years ago

Not really, @ZahraAnam, my issue seems related to this: https://github.com/nuclio/nuclio/issues/2158#issuecomment-1004542016

But I still can't fix it.

ZahraAnam commented 2 years ago

I haven't used nuclio

On Tue., Jan. 4, 2022, 10:45 a.m. João Drummond @.***> wrote:

Not really, @ZahraAnam https://github.com/ZahraAnam, my issue seems related to this: nuclio/nuclio#2158 (comment) https://github.com/nuclio/nuclio/issues/2158#issuecomment-1004542016

But I still can't fix it.

— Reply to this email directly, view it on GitHub https://github.com/openvinotoolkit/cvat/issues/2189#issuecomment-1004543386, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIIJTCQNU6LBLKWDTHAOJNDUUKCQXANCNFSM4RPH4PNQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

yeahydq commented 2 years ago

here is my solution, see if it helps for you. Updating the function.yaml like below

image
efcy commented 2 years ago

If you changed your docker daemon config usually located under /etc/docker/daemon.json to something like this

{
    "default-address-pools" : [
    {
      "base" : "172.240.0.0/16",
      "size" : 24
    }
    ]
}

then the default gateway is 172.240.0.1. However nuclio dashboard expects it to be 172.17.0.1. See the hardcoded values in https://github.com/nuclio/nuclio/blob/5e58e14fef02b4cdd466cb59b14f3fede5a0864b/pkg/platform/local/platform.go#L632 and https://github.com/nuclio/nuclio/blob/5e58e14fef02b4cdd466cb59b14f3fede5a0864b/pkg/platform/local/platform.go#L656

When you run your inference from cvat ui, cvat will make a request to http://nuclio:8070. This request is converted by the nuclio dashboard to :nuclio_function_port.

I had to rebuild the nuclio docker image after changing the hardcoded values and specified the new image in the https://github.com/openvinotoolkit/cvat/blob/develop/components/serverless/docker-compose.serverless.yml

The reason why I had to change my docker config is simply that it clashes with my institutes vpn. Since I can't change the vpn config I changed the docker one.

I hope it helps someone. Also it would be interesting to know how many people still have problems with nuclio.Maybe one day we can close this issue :smile:

rondondaniel commented 2 years ago

Wrap solution up:

  1. Modify functions yaml as @yeahydq did above: https://user-images.githubusercontent.com/18577463/152193385-f4e661ec-a617-44b5-a6f3-1747a7a62f35.png
  2. then just run: sudo ufw allow proto tcp from any to 172.17.0.1 as @KiirkH mentioned above. Yes, a security issue could rise from this.

These steps worked for me.

Thanks!