Cannot start Automatic annotation, Max retries exceeded with url: /

paddy0489 commented 2 years ago

Browser Console:

cvat-app.tsx:284 Error: Inference status for the task 396 is failed. requests.exceptions.ConnectionError: HTTPConnectionPool(host='host.docker.internal', port=32001): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77d9292160>: Failed to establish a new connection: [Errno -2] Name or service not known'))

Docker: cvat_worker_low

requests.exceptions.ConnectionError: HTTPConnectionPool(host='host.docker.internal', port=32001): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77d9292160>: Failed to establish a new connection: [Errno -2] Name or service not known'))
Traceback (most recent call last):
  File "/opt/venv/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/opt/venv/lib/python3.8/site-packages/urllib3/util/connection.py", line 72, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/lib/python3.8/socket.py", line 918, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/opt/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/opt/venv/lib/python3.8/site-packages/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/lib/python3.8/http/client.py", line 1256, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output
    self.send(msg)
  File "/usr/lib/python3.8/http/client.py", line 951, in send
    self.connect()
  File "/opt/venv/lib/python3.8/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/opt/venv/lib/python3.8/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f77d9292160>: Failed to establish a new connection: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/venv/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/opt/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "/opt/venv/lib/python3.8/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='host.docker.internal', port=32001): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77d9292160>: Failed to establish a new connection: [Errno -2] Name or service not known'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/venv/lib/python3.8/site-packages/rq/worker.py", line 936, in perform_job
    rv = job.perform()
  File "/opt/venv/lib/python3.8/site-packages/rq/job.py", line 684, in perform
    self._result = self._execute()
  File "/opt/venv/lib/python3.8/site-packages/rq/job.py", line 690, in _execute
    return self.func(*self.args, **self.kwargs)
  File "/home/django/cvat/apps/lambda_manager/views.py", line 638, in __call__
    LambdaJob._call_detector(function, db_task, labels, quality,
  File "/home/django/cvat/apps/lambda_manager/views.py", line 488, in _call_detector
    annotations = function.invoke(db_task, data={
  File "/home/django/cvat/apps/lambda_manager/views.py", line 265, in invoke
    response = self.gateway.invoke(self, payload)
  File "/home/django/cvat/apps/lambda_manager/views.py", line 96, in invoke
    reply = requests.post(url, timeout=NUCLIO_TIMEOUT, json=payload)
  File "/opt/venv/lib/python3.8/site-packages/requests/api.py", line 117, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/opt/venv/lib/python3.8/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/venv/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/venv/lib/python3.8/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/opt/venv/lib/python3.8/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='host.docker.internal', port=32001): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77d9292160>: Failed to establish a new connection: [Errno -2] Name or service not known'))

Steps to Reproduce (for bugs)

problems with NAT in previous version: sudo iptables -t nat -I PREROUTING 1 -d 172.17.0.1 -j DNAT --to 172.240.0.1 fix
Automatic Annotation works.
checkout newest version on develop
start Automatic Annotation on nuclio
Error: Fetching inference status for the task xyz

Your Environment

Git hash commit (git log -1): commit 4167a4e40c8cca2adcb9ebfcd40d859bfcebc5d9 (HEAD -> develop, origin/develop, origin/HEAD)
Docker version : 20.10.17

Operating System and version (e.g. Linux, Windows, MacOS):

Linux  5.15.0-46-generic #49~20.04.1-Ubuntu SMP Thu Aug 4 19:15:44 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.5 LTS
Release:    20.04
Codename:   focal

azhavoro commented 2 years ago

Hi, What is output of docker exec cvat_worker_low bash -c 'cat /etc/hosts'? Most likely the settings for this service are missing, try adding the following lines for cvat_worker_low service

  cvat_worker_low:
    extra_hosts:
      - "host.docker.internal:host-gateway"

MikoyChinese commented 2 years ago

@azhavoro

Also get this error:

Fetching inference status for the [task 60
Error: Inference status for the task 60 is failed. requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://host.docker.internal:49371/

MikoyChinese commented 2 years ago

Ooh, this my code error, my model return a error value to function handler, sorry.

  cvat_worker_low:
    extra_hosts:
      - "host.docker.internal:host-gateway"

This work for me. Thx.

MrRe0 commented 2 years ago

Ooh, this my code error, my model return a error value to function handler, sorry.
  cvat_worker_low:
    extra_hosts:
      - "host.docker.internal:host-gateway"
This work for me. Thx.

hi,where to modify this file? it's in the container of 'cvat_worker_low' or other? but it's likely that the fiels of the container will be restored when reboot the container

MrRe0 commented 2 years ago

@MikoyChinese I have the same error logs

MikoyChinese commented 2 years ago

@MrRe0 You can create a new docker compose file as override such as docker-compose.override.yml, then add some code like this:

version: '3.3'

services:
  cvat_worker_low:
    extra_hosts:
      - "host.docker.internal:host-gateway"

then run this:

docker compose -f docker-compose.yml \
    -f components/serverless/docker-compose.serverless.yml \
    -f docker-compose.override.yml up -d

Finally, you need to clean your browser cookies to visit your host.

If you run this on develop branch, don't forget add CVAT_VERSION=dev in your command enviroment.

MrRe0 commented 2 years ago

@MikoyChinese thanks a lot, now it works.

paddy0489 commented 2 years ago

  cvat_worker_low:
    extra_hosts:
      - "host.docker.internal:host-gateway"

it works for me, Thank you :)

hoangdung3498 commented 11 months ago

Ooh, this my code error, my model return a error value to function handler, sorry.
  cvat_worker_low:
    extra_hosts:
      - "host.docker.internal:host-gateway"
This work for me. Thx.

i have same error with you :

Tracker initialization error Request failed with status code 500. "500 Server Error: Internal Server Error for url: http://host.docker.internal:32769/".

But add extra_hosts to cvat_worker_low not work for me. Please help.

cvat-ai / cvat

Cannot start Automatic annotation, Max retries exceeded with url: / #4904

Steps to Reproduce (for bugs)

Your Environment