regel / loudml

Loud ML is the first open-source AI solution for ICT and IoT automation
Other
298 stars 93 forks source link

[Error] - socket.timeout: timed out | During handling of the above exception, another exception occurred | TypeError: __init__() takes from 1 to 2 positional arguments but 4 were given #386

Open toni-moreno opened 4 years ago

toni-moreno commented 4 years ago

Hi @regel , I have one loudml server working with 7 models ( testing which one better fit the metric shape).

Sometimes , in the log the following message is appearing, and I can not understand what is happening and what model is affected and what is the affectation. Could you help me please?

WARNING:root:detected anomaly for model 'swarm@cpu@mean@usage_active@host_worker2_cpu_cpu-total@time@5m' at 2020-08-05T15:45:00.000Z (score = 100.0)
 INFO:root:connecting to influxdb on influxdb:8086, using database 'loudml'
 INFO:root:connecting to influxdb on influxdb:8086, using database 'loudml'
 INFO:root:job[1c88bd07-40c6-4500-8a42-1959e059f463] done. Took 0:00:00.947908
     return self._sock.recv_into(b)
 socket.timeout: timed out

 During handling of the above exception, another exception occurred:

 Traceback (most recent call last):
   File "/opt/venv/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
     timeout=timeout
   File "/opt/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 720, in urlopen
     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
   File "/opt/venv/lib/python3.7/site-packages/urllib3/util/retry.py", line 400, in increment
     raise six.reraise(type(error), error, _stacktrace)
   File "/opt/venv/lib/python3.7/site-packages/urllib3/packages/six.py", line 735, in reraise
     raise value
   File "/opt/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 672, in urlopen
     chunked=chunked,
   File "/opt/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 423, in _make_request
     self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
   File "/opt/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 331, in _raise_timeout
     self, url, "Read timed out. (read timeout=%s)" % timeout_value
 urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='localhost', port=8077): Read timed out. (read timeout=5)

 During handling of the above exception, another exception occurred:

 Traceback (most recent call last):
   File "/opt/venv/lib/python3.7/site-packages/loudml/requests.py", line 32, in perform_request
     response = session.send(prepared_request, **send_kwargs)
   File "/opt/venv/lib/python3.7/site-packages/requests/sessions.py", line 643, in send
     r = adapter.send(request, **kwargs)
   File "/opt/venv/lib/python3.7/site-packages/requests/adapters.py", line 529, in send
     raise ReadTimeout(e, request=request)
 requests.exceptions.ReadTimeout: HTTPConnectionPool(host='localhost', port=8077): Read timed out. (read timeout=5)

 During handling of the above exception, another exception occurred:

 Traceback (most recent call last):
   File "/opt/venv/lib/python3.7/site-packages/loudml/server.py", line 105, in wrapper
     return job_func(*args, **kwargs)
   File "/opt/venv/lib/python3.7/site-packages/loudml/server.py", line 196, in daemon_exec_scheduled_job
     headers=None,
   File "/opt/venv/lib/python3.7/site-packages/loudml/requests.py", line 37, in perform_request
     raise errors.ConnectionTimeout('TIMEOUT', str(e), e)
 TypeError: __init__() takes from 1 to 2 positional arguments but 4 were given

 Traceback (most recent call last):
   File "/opt/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 421, in _make_request
     six.raise_from(e, None)
   File "<string>", line 3, in raise_from
   File "/opt/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 416, in _make_request
     httplib_response = conn.getresponse()
   File "/usr/local/lib/python3.7/http/client.py", line 1354, in getresponse
     response.begin()
   File "/usr/local/lib/python3.7/http/client.py", line 306, in begin
     version, status, reason = self._read_status()
   File "/usr/local/lib/python3.7/http/client.py", line 267, in _read_status
     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
   File "/usr/local/lib/python3.7/socket.py", line 589, in readinto
     return self._sock.recv_into(b)
 socket.timeout: timed out

 During handling of the above exception, another exception occurred:

 Traceback (most recent call last):
   File "/opt/venv/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
     timeout=timeout
   File "/opt/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 720, in urlopen
     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
   File "/opt/venv/lib/python3.7/site-packages/urllib3/util/retry.py", line 400, in increment
     raise six.reraise(type(error), error, _stacktrace)
   File "/opt/venv/lib/python3.7/site-packages/urllib3/packages/six.py", line 735, in reraise
     raise value
   File "/opt/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 672, in urlopen
     chunked=chunked,
   File "/opt/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 423, in _make_request
     self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
   File "/opt/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 331, in _raise_timeout
     self, url, "Read timed out. (read timeout=%s)" % timeout_value
 urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='localhost', port=8077): Read timed out. (read timeout=5)

 During handling of the above exception, another exception occurred:

 Traceback (most recent call last):
   File "/opt/venv/lib/python3.7/site-packages/loudml/requests.py", line 32, in perform_request
     response = session.send(prepared_request, **send_kwargs)
   File "/opt/venv/lib/python3.7/site-packages/requests/sessions.py", line 643, in send
     r = adapter.send(request, **kwargs)
   File "/opt/venv/lib/python3.7/site-packages/requests/adapters.py", line 529, in send
     raise ReadTimeout(e, request=request)
 requests.exceptions.ReadTimeout: HTTPConnectionPool(host='localhost', port=8077): Read timed out. (read timeout=5)

 During handling of the above exception, another exception occurred:

 Traceback (most recent call last):
   File "/opt/venv/lib/python3.7/site-packages/loudml/server.py", line 105, in wrapper
     return job_func(*args, **kwargs)
   File "/opt/venv/lib/python3.7/site-packages/loudml/server.py", line 196, in daemon_exec_scheduled_job
     headers=None,
   File "/opt/venv/lib/python3.7/site-packages/loudml/requests.py", line 37, in perform_request
     raise errors.ConnectionTimeout('TIMEOUT', str(e), e)
 TypeError: __init__() takes from 1 to 2 positional arguments but 4 were given
regel commented 4 years ago

Hi Toni, it seems like the code is trying to raise an exception after a timeout. Some argument may be missing in the exception constructor. Timeouts could indicate some kind of overload. What are interval and bucket_interval values?

toni-moreno commented 4 years ago

CPU on the server is about 60% ..

There is any way to configure the timeout?