search

nginxinc / nginx-amplify-agent

NGINX Amplify Agent
BSD 2-Clause "Simplified" License
23 stars 9 forks source link

bridge_manager failed to push data due to TypeError #79

Open tkriviradev opened 2 years ago

tkriviradev commented 2 years ago

Hello,

I have exactly the same issues it happen after debian 10 > 11 upgrade What I tried so far, I completely purged all old amplify agent configs directories and re-install.

Using how to (agent was reinstalled), after initial one min, agent start throwing error not able to reach https://receiver.amplify.nginx.com:443

Versions:

dpkg-query -l |grep nginx
nginx                                 1.20.2-1~buster                amd64        high performance web server
nginx-amplify-agent                   1.8.0-1~bullseye               amd64        Agent for NGINX Amplify monitoring platform
nginx-common                          1.10.3-1+deb9u3                all          small, powerful, scalable web/proxy server - common files
# Python
root@nginx:~# python3 --version
Python 3.9.2

# OS
 cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"

# Kernel
root@nginx:~# uname -a
Linux nginx 5.10.0-13-amd64 nginxinc/nginx-amplify-agent-priv#1 SMP Debian 5.10.106-1 (2022-03-17) x86_64 GNU/Linux
2022-03-28 12:24:54,721 [9328] nginx_metrics [None] get http://127.0.0.1:80/nginx_status 200 10 106 0.004
2022-03-28 12:24:54,721 [9328] nginx_metrics 34d39fc314628b9ff20bfb12138a4c0cb407d6395fa5512cef844b95d9467663 collect in 0.006
2022-03-28 12:24:54,831 [9328] nginx_meta 34d39fc314628b9ff20bfb12138a4c0cb407d6395fa5512cef844b95d9467663 collect in 0.075
2022-03-28 12:24:55,016 [9328] supervisor system objects: ['e5389f19c285e6b27ce9f3cb150d4a97bad0ab4fa472deff1a5cc5c7194c6ae7']
2022-03-28 12:24:55,033 [9328] supervisor ps nginx output: ['   7533       1 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf', '   7534    7533 nginx: worker process', '   7535    7533 nginx: worker process', '   7536    7533 nginx: cache manager process', '']
2022-03-28 12:24:55,043 [9328] supervisor nginx objects: ['34d39fc314628b9ff20bfb12138a4c0cb407d6395fa5512cef844b95d9467663']
2022-03-28 12:24:55,044 [9328] supervisor status objects: []
2022-03-28 12:24:55,044 [9328] supervisor api objects: []
2022-03-28 12:24:55,674 [9328] bridge_manager failed POST "https://receiver.amplify.nginx.com:443/1.4/XXXXXXXXXXXXX/update/", exception: "HTTPSConnectionPool(host='receiver.amplify.nginx.com', port=443): Read timed out. (read timeout=7.0)"
2022-03-28 12:24:55,675 [9328] bridge_manager 
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 1012, in _validate_conn
    conn.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 411, in connect
    self.sock = ssl_wrap_socket(
  File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3/dist-packages/gevent/_ssl3.py", line 114, in wrap_socket
    return self.sslsocket_class(
  File "/usr/lib/python3/dist-packages/gevent/_ssl3.py", line 312, in __init__
    raise x
  File "/usr/lib/python3/dist-packages/gevent/_ssl3.py", line 308, in __init__
    self.do_handshake()
  File "/usr/lib/python3/dist-packages/gevent/_ssl3.py", line 672, in do_handshake
    self._wait(self._read_event, timeout_exc=_SSLErrorHandshakeTimeout)
  File "src/gevent/_hub_primitives.py", line 317, in gevent._gevent_c_hub_primitives.wait_on_socket
  File "src/gevent/_hub_primitives.py", line 322, in gevent._gevent_c_hub_primitives.wait_on_socket
  File "src/gevent/_hub_primitives.py", line 313, in gevent._gevent_c_hub_primitives._primitive_wait
  File "src/gevent/_hub_primitives.py", line 314, in gevent._gevent_c_hub_primitives._primitive_wait
  File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_hub_primitives.py", line 55, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_waiter.py", line 151, in gevent._gevent_c_waiter.Waiter.get
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 65, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_gevent_c_greenlet_primitives.pxd", line 35, in gevent._gevent_c_greenlet_primitives._greenlet_switch
socket.timeout: The handshake operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 532, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/lib/python3/dist-packages/six.py", line 719, in reraise
    raise value
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 385, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=conn.timeout)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 336, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='receiver.amplify.nginx.com', port=443): Read timed out. (read timeout=7.0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/amplify/agent/common/util/http.py", line 94, in make_request
    r = self.session.post(
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 590, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 529, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='receiver.amplify.nginx.com', port=443): Read timed out. (read timeout=7.0)
2022-03-28 12:24:55,765 [9328] bridge_manager 
2022-03-28 12:24:55,766 [9328] bridge_manager [None] post https://receiver.amplify.nginx.com:443/1.4/5147cd2d2fd569c76dbe47a18e2de225/update/ 500 1447 0 7.809
2022-03-28 12:24:55,766 [9328] bridge_manager http delay set to 10 (fails: 1)
2022-03-28 12:24:55,766 [9328] bridge_manager failed to push data due to ReadTimeout
2022-03-28 12:24:55,766 [9328] bridge_manager additional info:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 1012, in _validate_conn
    conn.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 411, in connect
    self.sock = ssl_wrap_socket(
  File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3/dist-packages/gevent/_ssl3.py", line 114, in wrap_socket
    return self.sslsocket_class(
  File "/usr/lib/python3/dist-packages/gevent/_ssl3.py", line 312, in __init__
    raise x
  File "/usr/lib/python3/dist-packages/gevent/_ssl3.py", line 308, in __init__
    self.do_handshake()
  File "/usr/lib/python3/dist-packages/gevent/_ssl3.py", line 672, in do_handshake
    self._wait(self._read_event, timeout_exc=_SSLErrorHandshakeTimeout)
  File "src/gevent/_hub_primitives.py", line 317, in gevent._gevent_c_hub_primitives.wait_on_socket
  File "src/gevent/_hub_primitives.py", line 322, in gevent._gevent_c_hub_primitives.wait_on_socket
  File "src/gevent/_hub_primitives.py", line 313, in gevent._gevent_c_hub_primitives._primitive_wait
  File "src/gevent/_hub_primitives.py", line 314, in gevent._gevent_c_hub_primitives._primitive_wait
  File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_hub_primitives.py", line 55, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_waiter.py", line 151, in gevent._gevent_c_waiter.Waiter.get
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 65, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_gevent_c_greenlet_primitives.pxd", line 35, in gevent._gevent_c_greenlet_primitives._greenlet_switch
socket.timeout: The handshake operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 532, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/lib/python3/dist-packages/six.py", line 719, in reraise
    raise value
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 385, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=conn.timeout)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 336, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='receiver.amplify.nginx.com', port=443): Read timed out. (read timeout=7.0)

Connectivity test

curl -v https://receiver.amplify.nginx.com
*   Trying 35.155.143.94:443...
* Connected to receiver.amplify.nginx.com (35.155.143.94) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=receiver.amplify.nginx.com
*  start date: Mar 14 21:05:25 2022 GMT
*  expire date: Jun 12 21:05:24 2022 GMT
*  subjectAltName: host "receiver.amplify.nginx.com" matched cert's "receiver.amplify.nginx.com"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: receiver.amplify.nginx.com
> User-Agent: curl/7.74.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Server: nginx
< Date: Mon, 28 Mar 2022 09:31:51 GMT
< Content-Type: text/html
< Content-Length: 146
< Connection: keep-alive
< X-Amplify-ID: 2017ea69c9f904f966ae0433d6f205a0
< 
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Connection #0 to host receiver.amplify.nginx.com left intact
tkriviradev commented 2 years ago

UP^

nkashiv commented 2 years ago

@tkriviradev Please can you update your agent and check if that resolves this issue?

tkriviradev commented 2 years ago

Sure, I will test this weekend and revert back, thank you!.

achawla2012 commented 1 year ago

As pointed out earlier by @nkashiv, @tkriviradev were you able to update your agent and check if that resolved issue? Here are the instruction. https://amplify.nginx.com/docs/guide-installing-and-managing-nginx-amplify-agent.html#updating-the-agent