handling NURESTPushCenter connection errors

[mainTAP]

Dear team,

I've been experiencing several connection errors when using the Bambou PushCenter to listen to VSD Events after a random period of time. The connection is made to the nuageX VSD environment and re-runing the script connects and listens to events fine for some time again until another connection error occurs.

For instance :

(vsd) [user@server project-vsd]$ python listener.py 
Exception in thread Thread-11:
Traceback (most recent call last):
File "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
    self.run()
File "/usr/lib64/python2.7/threading.py", line 764, in run
    self.__target(*self.__args, **self.__kwargs)
File "/home/mmarko/project-vsd/vsd/lib/python2.7/site-packages/bambou/nurest_connection.py", line 373, in _make_request
    response = self.__make_request(requests_session=_NURESTSessionCurrentContext.session.requests_session, method=self._request.method, url=self._request.url, params=self._request.params, data=data, headers=headers, certificate=certificate)
File "/home/mmarko/project-vsd/vsd/lib/python2.7/site-packages/bambou/nurest_connection.py", line 407, in __make_request
    cert=certificate)
File "/home/mmarko/project-vsd/vsd/lib/python2.7/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
File "/home/mmarko/project-vsd/vsd/lib/python2.7/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
File "/home/mmarko/project-vsd/vsd/lib/python2.7/site-packages/requests/adapters.py", line 490, in send
    raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', error(104, 'Connection reset by peer'))

or

2017-10-29 23:52:35,493 WARNING < headers: {'Content-Length': '1044', 'Set-Cookie': 'rememberMe=deleteMe; Path=/nuage; Max-Age=0; Expires=Sat, 28-Oct-2017 23:52:35 GMT', 'Expires': 'Thu, 01 Jan 1970 00:00:00 UTC', 'Server': 'Apache-Coyote/1.1', 'Pragma': 'No-cache', 'Cache-Control': 'no-cache', 'Date': 'Sun, 29 Oct 2017 23:52:34 GMT', 'Content-Type': 'text/html;charset=utf-8'}
2017-10-29 23:52:35,494 WARNING < data:
null
2017-10-29 23:52:35,496 ERROR  [NURESTPushCenter]: Connection failure [401] {}
Exception in thread Thread-888167:
Traceback (most recent call last):
File "C:\Python27amd64\lib\threading.py", line 801, in __bootstrap_inner
    self.run()
File "C:\Python27amd64\lib\threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
File "C:\Python27amd64\lib\site-packages\bambou\nurest_connection.py", line 373, in _make_request
    response = self.__make_request(requests_session=_NURESTSessionCurrentContext.session.requests_session, method=self._request.method, url=self._request.url, params=self._request.params, data=data, headers=headers, certificate=certificate)
File "C:\Python27amd64\lib\site-packages\bambou\nurest_connection.py", line 407, in __make_request
    cert=certificate)
File "C:\Python27amd64\lib\site-packages\requests\sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
File "C:\Python27amd64\lib\site-packages\requests\sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
File "C:\Python27amd64\lib\site-packages\requests\adapters.py", line 490, in send
    raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', error(10054, 'An existing connection was forcibly closed by the remote host'))

or

Exception in thread Thread-1827:
Traceback (most recent call last):
  File "C:\Python27amd64\lib\threading.py", line 801, in __bootstrap_inner
    self.run()
  File "C:\Python27amd64\lib\threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "C:\Python27amd64\lib\site-packages\bambou\nurest_connection.py", line 373, in _make_request
    response = self.__make_request(requests_session=_NURESTSessionCurrentContext.session.requests_session, method=self._request.method, url=self._request.url, params=self._request.params, data=data, headers=headers, certificate=certificate)
  File "C:\Python27amd64\lib\site-packages\bambou\nurest_connection.py", line 407, in __make_request
    cert=certificate)
  File "C:\Python27amd64\lib\site-packages\requests\sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Python27amd64\lib\site-packages\requests\sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "C:\Python27amd64\lib\site-packages\requests\adapters.py", line 490, in send
    raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', BadStatusLine("''",))

Sometimes I only receive the error without the exception and the push_center continues to listening to the events ( this however repeats in a loop and doesn't attempt to re-connect to the VSD until the script is re-run ) :

2017-10-26 20:55:28,831 WARNING < headers: {'Content-Length': '1061', 'Set-Cookie': 'rememberMe=deleteMe; Path=/nuage; Max-Age=0; Expires=Wed, 25-Oct-2017 19:55:28 GMT', 'Expires': 'Thu, 01 Jan 1970 00:00:00 UTC', 'Server': 'Apache-Coyote/1.1', 'Connection': 'close', 'Pragma': 'No-cache', 'Cache-Control': 'no-cache', 'Date': 'Thu, 26 Oct 2017 19:55:28 GMT', 'Content-Type': 'text/html;charset=utf-8'}
2017-10-26 20:55:28,832 WARNING < data:
null
2017-10-26 20:55:28,834 ERROR  [NURESTPushCenter]: Connection failure [400] {}

The testing code is very simple, just runs the push_center and listen to the received events :

def did_receive_push(data):
    logger.debug("Keepalive : %s" % (data['uuid']))

def main():

    nc = vspk.NUVSDSession(username=username, password=password, enterprise=enterprise, api_url=sapi_url)
    nc.start()

    push_center = nc.push_center
    push_center.add_delegate(did_receive_push)
    push_center.start()

    while True:
        time.sleep(1000)

Can you please advise what would be the best way to handle these errors so that the push_center attempts to re-connect and continues listening to the events ?

Thank you very much.

[pdellaert]

Hi @mainTAP ,

The push channel solution to listen to events is more or less deprecated and is just left in Bambou for backward compatibility (but might be removed in the next year). The reason for this is that push channel uses HTTP Long poll, which has some flaws including the risk of causing memory issues if used improperly (luckily, bambou does it right, but still... it is not safe).

The preferred way of working with events is using JMS or AMQP. Both are supported in 5.1.1 and above. JMS has been supported for a long time (all 4.0 releases and above).

An example JMS client: https://github.com/nuagenetworks/jmsclient An example AMQP client (in python): https://github.com/nuagenetworks/AMQP-Client

On the issue you are facing: It could be that this has to do with API token that expires after a time, requiring a new session to be setup. The quick solution would be to capture the exception and the error itself, and re-initiate the push channel connection.

Hope this helps.

[mainTAP]

Hi @pdellaert ,

Thank you very much for pointing me to the JMS/AMQP route, these would certainly be much better option than the HTTP long poll. Is there anything necessary to do at the VSD to enable JMS ? I was trying to follow the VSP API Programming Guide ( set the iptables / ipset at the VSD node and create the client user in VSD Architect ) but unfortunately the example JMS client is still unable to connect . Many thanks.

[pdellaert]

Hi @mainTAP,

JMS is enabled by default on all VSDs in 4.0 and 5.x releases. JMS and AMQP is enabled by default on VSDs of release 5.1.1 and above.

You could try as a test and open up full IPtables by default and see if that helps. Make sure you use the right ports.

In all my tests, i've never had to open up ports myself, i think. Not sure if i had to add my client IP to the ipset, don't think so, but could be wrong.

Cheers