autopilotpattern / nginx

An Nginx container for container-native deployment and automatic backend discovery
Mozilla Public License 2.0
57 stars 25 forks source link

tests should check upstream block instead of making request to Nginx #45

Closed tgross closed 7 years ago

tgross commented 7 years ago

We're getting this intermittent exception in our integration tests on Triton:

Traceback (most recent call last):
  File "tests.py", line 55, in test_scaleup_and_down
    self.compare_backends()
  File "tests.py", line 113, in compare_backends
    actual = list(set([self.query_for_backend() for _ in range(10)])).sort()
  File "tests.py", line 113, in <listcomp>
    actual = list(set([self.query_for_backend() for _ in range(10)])).sort()
  File "tests.py", line 119, in query_for_backend
    r = requests.get('http://{}'.format(self.nginx_cns)) # TODO: SSL
  File "/usr/lib/python3.5/site-packages/requests/api.py", line 70, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/lib/python3.5/site-packages/requests/api.py", line 56, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.5/site-packages/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.5/site-packages/requests/sessions.py", line 596, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.5/site-packages/requests/adapters.py", line 487, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='nginx-frontend.svc.[redacted].us-sw-1.triton.zone', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f21faef5c50>: Failed to establish a new connection: [Errno -2] Name does not resolve',))

I've got a long timeout set up for wait_for_cns that makes a request to that same CNAME, but then in this later section the same CNS is no longer resolved, which doesn't make a lot of sense to me. Simply extending the timeout won't help. The test itself isn't terribly awesome in any case, as we're just querying Nginx a bunch of times to get the hostname of the backend it serves. Instead we should just docker exec and get the upstream block out of its configuration so we don't have to worry at all about DNS at that point.

tgross commented 7 years ago

Fixed as of: https://product-ci.joyent.us/job/autopilotpattern-nginx/31/