Broken geocoding cache test

[c0c0n3]

Describe the bug

The test_cache test in test_geocoding.py fails randomly on CircleCI. See e.g. #660, #661, #663.

To Reproduce

See linked PRs.

Expected behavior

Test should pass.

Additional context

Error example from CircleCI

#!/bin/bash -eo pipefail
export PIPENV_SHELL=no
export WORKON_HOME=~/.venv
sudo sysctl -w vm.max_map_count=262144
source setup_dev_env.sh
export tests=others
pipenv run sh run_tests.sh

vm.max_map_count = 262144
used ip: 172.17.0.1
creating test directory...
=======================================================================
        GEO-CODING TESTS
=======================================================================
============================= test session starts ==============================
platform linux -- Python 3.8.5, pytest-3.10.1, py-1.11.0, pluggy-1.0.0
rootdir: /home/circleci/project/src, inifile: pytest.ini
plugins: flask-0.15.1, cov-2.7.1, lovely-pytest-docker-0.3.0, lazy-fixture-0.6.3
collected 312 items                                                            

src/geocoding/geojson/tests/test_wktcodec.py ....................        [  6%]
src/geocoding/slf/tests/test_geotypes.py ............................... [ 16%]
........................................................................ [ 39%]
........                                                                 [ 41%]
src/geocoding/slf/tests/test_jsoncodec.py ..........                     [ 45%]
src/geocoding/slf/tests/test_locparser.py .............................. [ 54%]
.....                                                                    [ 56%]
src/geocoding/slf/tests/test_queryparser.py ............................ [ 65%]
.......................                                                  [ 72%]
src/geocoding/slf/tests/test_wktcodec.py ....                            [ 74%]
src/geocoding/tests/test_centroid.py ................................... [ 85%]
......                                                                   [ 87%]
src/geocoding/tests/test_factory.py .....                                [ 88%]
src/geocoding/tests/test_geocoding.py ............F.                     [ 93%]
src/geocoding/tests/test_location.py .....................               [100%]

=================================== FAILURES ===================================
_________________________________ test_caching _________________________________

self = Connection<host=172.17.0.1,port=6379,db=0>

    def connect(self):
        "Connects to the Redis server if not already connected"
        if self._sock:
            return
        try:
>           sock = self._connect()

../.venv/project-zxI9dQ-Q/lib/python3.8/site-packages/redis/connection.py:559: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = Connection<host=172.17.0.1,port=6379,db=0>

    def _connect(self):
        "Create a TCP socket connection"
        # we want to mimic what socket.create_connection does to support
        # ipv4/ipv6, but we want to set options prior to calling
        # socket.connect()
        err = None
        for res in socket.getaddrinfo(self.host, self.port, self.socket_type,
                                      socket.SOCK_STREAM):
            family, socktype, proto, canonname, socket_address = res
            sock = None
            try:
                sock = socket.socket(family, socktype, proto)
                # TCP_NODELAY
                sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)

                # TCP_KEEPALIVE
                if self.socket_keepalive:
                    sock.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
                    for k, v in iteritems(self.socket_keepalive_options):
                        sock.setsockopt(socket.IPPROTO_TCP, k, v)

                # set the socket_connect_timeout before we connect
                sock.settimeout(self.socket_connect_timeout)

                # connect
                sock.connect(socket_address)

                # set the socket_timeout now that we're connected
                sock.settimeout(self.socket_timeout)
                return sock

            except socket.error as _:
                err = _
                if sock is not None:
                    sock.close()

        if err is not None:
>           raise err

../.venv/project-zxI9dQ-Q/lib/python3.8/site-packages/redis/connection.py:615: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = Connection<host=172.17.0.1,port=6379,db=0>

    def _connect(self):
        "Create a TCP socket connection"
        # we want to mimic what socket.create_connection does to support
        # ipv4/ipv6, but we want to set options prior to calling
        # socket.connect()
        err = None
        for res in socket.getaddrinfo(self.host, self.port, self.socket_type,
                                      socket.SOCK_STREAM):
            family, socktype, proto, canonname, socket_address = res
            sock = None
            try:
                sock = socket.socket(family, socktype, proto)
                # TCP_NODELAY
                sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)

                # TCP_KEEPALIVE
                if self.socket_keepalive:
                    sock.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
                    for k, v in iteritems(self.socket_keepalive_options):
                        sock.setsockopt(socket.IPPROTO_TCP, k, v)

                # set the socket_connect_timeout before we connect
                sock.settimeout(self.socket_connect_timeout)

                # connect
>               sock.connect(socket_address)
E               ConnectionRefusedError: [Errno 111] Connection refused

../.venv/project-zxI9dQ-Q/lib/python3.8/site-packages/redis/connection.py:603: ConnectionRefusedError

During handling of the above exception, another exception occurred:

docker_redis = None
air_quality_observed = {'CO': {'type': 'Text', 'value': 'nr'}, 'NO2': {'type': 'Text', 'value': '69'}, 'O3': {'type': 'Text', 'value': '45'}, 'PM10': {'type': 'Text', 'value': '139'}, ...}
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f3bc364a880>

    def test_caching(docker_redis, air_quality_observed, monkeypatch):
        air_quality_observed.pop('location')

        air_quality_observed['address']['value'] = {
            "streetAddress": "IJzerlaan",
            "postOfficeBoxNumber": "18",
            "addressLocality": "Antwerpen",
            "addressCountry": "BE",
        }

        from geocoding.geocache import temp_geo_cache
        cache = next(temp_geo_cache(REDIS_HOST, REDIS_PORT))
>       assert len(cache.redis.keys('*')) == 0

src/geocoding/tests/test_geocoding.py:193: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../.venv/project-zxI9dQ-Q/lib/python3.8/site-packages/redis/client.py:1661: in keys
    return self.execute_command('KEYS', pattern)
../.venv/project-zxI9dQ-Q/lib/python3.8/site-packages/redis/client.py:898: in execute_command
    conn = self.connection or pool.get_connection(command_name, **options)
../.venv/project-zxI9dQ-Q/lib/python3.8/site-packages/redis/connection.py:1192: in get_connection
    connection.connect()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = Connection<host=172.17.0.1,port=6379,db=0>

    def connect(self):
        "Connects to the Redis server if not already connected"
        if self._sock:
            return
        try:
            sock = self._connect()
        except socket.timeout:
            raise TimeoutError("Timeout connecting to server")
        except socket.error as e:
>           raise ConnectionError(self._error_message(e))
E           redis.exceptions.ConnectionError: Error 111 connecting to 172.17.0.1:6379. Connection refused.

../.venv/project-zxI9dQ-Q/lib/python3.8/site-packages/redis/connection.py:563: ConnectionError
----------------------------- Captured stderr call -----------------------------
Exception ignored in: <generator object temp_geo_cache at 0x7f3bc35853c0>
Traceback (most recent call last):
  File "/home/circleci/project/src/geocoding/geocache.py", line 16, in temp_geo_cache
    gc.redis.flushall()
  File "/home/circleci/.venv/project-zxI9dQ-Q/lib/python3.8/site-packages/redis/client.py", line 1275, in flushall
    return self.execute_command('FLUSHALL', *args)
  File "/home/circleci/.venv/project-zxI9dQ-Q/lib/python3.8/site-packages/redis/client.py", line 898, in execute_command
    conn = self.connection or pool.get_connection(command_name, **options)
  File "/home/circleci/.venv/project-zxI9dQ-Q/lib/python3.8/site-packages/redis/connection.py", line 1192, in get_connection
    connection.connect()
  File "/home/circleci/.venv/project-zxI9dQ-Q/lib/python3.8/site-packages/redis/connection.py", line 563, in connect
    raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 111 connecting to 172.17.0.1:6379. Connection refused.
- generated xml file: /home/circleci/project/test-results/junit-geocoding.xml --

===================== 1 failed, 311 passed in 9.59 seconds =====================

[Ravisaketi]

Hi, @c0c0n3 actually geocoding cache test fails only with newly raised PRs and only in CircleCI and GitHub workflows. we merged #660 #661 and #663 PRs in this case master branch should also fail but the master branch geocoding cache test are passed. and I clone my master branch and create a demo branch and I didn't change anything in this demo branch, just push this demo branch into CircleCI, in this case, the demo branch should be passed, but it failed. so can you please suggest for further improvement on this issue?

[c0c0n3]

@Necravisaketi thanks so much for looking into this, much appreciated. To get to the bottom of this you'd have to try replicating the issue locally in an env similar to CircleCI's---e.g. using the same VM...

[Ravisaketi]

Hi, @c0c0n3 After merging pr #677 the test_cache test in test_geocoding.py was passed. So, as per my understanding, we can close this issue if we don't have any other requirements. Please correct me if I have misunderstood the requirement or if we want any other changes to be considered through this issue.

[c0c0n3]

Hi @Necravisaketi, yea I agree, let's close this issue.