aiopg '<' not supported between instances of 'int' and 'NoneType' error on marxantraining.org

[andrewcottam]

This happens when running the stress test 19 with 30 users on marxantraining.org with 8 CPU. It doesn't happen on andrewcottam.com which has 1 CPU.

Full trace:

[E 05-05-20 09:12:27.081] Uncaught exception GET /marxan-server/runMarxan?user=user_25&project=Start%20project (18.200.163.92)
    HTTPServerRequest(protocol='https', host='marxantraining.org:443', method='GET', uri='/marxan-server/runMarxan?user=user_25&project=Start%20project', version='HTTP/1.1', remote_ip='18.200.163.92')
    Traceback (most recent call last):
      File "/home/marxancloud/marxan-server/marxan-server.py", line 1540, in execute
        with (await self.pool.cursor()) as cur:
      File "/home/marxancloud/miniconda3/lib/python3.7/site-packages/aiopg/pool.py", line 259, in cursor
        conn = await self.acquire()
      File "/home/marxancloud/miniconda3/lib/python3.7/site-packages/aiopg/pool.py", line 164, in _acquire
        await self._fill_free_pool(True)
      File "/home/marxancloud/miniconda3/lib/python3.7/site-packages/aiopg/pool.py", line 208, in _fill_free_pool
        if override_min and self.size < self.maxsize:
    TypeError: '<' not supported between instances of 'int' and 'NoneType'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/home/marxancloud/miniconda3/lib/python3.7/site-packages/tornado/websocket.py", line 958, in _accept_connection
        await open_result
      File "/home/marxancloud/marxan-server/marxan-server.py", line 2682, in open
        await super().open({'info': "Running Marxan.."})
      File "/home/marxancloud/marxan-server/marxan-server.py", line 2624, in open
        await _getProjectData(self)
      File "/home/marxancloud/marxan-server/marxan-server.py", line 432, in _getProjectData
        df2 = await pg.execute("select * from marxan.get_planning_units_metadata(%s)", data=[value], returnFormat="DataFrame")
      File "/home/marxancloud/marxan-server/marxan-server.py", line 1580, in execute
        raise MarxanServicesError(e.args[0])
    MarxanServicesError: '<' not supported between instances of 'int' and 'NoneType'

This is a bug in aiopg: if you set maxsize=0 then in this test (self.size < self.maxsize) maxsize is None.

[andrewcottam]

This is difficult to reproduce but changing the maxsize from 0 to 100 stopped the error initially. The line:

with (await self.pool.cursor()) as cur:

Is the line that caused the error in #213

[andrewcottam]

aiopg is quirky and the docs don't really explain some of the weird behavior. All of the aiopg raised Exceptions (unclosed connection, GeneratorExit etc) are due to the minsize and maxsize parameters of the pool. You must set minsize to 10 to avoid these errors - not sure why.