Tests: Flakyness within "doctests/itests.txt" and "doctests/client.txt"

[amotl]

Hi there,

while working on crate/crate-python#388, I just observed tests within doctests/itests.txt might be flaky [1]. The test has been run on GHA/macOS.

Failure in test /Users/runner/work/crate-python/crate-python/src/crate/client/sqlalchemy/doctests/itests.txt
Failed doctest test for itests.txt
  File "/Users/runner/work/crate-python/crate-python/src/crate/client/sqlalchemy/doctests/itests.txt", line 0

----------------------------------------------------------------------
File "/Users/runner/work/crate-python/crate-python/src/crate/client/sqlalchemy/doctests/itests.txt", line 146, in itests.txt
Failed example:
    result.rowcount
Expected:
    10
Got:
    4

Maybe we should run a REFRESH TABLE xxx here instead of just sleeping for a while?

https://github.com/crate/crate-python/blob/7e12a49d46da36d5f502db38bfc780793b50b786/src/crate/client/sqlalchemy/doctests/itests.txt#L143

With kind regards, Andreas.

[1] https://github.com/crate/crate-python/runs/1358981202?check_suite_focus=true#step:5:142

[amotl]

At [2], I've just observed another flakyness within doctests/client.txt, also on GHA/macOS.

File "/Users/runner/work/crate-python/crate-python/src/crate/client/doctests/client.txt", line 96, in client.txt
Failed example:
    cursor.execute("REFRESH TABLE locations")
Exception raised:
    Traceback (most recent call last):
      File "/Users/runner/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/doctest.py", line 1336, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest client.txt[15]>", line 1, in <module>
        cursor.execute("REFRESH TABLE locations")
      File "/Users/runner/work/crate-python/crate-python/src/crate/client/cursor.py", line 53, in execute
        self._result = self.connection.client.sql(sql, parameters,
      File "/Users/runner/work/crate-python/crate-python/src/crate/client/http.py", line 396, in sql
        content = self._json_request('POST', self.path, data=data)
      File "/Users/runner/work/crate-python/crate-python/src/crate/client/http.py", line 522, in _json_request
        response = self._request(method, path, data=data)
      File "/Users/runner/work/crate-python/crate-python/src/crate/client/http.py", line 513, in _request
        self._drop_server(next_server, ex_message)
      File "/Users/runner/work/crate-python/crate-python/src/crate/client/http.py", line 581, in _drop_server
        raise ConnectionError(
    crate.client.exceptions.ConnectionError: No more Servers available, exception from last server: 
HTTPConnectionPool(host='127.0.0.1', port=44209): Max retries exceeded with url: /_sql (Caused by ReadTimeoutError("HTTPConnectionPool(host='127.0.0.1', port=44209): Read timed out. (read timeout=2)"))

[2] https://github.com/crate/crate-python/runs/1359096616?check_suite_focus=true#step:5:154

[amotl]

This will probably be resolved by 06d25f38 coming from crate/crate-python#386.

[amotl]

Same thing happened again.

Failure in test /Users/runner/work/crate-python/crate-python/src/crate/client/doctests/client.txt
Failed doctest test for client.txt
  File "/Users/runner/work/crate-python/crate-python/src/crate/client/doctests/client.txt", line 0

----------------------------------------------------------------------
File "/Users/runner/work/crate-python/crate-python/src/crate/client/doctests/client.txt", line 96, in client.txt
Failed example:
    cursor.execute("REFRESH TABLE locations")
Exception raised:
    Traceback (most recent call last):
      File "/Users/runner/hostedtoolcache/Python/3.9.12/x64/lib/python3.9/doctest.py", line 1334, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest client.txt[15]>", line 1, in <module>
        cursor.execute("REFRESH TABLE locations")
      File "/Users/runner/work/crate-python/crate-python/src/crate/client/cursor.py", line 53, in execute
        self._result = self.connection.client.sql(sql, parameters,
      File "/Users/runner/work/crate-python/crate-python/src/crate/client/http.py", line 396, in sql
        content = self._json_request('POST', self.path, data=data)
      File "/Users/runner/work/crate-python/crate-python/src/crate/client/http.py", line 522, in _json_request
        response = self._request(method, path, data=data)
      File "/Users/runner/work/crate-python/crate-python/src/crate/client/http.py", line 513, in _request
        self._drop_server(next_server, ex_message)
      File "/Users/runner/work/crate-python/crate-python/src/crate/client/http.py", line 590, in _drop_server
        raise ConnectionError(
    crate.client.exceptions.ConnectionError: No more Servers available, exception from last server: 
HTTPConnectionPool(host='127.0.0.1', port=44209): Read timed out. (read timeout=2)

-- https://github.com/crate/crate-python/runs/6013809996?check_suite_focus=true#step:6:327

[amotl]

Another fragment coming from layer.txt.

File "/Users/runner/work/crate-python/crate-python/src/crate/testing/doctests/layer.txt", line 33, in layer.txt
Failed example:
    layer.start()
Exception raised:
    Traceback (most recent call last):
      File "/Users/runner/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/doctest.py", line 1337, in __run
        compileflags, 1), test.globs)
      File "<doctest layer.txt[6]>", line 1, in <module>
        layer.start()
      File "/Users/runner/work/crate-python/crate-python/src/crate/testing/layer.py", line 319, in start
        self._wait_for_start()
      File "/Users/runner/work/crate-python/crate-python/src/crate/testing/layer.py", line 376, in _wait_for_start
        self._wait_for(validator)
      File "/Users/runner/work/crate-python/crate-python/src/crate/testing/layer.py", line 355, in _wait_for
Error:         raise SystemError('Failed to start Crate instance in time.')
    SystemError: Failed to start Crate instance in time.

-- https://github.com/crate/crate-python/runs/6175214795?check_suite_focus=true#step:6:182

P.S.: The same error has been observed and reported previously at https://github.com/crate/crate-python/pull/363#issuecomment-664895999.

[amotl]

Errors like https://github.com/crate/crate-python/issues/629 have been observable a number of times while working on crate/crate-python#400. 90a97c7d might improve the situation.

[amotl]

The root cause for SystemError: Failed to start Crate instance in time. (see https://github.com/crate/crate-python/issues/629) is that a previous instance of CrateDB might not have been terminated correctly.

In that case, a subsequent attempt to start it will croak like [1]

Caused by: io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Address already in use

[1] https://github.com/crate/crate-python/runs/6210165871#step:6:192

[amotl]

I am closing this issue. The initial problems have not been observed lately, the second problem is still observable and now moved to crate/crate-python#630.