BWC tests are failing very regularly

[amotl]

Hi there,

@matriv reported that the BWC tests are failing very regularly, sometimes with timeouts, sometimes with 500 server errors. An example is ^1.

With kind regards, Andreas.

[BaurzhanSakhariev]

Error message:

ERROR: test_upgrade_paths (test_upgrade.StorageCompatibilityTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/lib/jenkins/workspace/CrateDB/qa/crate_qa/tests/bwc/test_upgrade.py", line 160, in test_upgrade_paths
    self._test_upgrade_path(path, nodes=3)
  File "/var/lib/jenkins/workspace/CrateDB/qa/crate_qa/tests/bwc/test_upgrade.py", line 190, in _test_upgrade_path
    digest = container.put(BytesIO(b'sample data'))
  File "/var/lib/jenkins/workspace/CrateDB/qa/crate_qa/env/lib/python3.8/site-packages/crate/client/blob.py", line 67, in put
    created = self.conn.client.blob_put(self.container_name,
  File "/var/lib/jenkins/workspace/CrateDB/qa/crate_qa/env/lib/python3.8/site-packages/crate/client/http.py", line 440, in blob_put
    _raise_for_status(response)
  File "/var/lib/jenkins/workspace/CrateDB/qa/crate_qa/env/lib/python3.8/site-packages/crate/client/http.py", line 209, in _raise_for_status
    raise ProgrammingError(message)
crate.client.exceptions.ProgrammingError: 500 Server Error: Internal Server Error

[matriv]

We'd probably need to expose the crate node logs to get some help. I have played around a bit and never managed to make it fail locally so far.

[mfussenegger]

Could probably use a similar pattern to what we did in crate to dump the server logs on errors:

https://github.com/crate/crate/commit/2849a2cee64eed02d030cc55ea7357767119eff9

[matriv]

There is again failure regarding the blob test, reopening.

[mfussenegger]

Should be fixed with https://github.com/crate/crate-qa/pull/267