Intermittent test failure for `TestREClientIntegration.test_admin_empty_query`

[n1mus]

Skipped that then got intermittent failure for test_admin_missing_params with also a connection error. My impression was that that was the next error-mode test to run alphabetically, and maybe that's why it failed? IIRC there were some non-error-mode tests that ran beforehand just fine.

[MrCreosote]

I'm consistently seeing these four tests fail every other run:

FAILED client_src/test/test_integration.py::TestREClientIntegration::test_admin_empty_query
FAILED client_src/test/test_integration.py::TestREClientIntegration::test_admin_missing_param
FAILED client_src/test/test_integration.py::TestREClientIntegration::test_admin_query_empty_auth
FAILED client_src/test/test_integration.py::TestREClientIntegration::test_admin_query_invalid_auth

Although I've seen these fail once:

FAILED client_src/test/test_integration.py::TestREClientIntegration::test_admin_empty_query
FAILED client_src/test/test_integration.py::TestREClientIntegration::test_admin_missing_param
FAILED client_src/test/test_integration.py::TestREClientIntegration::test_admin_query_empty_auth
FAILED client_src/test/test_integration.py::TestREClientIntegration::test_admin_query_invalid_auth
FAILED client_src/test/test_integration.py::TestREClientIntegration::test_admin_query_ok
FAILED client_src/test/test_integration.py::TestREClientIntegration::test_admin_raise_not_found
FAILED client_src/test/test_integration.py::TestREClientIntegration::test_save_docs_empty_auth
FAILED client_src/test/test_integration.py::TestREClientIntegration::test_save_docs_invalid_auth
FAILED client_src/test/test_integration.py::TestREClientIntegration::test_save_docs_invalid_docs
FAILED client_src/test/test_integration.py::TestREClientIntegration::test_save_docs_ok
FAILED client_src/test/test_integration.py::TestREClientIntegration::test_save_docs_unknown_coll
FAILED client_src/test/test_integration.py::TestREClientIntegration::test_stored_query_missing_bind_vars
FAILED client_src/test/test_integration.py::TestREClientIntegration::test_stored_query_ok
FAILED client_src/test/test_integration.py::TestREClientIntegration::test_stored_query_raise_not_found
FAILED client_src/test/test_integration.py::TestREClientIntegration::test_stored_query_unknown_query

It seems like it repeatably fails every other test, suggesting that the passing tests are leaving state that breaks the next run, and the failing tests somehow clear it up

[MrCreosote]

Ok, I've run the tests 12 times in a row now, recording the results, and it's alternating fail / pass every time. I ran the tests several times before that without rigorously checking the pattern but I'm pretty sure it was the same way

[MrCreosote]

After a reboot, the first test failed and the second test passed. I assume this means that automated tests are going to fail.

[MrCreosote]

It looks to me like the client tests start running before the server is up?

Validation succeeded!
Creating collection ncbi_taxon (edge: False)
============================= test session starts ==============================
platform linux -- Python 3.7.13, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /app
collecting ... Successfully created collection ncbi_taxon
Creating fulltext index for collection ncbi_taxon: {'type': 'fulltext', 'fields': ['scientific_name'], 'minLength': 1}
Successfully created fulltext index on ['scientific_name'] for ncbi_taxon.
Creating persistent index for collection ncbi_taxon: {'type': 'persistent', 'fields': ['id', 'expired', 'created']}
Successfully created persistent index on ['id', 'expired', 'created'] for ncbi_taxon.
Creating persistent index for collection ncbi_taxon: {'type': 'persistent', 'fields': ['expired', 'created', 'last_version']}
collected 198 items / 1 skipped / 197 selected                                 

client_src/test/test_integration.py Successfully created persistent index on ['expired', 'created', 'last_version'] for ncbi_taxon.
Creating collection test_edge (edge: True)
F.Successfully created collection test_edge
Creating collection test_vertex (edge: False)
Successfully created collection test_vertex
Creating view test_vertices
FFEnsuring indexes for /spec/collections/ncbi/ncbi_taxon.yaml
All index specs ensured
Ensuring view /spec/views/test_vertices.json
All view specs ensured
All analyzer specs ensured
[2022-07-26 18:35:46 +0000] [44] [INFO] Starting gunicorn 20.1.0
[2022-07-26 18:35:46 +0000] [44] [INFO] Listening at: http://0.0.0.0:5000 (44)
[2022-07-26 18:35:46 +0000] [44] [INFO] Using worker: sync
[2022-07-26 18:35:46 +0000] [46] [INFO] Booting worker with pid: 46
F[2022-07-26 18:35:46 +0000] [47] [INFO] Booting worker with pid: 47
PUT /api/v1/documents -> 200 OK
POST /api/v1/query_results -> 200 OK
.POST /api/v1/query_results -> 200 OK
.PUT /api/v1/documents -> 400 BAD REQUEST

Note the F test failure markers are all prior to the last Booting worker message