CI: tests fail sometimes due missing testing files

[MyPyDavid]

Description / Beschreibung

there is/was a failing test

[gw0] linux -- Python 3.8.18 /opt/hostedtoolcache/Python/3.8.18/x64/bin/python

db = None settings = <pytest_django.fixtures.SettingsWrapper object at 0x7fe6c9a6d280> client = <django.test.client.Client object at 0x7fe6c8a0d4c0> files = PosixPath('/tmp/pytest-of-runner/pytest-0/popen-gw0/test_project_create_import_pos10/media') username = 'site', password = 'site'

I think some tests fail often in the Action because there are some files/fxiture files (under testing/ for example) missing during the test run.. It happens in different test jobs and in different PRs. Do you have an idea what could cause this? Usually a re-run fixes the failing test. Do you know if this is something more common in Github Actions or would there be a way to prevent this?

Should I open another issue to discuss this problem?

Originally posted by @MyPyDavid in https://github.com/rdmorganiser/rdmo/issues/1061#issuecomment-2252955458

Expected behaviour / Erwartetes Verhalten

• ci tests run without failing due to missing files

Steps to reproduce / Schritte zum Reproduzieren

make PR>let checks run

Context / Kontext

Please state your operating system, the RDMO version, and (if applicable) the browser the error occured in.

References / Verweise

Yes, a new issue would be a good idea.

Which files are not available?

The tests run in parallel. But each test that needs the files under media, should have the files in a temporary media root available.

https://github.com/rdmorganiser/rdmo/blob/b6ac5894d23e85d174d734f493ef8277c09d4345/conftest.py#L45-L50

Originally posted by @afuetterer in https://github.com/rdmorganiser/rdmo/issues/1061#issuecomment-2252980501

Sind die Tests "flaky"?

https://www.jetbrains.com/de-de/teamcity/ci-cd-guide/concepts/flaky-tests/

Originally posted by @afuetterer in https://github.com/rdmorganiser/rdmo/issues/1061#issuecomment-2252983734

[MyPyDavid]

Sind die Tests "flaky"?
https://github.com/rdmorganiser/rdmo/pull/1061#issuecomment-2252983734

@afuetterer , any test that uses a fixture file from testing/ seems to be able to fail, but I don't think they are flaky by themselves, locally they should all pass.

[afuetterer]

I will investigate by using only the tests with the files fixture and run them on repeat.

Ref: https://github.com/pytest-dev/pytest-repeat?tab=readme-ov-file#repeating-a-test-until-failure

[afuetterer]

@afuetterer , any test that uses a fixture file from testing/ seems to be able to fail, but I don't think they are flaky by themselves, locally they should all pass.

What do you mean by fixture file? I thought about the files in the temp media root, as described above.

A "fixture file" is also the json fixtures. They live in testing as well. They are inserted into the test db at startup of the test suite. Literally every test uses them.

[afuetterer]

Ref: https://github.com/rdmorganiser/rdmo/actions/runs/10112894609/job/27968084168#step:10:625

[MyPyDavid]

Here is another strange case, https://github.com/rdmorganiser/rdmo/actions/runs/10058095113/job/28047314566#step:9:394. At first the CI failed at the e2e import option test, with the uploaded xml file and now after a re-run it fails at asserting something with settings.MEDIA_ROOT (assert Path(settings.MEDIA_ROOT).joinpath(file_value).exists()). I can try to re-run it again and see if/where it will fail...

PS

Now it failed at the first one again, I made a mistake before and just needed to fix the test..

[MyPyDavid]

          I think I found out something about the failing tests with `MEDIA_ROOT`.

I think the fixtures files from conftest.py needs to be added to the test args in tests that use settings.MEDIA_ROOT, otherwise the fixtures is not used or something like that I guess... Like so ..

def test_update(db, client, files, username, password, project_id, snapshot_id):
    ....

Should I add that in another PR? (not in this one right, as long as its not failing here)

Originally posted by @MyPyDavid in https://github.com/rdmorganiser/rdmo/issues/1061#issuecomment-2258018459

[MyPyDavid]

looks so far so good on dev-2.2.0 after the merge.

[MyPyDavid]

maybe some missing xml files might still lead to failing tests:

Test (Python: 3.12, DB: postgres): rdmo/management/tests/test_viewset_upload.py#L129
test_create_error[xml/elements/legacy/catalog-error-key.xml-XML Parsing Error: Missing legacy elements-editor-editor]

AssertionError: assert 'XML Parsing Error: Missing legacy elements' in 
'The content of the XML file does not consist of well-formed data or markup. 
XML Parsing Error: syntax error: line 1, column 0'

[MyPyDavid]

another case:

                    ) from exc
E                   django.core.management.base.CommandError: Database test_rdmo_gw0 couldn't be flushed. Possible reasons:
E                     * The database isn't running or isn't configured correctly.
E                     * At least one of the expected database tables doesn't exist.
E                     * The SQL was invalid.
E                   Hint: Look at the output of 'django-admin sqlflush'. That's the SQL this command wasn't able to run.

/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/django/core/management/commands/flush.py:75: CommandError
=========================== short test summary info ============================
ERROR rdmo/management/tests/e2e/test_frontend_management_elements.py::test_management_navigation[chromium-helper6] - django.core.management.base.CommandError: Database test_rdmo_gw0 couldn't be flushed. Possible reasons:
  * The database isn't running or isn't configured correctly.
  * At least one of the expected database tables doesn't exist.
  * The SQL was invalid.
Hint: Look at the output of 'django-admin sqlflush'. That's the SQL this command wasn't able to run.