Fix failing systems tests

[FreddieAkeroyd]

As a developer i would like the systems tests to not fail. They are currently failing due to a python issue that I guess is a result of the dependency update, but this could be a coincidence. One error reports

Traceback (most recent call last):
  File "C:\Jenkins\workspace\System_Tests_galilold\test_memory_usage.py", line 65, in test_GIVEN_typical_config_with_IOCs_blocks_and_LVDCOM_IOC_WHEN_dae_is_not_doing_a_run_THEN_memory_usage_stays_under_7point5gb
    assert_that(memory_used, less_than(system_threshold-ASSUMED_NON_IBEX_USAGE))
AssertionError: 
Expected: a value less than <5.5>
     but: was <20.785423278808594>

however later it says

Memory at start, after, diff: 0.0, 20.785423278808594, 20.785423278808594 GB

Note that start is 0.0 so it now counting all system memory, not just the increase in system memory, as due to ibex. There are several module load errors in the system tests jenkins logs

ModuleNotFoundError: No module named 'compose' (docker-compose?)

ModuleNotFoundError: No module named 'configobj' 

AttributeError: cython_sources

I think the configobj error is most relevant here, but they should all be checked for impact - the compose error is probably causing the datastreaming test to fail

acceptance criteria

• import errors fixed
• memory test calculates correct starting memory
• run_tests.bat prints an additional error message is python setup for memory fails

Notes

This seems to be a pyyaml / docker / cython versions issue - currently datastreamign tests are disabled as a workaround. To test you will need to:

• rename notest_datastreaming.py back to test_datastreaming.py
• uncomment docker and docker-compose in requirements.txt
• run_tests.bat -t test_datastreaming

[rerpha]

Just having a look with Zsolt now.

The issue comes from the datastreaming tests - they rely on the pip package docker-compose.

docker-compose used to be a pip package you could install, but it has been discontinued since 2021 in favour of V2 (and now V3) which is now an executable part of docker desktop.

docker-compose 1.29.2 (the last version of 1.x.x) relies on pyyaml 5.4.1, which is not supported by Python 3.11 (which we have recently upgraded to) due to a cython<3 within its requirements.txt - more info here V2+ is written in Go so doesn't provide a nice python API so we can't just upgrade.

We could try and work around this by using something like podman-compose instead, or updating our compose images to V2/V3 then using a sys call to docker-compose.exe in the tests but I think our compose files are V1 format so are out of date regardless. Rather than trying to fix this, I think we should just remove the datastreaming tests - we aren't using it in production so seems a bit pointless to keep running the tests for them. We always have version control if we decide we want to upgrade the compose files and get them working again (although we'd need a machine with python <3.9 on...).

The memory usage test - this is because configobj was not installed (configobj is just after compose alphabetically so wasn't pip installed after the pip error with compose) proof of this is that the test is now passing now docker-compose has been removed from requirements.txt

[LowriJenkins]

Can this be picked up for review by anyone? its marked as rework, but has been sitting in review and not under review for a month

[danielmaclaren]

Can this be picked up for review by anyone? its marked as rework, but has been sitting in review and not under review for a month

Sorry Lowri, I didn't see your message! You can review this if you still wish, or I'll start it!