System tests with a real AreaDetector IOC

bluesky / ophyd-async

Hardware abstraction for bluesky written using asyncio

https://blueskyproject.io/ophyd-async

BSD 3-Clause "New" or "Revised" License

8 stars 23 forks source link

System tests with a real AreaDetector IOC #518

Open callumforrester opened 1 month ago

callumforrester commented 1 month ago

This issue captures a discussion with @jwlodek @mrakitin @evalott100 @prjemian and others.

Despite our best efforts, updates to ophyd-async are still reasonably likely to break for someone somewhere, primarily because IOCs are fiddly, idiosyncratic, have no defined schemas and can vary per facility. @coretl actually identified this as a major risk to the project in a review a few weeks ago. This brittleness is particularly evident with AreaDetectors, so if we're going to put any work into mitigating this problem, that would be the low-hanging fruit to start with.

@jwlodek's proposal was a set of system tests that run in CI against a containerised simulated areadetector. @gilesknap could probably comment on how easy such a thing would be to set up. The main flaw is that it would only pick up problems with ADCore, and not any specific detectors, but it would be a start.

Comment if I've missed anything :)

gilesknap commented 1 month ago

If you would like to run a set of tests against a SimDetector then the container already exists here: ghcr.io/epics-containers/ioc-adsimdetector-runtime:2024.8.2

Source for the container is at: https://github.com/epics-containers/ioc-adsimdetector

We could easily spin up an instance of this in GitHub actions and run some tests against it. I could help with getting that working.

coretl commented 3 weeks ago

Discussed this with @DominicOram in relation to an Odin system test. To summarize:

We like the idea of separating out tests/ and system-tests/ so the existing test suite isn't burdensome to run
Anything in system-tests/ will have dependent services like the EPICS containerised IOC
These services should be run up in the same way as @garryod does for his fake databases, can't remember if it was docker compose or a GitHub actions construct

DominicOram commented 3 weeks ago

Yep, Garry pointed me at the repos. I will do some of this as part of https://github.com/DiamondLightSource/dodal/issues/700 but will probably spin out a new issue for the CI

jwlodek commented 3 weeks ago

I like the idea of a separate system_tests that have some more rigorous "real" tests with external service dependencies. Something we can run before releases to better catch issues without compromising the simple/fast current suite of tests.

callumforrester commented 3 weeks ago

@jwlodek agreed but I think running it just around releases is not often enough. Given containerised IOCs etc. there's no reason not to also run it in CI for at least every PR.