Make a simulated detector that can write HDF files

[coretl]

Based on the code here: https://github.com/dls-controls/bluefly/blob/master/bluefly/areadetector_sim.py

### Tasks
- [x] create a device accepting HDFWriter in params https://github.com/dls-controls/bluefly/blob/master/bluefly/areadetector.py
- [x] create a test for writing to a local filesystem
- [x] add basic detection capability with `sim` option
- [x] run the device in a basic plan
- [x] end to end test of the plan, device and the resulting file

[stan-dot]

@coretl what is the classname and location? should it be in core, epics, or somewhere else? should I make a simprovider like over there? https://github.com/dls-controls/bluefly/blob/master/bluefly/simprovider.py or just use simsignal backend?

[coretl]

Please can you make:

• ophyd_async.sim.PatternGenerator a class that is not a Device, but does the logic from https://github.com/dls-controls/bluefly/blob/master/bluefly/areadetector.py with methods like open_file(), set_exposure(), set_x(), set_y(), write_image_to_file(), etc.
• ophyd_async.sim.SimPatternDetectorControl - subclass of ophyd_async.core.DetectorControl that takes a PatternGenerator at __init__ and sets the exposure and starts a task writing images to file
• ophyd_async.sim.SimPatternDetectorWriter - subclass of ophyd_async.core.DetectorWriter that takes a PatternGenerator at __init__ and opens and closes file and reports how many frames are written on request
• ophyd_async.sim.SimPatternDetector - subclass of ophyd_async.core.StandardDetector that makes the above 3 items

At some point in the future we will make a SimSample that has x and y motors which call set_x and set_y on the PatternGenerator

[coretl]

The bluefly code is for the maths only, the rest of the framework has been superceded by ophyd-async

[stan-dot]

Must be supplied instances of classes that inherit from DetectorControl and DetectorData, to dictate how the detector will be controlled (i.e. arming and disarming) as well as how the detector data will be written (i.e. opening and closing the writer, and handling data writing indices).

that's the description of ophyd_async.core.StandardDetector which I take to mean that the Control and Writer objects need to be passed to the SimPatternDetector constructor, not made inside of it

[coretl]

The aim is to make something that looks like: https://github.com/DiamondLightSource/dodal/blob/directory_provider/src/dodal/devices/areadetector/pilatus.py

so subclass, then make the Control and Writer in __init__ passing them to the superclass __init__

[stan-dot]

I'm slowly getting more familiar with asyncio, h5py and a couple of things are not clear to me.

1. which processes are initialized as Tasks and which not?
2. should I use DirectoryProvider?
3. what should the deadtime value be?
4. how is the holding of state spread out? One way is for PatternGenerator to holder exposure, x, y, image array and the file handle, writer holding indices_written. or should PatternGenerator have no state?
5. why should patternGenerator have all those methods: open & write to file and set x, y?

[coretl]

1. which processes are initialized as Tasks and which not?

I think I gave you the wrong link, it shouldn't be areadetector.py from bluefly, but https://github.com/dls-controls/bluefly/blob/master/bluefly/areadetector_sim.py. There aren't any Tasks in there.

2. should I use DirectoryProvider?

Yes please, to know where to write the file.

3. what should the deadtime value be?

It doesn't really matter, but let's make it 1ms so it looks like a real detector.

4. how is the holding of `state` spread out? One way is for PatternGenerator to holder exposure, x, y, image array and the file handle, writer holding indices_written. or should PatternGenerator have no state?

PatternGenerator should hold the state.

5. why should patternGenerator have all those methods: open & write to file and set x, y?

So that:

• DetectorWriter can open and close file
• DetectorControl can set exposure and periodically make it write a new frame into it
• The future SimMotor can set x and y which will change the results of the next frame

[stan-dot]

in the comment before you mention:

ophyd_async.sim.SimPatternDetectorControl - subclass of ophyd_async.core.DetectorControl that takes a PatternGenerator at init and sets the exposure and starts a task writing images to file

so I assumed that there's a Task there somewhere

[stan-dot]

RuntimeError: Unable to start swmr writing (file superblock version - should be at least 3) working on solving this

[stan-dot]

not sure what is the reasonf ror the logic in this method. in what occasion would self._file be empty (uninitalized) here? why not initalize in init / open?

[coretl]

because you can't get the full filename the HDF writer will write until the first frame comes in...

[stan-dot]

and how does that impact that stream resources are yielded and only later stream data?

this is quite confusing

should then the self.directory_provider be initialized in init? I was assuming that open_file is always called first

[coretl]

and how does that impact that stream resources are yielded and only later stream data?

We defer yielding StreamResource until the file is open and then emit it along with the first StreamDatum

should then the self.directory_provider be initialized in init? I was assuming that open_file is always called first

Do you mean should we call self._directory_provider() in __init__? No, we need to call it once per file we need to open

[stan-dot]

To clarify, this problem is related to handling of NumPy's Unicode string type. HDF5 (and h5py) don't support this type. Details here: h5py: What about NumPy’s U type?

potential numpy types issue

[stan-dot]

Failed validating 'additionalProperties' in schema['patternProperties']['^([^./]+)$']: {'additionalProperties': False, 'patternProperties': {'^([^./]+)$': {'$ref': '#/definitions/DataType'}}, 'title': 'DataType'}

typecheck for descriptor fails On instance['data_keys']: {'/entry/data/data': {'dtype': 'array', 'external': 'STREAM:', 'object_name': 'PATTERN1', 'shape': (1, 240, 320), 'source': 'sim:///entry/data/data'}, '/entry/sum': {'dtype': 'array', 'external': 'STREAM:', 'object_name': 'PATTERN1', 'shape': (1,), 'source': 'sim:///entry/sum'}}

[stan-dot]

File "/scratch/xma12127/projects/forks/ophyd-async/src/ophyd_async/sim/SimPatternDetectorControl.py", line 55, in _coroutine_for_image_writing await self.pattern_generator.write_image_to_file() File "/scratch/xma12127/projects/forks/ophyd-async/src/ophyd_async/sim/PatternGenerator.py", line 186, in write_image_to_file self._handle_for_h5_file[DATA_PATH].resize(target_dimensions) File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper File "/venv/lib/python3.9/site-packages/h5py/_hl/group.py", line 357, in __getitem__ oid = h5o.open(self.id, self._e(name), lapl=self._lapl) File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper File "h5py/h5o.pyx", line 189, in h5py.h5o.open KeyError: 'Unable to synchronously open object (component not found)'

threads here do not succeed. did not expect the .resize line to fail of all the possible ones

[stan-dot]

CI fails tests with: =========================== short test summary info ============================ FAILED tests/core/test_flyer.py::test_hardware_triggered_flyable - RuntimeError: Collect now emits EventPages (stream=False), so emitting Events (stream=True) is no longer supported ================== 1 failed, 185 passed, 1 skipped in 11.77s =================== ERROR: InvocationError for command /opt/hostedtoolcache/Python/3.10.13/x64/bin/pytest --cov=ophyd_async --cov-report term --cov-report xml:cov.xml (exited with code 1) ___________________________________ summary ____________________________________ ERROR: pytest: commands failed

and lint with nested conftest