Reading configuration of devices in a run doesn't work with tiled server

[Bilchreis]

This issue seems somewhat similar to #745

Expected Behavior

With databroker backed by intake I am able to access the configuration parameters of devices like this:

db = databroker.catalog['xyz']
run = db[-1]
run.primary.config['det'].read()

This gives an xarray of device configuration parameters. I would like the same functionality using a databroker with a tiled server like this:

from tiled.client import from_uri
db = from_uri("http://localhost:8000/api")
run = db[-1]
run.primary.config['det'].read()

Current Behavior

I get the following error when I try to access the config data:

HTTPStatusError: Server error '500 Internal Server Error' for url 'http://localhost:8000/api/v1/metadata/df8df11a-28f6-4e43-af52-e9c5722d271a/primary/config/det'
For more information, server admin can search server logs for correlation ID None.

Possible Solution

Steps to Reproduce (for bugs)

from bluesky import RunEngine

from bluesky.plans import scan
from ophyd.sim import det,motor

from tiled.client import from_uri
## Set up env

RE = RunEngine({})
db = from_uri("http://localhost:8000",api_key="secret")

def post_document(name,doc):
    db.post_document(name, doc)

RE.subscribe(post_document)

RE(scan([det],motor,1,2,10))

db[-1].primary.config['motor'].read() # this somehow returns config data

db[-1].primary.config['det'].read() # this fails

Context

I want to use the data inside the configuration parameters to populate a Nexus File . Similarly to the Issue #745 the data is still accessible under:

run.primary.data.metadata['descriptors'][0]['configuration']['det']['data']

Your Environment

databroker==2.0.0b46
tiled==0.1.0b7

[Bilchreis]

So we narrowed it down a little, I had a mismatch between the datatype reported in dtype and the actual value that was returned by some signal i hat. the same thing happens with det and its noise EnumSignal .

noisy_det.noise.describe()
--> {'noisy_det_noise': {'source': 'SIM:noisy_det_noise',
  'dtype': 'integer',
  'shape': [],
  'enum_strs': ('none', 'poisson', 'uniform')}}

det.noise.read()
--> {'noisy_det_noise': {'value': 'uniform', 'timestamp': 1727074877.150907}}

And if there is a single mismatch, run.primary.config is not getting populated. (same thing happens if there is a single mismatch in the read signals). So this explains this behaviour:

db[-1].primary.config['motor'].read() # this returns config data

db[-1].primary.config['det'].read() # this fails

Further investigation:

But after fixing the mismatch in my code, reading run.primary.config still failed. We narrowed it down to a structured numpy arrays. So if any signal contains a strutured numpy array, reading from tield fails.

Example:

So if a Signal similar to this one is present reading the config data from tiled fails.

await gas_dosing.massflow_contr1.status.describe()
--> {'gas_dosing-massflow_contr1-status': {
  'source': 'localhost:10800:gas_dosing:massflow_contr1:status',
  'dtype_str': '|V408',
  'dtype_descr': "[('f0', '<i8'), ('f1', '<U100')]",
  'dtype': 'array',
  'shape': []}
}

await gas_dosing.massflow_contr1.status.read()
--> {'gas_dosing-massflow_contr1-status': {'value': array((100, 'at target'), dtype=[('f0', '<i8'), ('f1', '<U100')]),
  'timestamp': 1727076275.2028334}}

Note: When I use the temp() database everything is stored properly, and accessible.

[danielballan]

Thanks @Bilchreis. It may be next week until we properly follow up, as several of us are at a conference this week.

Two immediate thoughts:

• If data types mismatch in a way that Tiled cannot recover from, it should at least issue a warning in the server logs.
• Numpy structured arrays are intended to be supported, so this should be a fixable bug.