bluesky / databroker

Unified API pulling data from multiple sources
https://blueskyproject.io/databroker
BSD 3-Clause "New" or "Revised" License
35 stars 46 forks source link

Add CLI with shape-fixer. #732

Closed danielballan closed 2 years ago

danielballan commented 2 years ago

Description

This adds a CLI to databroker. In this PR it only has one command, but it is expected to grow more.

Motivation and Context

This captures the "shape-fixer" which we use to retroactively edit the shape in the EventDescriptor to match the actual shape. This is intended to make it easily usable by people other than me. :-)

How Has This Been Tested?

The shape-fixer has been used on about ten beamlines. This exact PR branch was run at ISS today.

danielballan commented 2 years ago

Example usage, which we should put into docs once we add documentation about the migration path:

PYTHONPATH=~/.config/tiled/ databroker admin shape-fixer  mongodb://... --asset-registry-uri mongodb://... --query "TimeRange(since='2022-08')" --limit 10 --dry-run --strict --handler 'APB = iss_patches:APBBinFileHandler' --handler 'PIZZABOX_ENC_FILE_TXT_PD = iss_patches:PizzaBoxEncHandlerTxtPD' --handler 'XSP3 = iss_patches:ISSXspress3HDF5Handler' --handler 'APB_TRIGGER = iss_patches:APBTriggerFileHandler' --handler 'PIL100k_HDF5 = iss_patches:ISSPilatusHDF5Handler' --handler 'AD_HDF5 = area_detector_handlers.handlers:AreaDetectorHDF5Handler'
danielballan commented 2 years ago

Example output:

image

danielballan commented 2 years ago

Before merging we should get this working via

databroker ...

not just

python -m databroker.cli ...

We rely on scripts like iss_patches.py being importable from the CWD. Annoyingly, it seems like that using the proper CLI (databroker ...) breaks that somehow. I haven't run into this before.

jklynch commented 2 years ago

This would benefit from automated tests, if not to ensure the code works now then to ensure it doesn't break in the future. I imagine this is difficult because of the dependencies on external services. The solution is to make it easy to generate simulated runs of bluesky documents.

danielballan commented 2 years ago

Agreed.