DiamondLightSource / blueapi

Apache License 2.0
6 stars 6 forks source link

Investigate use of write_path_template and read_path_template for mapping file systems between detector and filewriting mounts #290

Closed DiamondJoseph closed 6 months ago

DiamondJoseph commented 1 year ago

If read_path_template is set, this is validated by Ophyd and write_path_template is not.

Which is used to write the ResourceDocument? Can we use a non-matching pair to have the detector write to /data/foo and the filewriter write to /dls/data/foo?

DiamondJoseph commented 1 year ago

From talking with Eva, it is the Read Path that is used in the ResourceDocuments.

Although, this is not as helpful as I had wanted, as the root is still consistent between (with the assumption that root can be changed later so long as the non-root resource path is consistent).

TODO: Mapping between filewriting, detector mounts.

See also https://github.com/bluesky/ophyd/issues/1143

tacaswell commented 1 year ago

The ReadPath and the root have to be consistent and are what go into the resource document. If you provide both, then write path is not validated.

https://github.com/bluesky/ophyd/blob/5c413cc0848ef035b9839de328064d6af47b589b/ophyd/areadetector/filestore_mixins.py#L115-L127

In retrospect it would have been clearer to require read path (and always validate it) and make write path the optional one if you have the same file system mounted differently in two places.

tacaswell commented 1 year ago

As an example (and the motivating case when we wrote this 😉 ) see

https://github.com/NSLS-II-PDF/profile_collection/blob/a9fe5d6987c39b0385d6c3e4429e31a7abfd20cc/startup/80-areadetector.py#L153-L163

which is an IOC that running on a windows machine so the paths look wildly different.

DiamondJoseph commented 1 year ago

We're looking at having IOCs running in containers on Kubernetes, with as little of the file system mounted as possible. And a service for consuming documents and writing Nexus files, again on Kubernetes and mounting as little of the file system as possible. But writing those files such that they are visible from the entire file system and portable. I think eventually we're going to want something like

Staging sets root = "/%visit/" write_path_template = "/data/%year/%month/%day" read_path_template = "%experiment/"

Which we write as links into our output files as ./%experiment/file

tacaswell commented 1 year ago

read_path needs to be where the written file is visible to the consumer of the documents.

It is unrelated to where that downstream consumer is going eventually write derived output.

callumforrester commented 1 year ago

Pending https://github.com/bluesky/ophyd/pull/1147

DiamondJoseph commented 6 months ago

Deprecated by use of ophyd-async and its DirectoryProvider, recent re-thinkings of StreamResource model.