Store operational information locally for use outside of bluesky

[prjemian]

Users want to see a list of reflections, samples, etc. from their recent session, even after bluesky stops. Multiple sessions could (optionally) access the same list or use a different one.

This would be a local file, stored in the pwd. A session could load this file by default, start a new one, or load from somewhere else.

@strempfer : Thanks for the suggestion!

[prjemian]

@rodolakis - You described an implementation of this idea that uses EPICS PVs. It was useful for when the bluesky session needed to be restarted. (So that any session info could be restored.)

[prjemian]

One way is to generalize the support so it is easy and consistent to call. Consider this class:

from dataclasses import dataclass

from hkl.diffract import Diffractometer

@dataclass
class DiffractometerPreservation:
    """Preserve Diffractometer Configuration."""

    meter: Diffractometer

    def export_dict(self):
        me = self.meter

        d = {
            "name": me.name,
            "geometry": me.calc._geometry.name_get(),
            "class": me.__class__.__name__,
            "engine": me.calc.engine.name,
            "mode": me.calc.engine.mode,
            "class": me.__class__.__name__,
            "hklpy_version": me._hklpy_version_,
            "energy_keV": me.calc.energy,
            "wavelength_angstrom": me.calc.wavelength,
            # fmt: off
            "constraints": {
                k: {
                    nm: getattr(v, nm)
                    for nm in "low_limit high_limit value fit".split()
                }
                for k, v in me._constraints_dict.items()
            },
            "samples": {
                sname: {
                    "name": sample.name,
                    "reflections": sample.reflections_details,
                    "U": sample.reflections_details,
                    "UB": sample.reflections_details,
                }
                for sname, sample in me.calc._samples.items()
            },
            # fmt: on
            "real_axes": list(me.RealPosition._fields),
            "reciprocal_axes": list(me.PseudoPosition._fields),
        }
        return d

    def export_json(self):
        import json

        return json.dumps(self.export_dict())

    def export_yaml(self):
        import yaml

        return yaml.dump(self.export_dict())

An agent could be created: agent = DiffractometerPreservation(e4cv). The default configuration could be exported as JSON by calling agent.export_json():

{"name": "e4cv", "geometry": "E4CV", "class": "SimulatedE4CV", "engine": "hkl", "mode": "bissector", "hklpy_version": "1.0.3", "energy_keV": 8.050921974025975, "wavelength_angstrom": 1.54, "constraints": {"omega": {"low_limit": -180.0, "high_limit": 180.0, "value": 0.0, "fit": true}, "chi": {"low_limit": -180.0, "high_limit": 180.0, "value": 0.0, "fit": true}, "phi": {"low_limit": -180.0, "high_limit": 180.0, "value": 0.0, "fit": true}, "tth": {"low_limit": -180.0, "high_limit": 180.0, "value": 0.0, "fit": true}}, "samples": {"main": {"name": "main", "reflections": [], "U": [], "UB": []}}, "real_axes": ["omega", "chi", "phi", "tth"], "reciprocal_axes": ["h", "k", "l"]}

When orientation info exists for a sample (sample and reflections are added and UB matrix is computed):

{"name": "e4cv", "geometry": "E4CV", "class": "SimulatedE4CV", "engine": "hkl", "mode": "bissector", "hklpy_version": "1.0.3", "energy_keV": 8.050921974025975, "wavelength_angstrom": 1.54, "constraints": {"omega": {"low_limit": -180.0, "high_limit": 180.0, "value": 0.0, "fit": true}, "chi": {"low_limit": -180.0, "high_limit": 180.0, "value": 0.0, "fit": true}, "phi": {"low_limit": -180.0, "high_limit": 180.0, "value": 0.0, "fit": true}, "tth": {"low_limit": -180.0, "high_limit": 180.0, "value": 0.0, "fit": true}}, "samples": {"main": {"name": "main", "reflections": [], "U": [[1.0, 0.0, 0.0], [0.0, 1.0, 0.0], [0.0, 0.0, 1.0]], "UB": [[4.079990459207523, -2.4982736282101165e-16, -2.4982736282101165e-16], [0.0, 4.079990459207523, -2.4982736282101165e-16], [0.0, 0.0, 4.079990459207523]]}, "silicon": {"name": "silicon", "reflections": [{"reflection": {"h": 4.0, "k": 0.0, "l": 0.0}, "flag": 1, "wavelength": 1.54, "position": {"omega": -145.451, "chi": 0.0, "phi": 0.0, "tth": 69.0966}, "orientation_reflection": true}, {"reflection": {"h": 0.0, "k": 4.0, "l": 0.0}, "flag": 1, "wavelength": 1.54, "position": {"omega": -145.451, "chi": 0.0, "phi": 90.0, "tth": 69.0966}, "orientation_reflection": true}], "U": [[-1.2217304763832569e-05, -0.9999999999253688, 0.0], [0.0, 0.0, 1.0], [-0.9999999999253688, 1.2217304763832569e-05, 0.0]], "UB": [[-1.4134284639502065e-05, -1.1569069374686927, 7.084098436944218e-17], [0.0, 0.0, 1.156906937555034], [-1.1569069374686927, 1.4134284639572905e-05, 7.083925341879798e-17]]}}, "real_axes": ["omega", "chi", "phi", "tth"], "reciprocal_axes": ["h", "k", "l"]}

Export as YAML is demonstrated, above. Could add exporters to file, EPICS PV (CA or PVA), or other.

Importers would have to match the geometry of the diffractometer object first. Basically a reverse of the export steps. Convert from incoming format to a standard dict, then a single code that takes the standard dict and updates the diffractometer object.

[prjemian]

The export_dict() method (above) could be part of the Diffractometer() class.

[prjemian]

While hklpy should not design how to persist such information outside the package, it can provide an easy way to import & export. The export_dict() method above is an example.

[prjemian]

Not necessary for the 1.0.4 release.

[prjemian]

The dictionary stored as a run's configuration: diffractometer.read_configuration() but this dict only contains a single sample.

[ambarb]

@prjemian I am reading through the associated PR (https://github.com/bluesky/hklpy/pull/279) and trying to get a more clear picture before I can clearly comment. Can you confirm if my understanding is correct or add to it.

Is the objective here to make a file in a user specified directory that contains python/bluesky friendly information about crystal reflections and lattice parameters and diffractometer geo's, akin to the .or file of spec?

You have provided the tools to save and load "the .or like file" (export/restore) using the nested dictionaries (already provisioned in earlier releases). You give the ability to use .json or .ymal and higher level functions for the front-end user for export/restore.

It appears that multiple samples are now allowed for one file, with as many reflections "stored" as one wants per sample. Is this new, and I never noticed before? I see the value here for film versus substrate or 2 different competing cystallographic phases or other instances when more than one UB matrix could be used to navigate reciprocal space for a SINGLE SAMPLE.

But do you envision people using this as a database-like way for high-throughput crystallography?

I haven't seen it yet in the PR, but what about multiple calc engines and constraints within one experiment? Or in this case, do you think it is cleaner to just change the calc engine as one goes (either in scripts or manually )

[prjemian]

@ambarb: Thanks for your comments. Your understanding is correct, yet this PR stops short of writing any files. It only provides examples how to write & read such files.

The end goal is to be able to export and restore crystal orientation (and other) information in case the session crashes. This addition establishes the structure (the API) to save & restore a complete configuration. Exactly how the configuration is saved (local file, EPICS, database, ...) is not part of this PR. That's for the next phase, once this interface is defined.

Support for multiple samples has been there but, as you notice, has been lurking in the background. You describe good use cases (film & substrate, multi-phase). My first imagination on finding the sample dict was a bicrystal, but hklpy and libhkl only consider one oriented crystal at a time with the present code. With an upgrade (where libhkl is called only when needed), support for multi-phase could become easier.

The new API might facilitate high-throughput should it be possible to create the requisite orientation information first. Once known, it is easy to create one of the API formats (dict, JSON string, or YAML string).

Multiple calc engines? How can we describe their configuration? Are they specific to a specific backend library (such as libhkl)?

Multiple constraints is another aspect where the configuration might better be described as a list of constraints. Perhaps allowing for a more complex rule, such as this hypothetical: $20 <= \omega < 44.1$ and $44.2 <= \omega < 140$. What do you think about that? Presently, the code allows for a limited description of constraints, one for each real-space axis. A recent addition allows for a stack of these constraints (implemented as a Python list), but still only one constraint per axis.

[prjemian]

The API should include (somewhere) the version of its structure. That could be compared as needed for future compatibility as the structure evolves.

[prjemian]

@ambarb

Is the objective here to make a file ... , akin to the .or file of spec?

Nice suggestion. Seems like it must be. I will see what these files contain.

[prjemian]

@ambarb: Does this page describe the .or files? https://www.certif.com/spec_manual/fourc_4_13.html

[prjemian]

could add schema version to this structure

Version of hklpy is included in the API. That implies a schema version. Good enough?

[prjemian]

    "datetime": "2023-10-28 14:37:16.925861",
    "energy_keV": 8.050921974025975,
    "engine": "hkl",
    "hklpy_version": "1.0.5.dev86+g5541eee",
    "library_version": "v5.0.0.3001",
    "python_class": "SimulatedE4CV"