Improve calibration metadata in pdCIF files

[rowlesmr]

Originally from #46, then a new issue in #53, now a PR.

This initial commit creates PD_CALIB_INTENSITY

[rowlesmr]

Intensity

PD_CALIB_INTENSITY is a Loop category for intensity scaling. Its Loop as you need to be able to loop over all detectors. It's keyed on _pd_calib_intensity.id and _pd_calib_intensity.detector_id.

The category consists of:

• _pd_calib_intensity.block_id: the block_id of the data block containing the diffractogram and phase of the the intensity std (if it was a sample).
• _pd_calib_intensity.detector_id: To identify a detector in a multi-detector system - taken from _pd_calib.detector_id
• _pd_calib_intensity.detector_response: (and *_su) relative sensitivity of a detector in a multi-detector system. Can we specify somehow that this is used to scale _pd_meas.intensity_*
• _pd_calib_intensity.diffractogram_id: the diffractogram_id of the diffractogram collected of the the intensity std (if it was a sample).
• _pd_calib_intensity.id: To identify a particular intensity calibration.
• _pd_calib_intensity.incident_intensity: (and *_su) the value by which _pd_meas.intensity_total values are divided to get _pd_proc.intensity_total. The source of this value could be from a calibration sample, or some beam intensity monitor, or something else - see special_details - but is a constant for the entire diffractogram; see _pd_meas.intensity_monitor.
• _pd_calib_intensity.phase_id: the phase_id of the phase used as the intensity std (if it was a sample).
• _pd_calib_intensity.phase_name: Identity of material used as a standard.
• _pd_calib_intensity.special_details: 'cause sometimes you need the space to give some info

[rowlesmr]

Create PD_CALIB_WAVELENGTH

Wavelength

PD_CALIB_WAVELENGTH is a Loop category for denoting the reference material from which the wavelength was determined (see also _diffrn_radiation_wavelength.determination). Its Loop as you may have multiple phases and diffractograms used in the calibration. It's keyed on _pd_calib_wavelength.id.

The category consists of:

• _pd_calib_wavelength.block_id: the block_id of the data block containing the diffractogram of the the wavelength std.
• _pd_calib_wavelength.diffractogram_id: the diffractogram_id of the diffractogram collected of the the wavelength std.
• _pd_calib_wavelength.id: To identify a particular wavelength calibration.
• _pd_calib_wavelength.phase_id: the phase_id of the phase used as the wavelength std.
• _pd_calib_wavelength.phase_name: Identity of material used as a standard.
• _pd_calib_wavelength.special_details: 'cause sometimes you need the space to give some info

[rowlesmr]

X-coordinate

PD_CALIB_XCOORD is a Loop category for for x-coordinate calibration. Its Loop as you need to be able to loop over all detectors, as well as looping to provide an offset to every data point/many calibration points. It's keyed on _pd_calib_xcoord.id and _pd_calib_xcoord.detector_id. It is able to record calibrations for 2theta, position, and TOF. The enumeration default of _pd_calib_xcoord.id = _pd_calib_xcoord.point_id

The category consists of these basic data items:

• _pd_calib_xcoord.block_id: the block_id of the data block containing the diffractogram used for x-coord calibration, if a reference material was used.
• _pd_calib_xcoord.detector_id: To identify a detector in a multi-detector system
• _pd_calib_xcoord.diffractogram_id: the diffractogram_id of a diffractogram used for x-coord calibration, if a reference material was used. It is possible that more than one diffractogram was used
• _pd_calib_xcoord.id: To identify a particular angle calibration.
• _pd_calib_xcoord.phase_id: the phase_id of a phase used for x-coord calibration, if a reference material was used. This phase can be an internal or external reference material. It is possible that more than one phase was used.
• _pd_calib_xcoord.phase_name: Identity of material(s) used as a reference material, if a reference material was used. .
• _pd_calib_xcoord.special_details: 'cause sometimes you need the space to give some info

and these specific data items:

• _pd_calib_xcoord.point_id: to identify the specific point to which an offset applied.
• _pd_calib_xcoord.2theta: offset angle used to calibrate angle as defined in _pdmeas.2theta*. The offset is added.
• _pd_calib_xcoord.2theta_nominal: the nominal angle to which the associated offset applies.
• _pd_calib_xcoord.position: offset position used to calibrate position as defined in _pd_meas.position. The offset is added.
• _pd_calib_xcoord.position_nominal: the nominal position to which the associated offset applies.
• _pd_calib_xcoord.time_of_flight: offset time_of_flight used to calibrate time_of_flight as defined in _pd_meas.time_of_flight. The offset is added.
• _pd_calib_xcoord.time_of_flight_nominal: the nominal time_of_flight to which the associated offset applies.

I haven't included the equivalent of _pd_calib.2theta_off_min/max in here. Should they be replicated for each x-coordinate?

[jamesrhester]

I'm working through in my head how these categories might be applied to our bespoke setup on the CW neutron Echidna powder diffractometer. As a refresher, Echidna has 128 He3 tube detectors that each scan 1.25 degrees. The detectors are vertically position sensitive, so that we obtain a 2D map after scanning has finished.

Meanwhile, these would benefit from being split into 3 PR requests, one for each type of calibration, as they involve radically different processes. Not sure how easy that is to do.

Wavelength

We collect a standard as usual for most powder instruments, so no particular issues here.

Intensity

We scan the diffraction of a solid V cylinder, which produces a flood field. The inverse of the resulting 2D map is multiplied by the raw data. This is the process covered by _array_intensities.gain from imgCIF. Any area detectors on instruments will also have this type of gain correction applied, so I think we need to be more specific about when to use the intensity calibration described above and when to use imgCIF. I think the conclusion is that these data names apply only where the raw data consist of a single value per x coordinate for both the calibration measurement and data measurement.

X-coord

This is quite complicated for us so if this could be a separate PR that would be great.

[rowlesmr]

I'm working through in my head how these categories might be applied to our bespoke setup on the CW neutron Echidna powder diffractometer.

That's how I try and do everything.

I can split these out into three PRs: wavelength, intensity, and x-coord.

I think the conclusion is that these data names apply only where the raw data consist of a single value per x coordinate for both the calibration measurement and data measurement.

That argument is sound for me; if you've got area data, use an area format. If it's linear, then you can use pdCIF categories. Maybe this can be expanded to the definition of the PD_DATA intensity-type data items.

.

The reasoning behind this (large!) change was to try and consolidate what each calibration category was doing. PD_CALIB was recording the relative sensitivity of each detector id, as well as the internal standard mass percentage for QPA. PD_CALIB_STD refers to an external standard used for either (or both) intensity or angular calibrations. PD_CALIB_OFFSET records offsets in 2Th only. PD_CALIBRATION is a human-readable equation to convert from detector_id/pd_meas.position to 2th/q/E, which I haven't touched (apart from formalising how to convert from d to TOF).

[rowlesmr]

Discussion to carry on in #91 #92 #93