Add multi-detector support for NXxas

[padraic-shafer]

Background

X-ray absorption spectroscopy (XAS) is routinely performed with multiple detectors simultaneously collecting data during the measurement. Each detector measures a different physical mechanism that is proportional to a material's absorption properties, to account for multiple probing depths, multiple decay channels, or experimental constraints. Currently the NXxas definition recognizes only a single detector: absorbed_beam.

While the details of each detector vary across instruments and facilities, there are a small number of basic detection schemes that are employed consistently for XAS:

• total electron yield (TEY)
• fluorescence yield (FY): either total (TFY), partial (PFY), or inverse partial (iPFY)
• luminescence yield (LY)
• transmission (TX)
• and perhaps a few others

Attempted Workarounds (and why not to use them)

Storing data from multiple simultaneous detectors in the NXxas scheme today could be accommodated by:

1. adding optional (unspecified) NXdetector groups within NXxas: INSTRUMENT; or
2. adding multiple NXxas entries in the data file, with one detector per NXxas entry.

An issue with approach (1) is that without specifying additional detectors in the definition, they will go overlooked by automated data processing and viewing routines. Yet the scientists who collect and analyze this data will be expecting the information from these detectors, and so will be tempted to define their own detector fields. This will likely result in a multitudinous jumble of detector names ("fluorescence", "fluorescence yield", "fluor", "fl", "FY", "fy", etc.) thereby defeating the purpose of a common data format.

Approach (2) would require the overhead of the NXxas to be repeatedly declared, even though the intent is simply to indicate an additional data channel. Including the multiple detectors within a single NXxas entry would be clearer and more representative of the physical experiment; whereas multiple NXxas entries could be misinterpreted as spectra measured at different times or from different samples. Additionally there would still be no consistent mechanism for determine the type of signal being stored in the absorbed_beam field, thereby hindering proper interpretation and analysis of the data.

Proposed Solution

Update the NXxas definition to include (in the INSTRUMENT group) multiple named NXdetector groups for commonly used detection schemes. Here are some preliminary suggestions:

• total_electron_yield[nP]: (optional) NXdetector
• fluorescence_yield[nP]: (optional) NXdetector
• total_fluorescence_yield[nP]: (optional) NXdetector
• partial_fluorescence_yield[nP]: (optional) NXdetector
• inverse_partial_fluorescence_yield[nP]: (optional) NXdetector
• luminescence_yield[nP]: (optional) NXdetector
• transmission_yield[nP]: (optional) NXdetector

The names of these fields are indicative of standard detection schemes employed routinely at light sources and research labs worldwide. This would allow researchers to properly analyze and correct the data based on known physical models of each mechanism; e.g., to extract the sample's true absorption coefficients the "standard" correction routines for total electron yield differ from those for total fluorescence yield.

Future role of absorbed_beam

The only detector currently required in NXxas is the ambiguously named absorbed_beam. This name is problematic because some detectors measure absorption (that is, their signal intensity increases "proportionally" to the number of photons being absorbed by the sample) while others measure transmission (that is, their signal intensity decreases exponentially with the thickness of the sample due to absorption by the sample). The NXxas definition provides no guidance on what class of signal was measured, and therefore no indication of how to interpret or process this data.

absorbed_beam could be removed from NXxas and replaced by the new detectors (suggested above) to avoid this ambiguity. However that would leave NXxas with no required detector. For backward compatibility, absorbed_beam could remain in the NXxas definition, to be filled with dummy data (value = '1' for every array index, nP) or could point to the data of one of the other detectors.

[padraic-shafer]

@dylanmcreynolds, @JulReinhardt, @paglans, and I are willing to submit a PR. We are open to feedback on this issue and our proposed solution.

[dylanmcreynolds]

This seems to be a related issue with NXmx, with a slightly different proposal: https://github.com/nexusformat/definitions/issues/940

[newville]

With this issue now being a year old and showing little progress, I feel compelled to voice my opinion here. Yes, being able to convey data from multi-element data would be worth considering. It is not the only problem with this definition.

I find the XAS definition for NeXus to be essentially useless. It looks to have been written by people who have not ever done XAS and who are unfamiliar with the XAS literature, which has defined terms and descriptions of metadata. In conversations with other spectroscopists, I can report real concerns about the combined reality that facilities are beginning to require the nominal use of NeXus as an exchange format without regard to the actual quality of the definitions, and the lack of basic fitness-for-purpose of the XAS definition.

I can completely believe this will not be a popular view among NeXus maintainers, but either the XAS definition needs to change to match how spectroscopists actually exchange data, or the basic premise of NeXus will be revealed as bogus.

I do not understand the process for changing these definitions.
How are they curated, reviewed, and maintained?

[prjemian]

A PR would be welcomed, especially from someone with the community. You (or anyone) could fork the repo, ake changes, and submit a PR.

NeXus holds monthly teleconferences, the next one is less than 12 hours from now. See http://www.nexusformat.org/Teleconferences.html for the details on that. Discussion of a PR is a good topic for one of these teleconferences, when any interested parties can participate.

[padraic-shafer]

Here is a work in progress. It's not quite PR-ready, but it could be made ready for next month's call.

• NXmultixas
  • It allows for multiple named signals (which are denoted as either absorptive or transmissive)
  • It is aware of x-ray polarization
  • It uses the proposed NXamplifier class for each signal
  • If the value of a configuration setting does not change for each energy (i.e., it is constant) then it can be specified as a 1-point array.
• NXamplifier
  • Tracks gain/sensitvity
  • Optionally is aware of saturation values
  • Has input and output units
  • Can be chained for multi-stage amplification

In any case, early feedback is welcomed.

[newville]

@prjemian Thanks. Sorry that I was not able to attend this teleconference, I will look for future ones. I guess my first question would be: what is (or should be) the process for making these definitions? In the XAS community, we have had working groups, meetings, discussions, and results about this sort of thing. I imagine that other groups - say IUCr commissions - have had similar discussions. I don't know how those discussions (or even actual written recommendations, format specs, etc) get turned into maintained definitions. Is there a formal process or review?

On the XAS definition itself, yes, I would be happy to propose changes. In fact, those would be informed by such an XAS community group and appropriate IUCr commission.

I would think that NXmultixas would be the same as Nxxas. For sure, many XAS measurements would use many different detector channels simultaneously.

The usage of terms like "absorbed beam" is confusing. Maybe this is overly pedantic, but one does not measure "beams", one measures "intensities". Worse, using "absorbed" when one probably means "transmitted" is a disaster - they are exact opposites. It is possible to measure "absorption" with, say, total photo-current generated in a sample -- that is extremely rare. Measuring transmission is very common. Measuring emission in one of many modes (X-ray or electrons out) is very common, but data measured in transmission and emission (or actual absorption) are dealt with in different ways.

The use of "yield" is also problematic: yield is a fractional value (and typically meant to convey a fraction of total emission from any one decay channel) whereas an "intensity" ("fluoresced", "electron yield", etc) does not imply fraction -- and so will clearly need scaling to the "incident intensity" channel.

I find "monochromator" to be confusing: does that describe the physical object or the array of X-ray energies reflected from the monochromator? As it happens, yes the "form" of the monochromator (one might assume double-crystal monochromator for XAS unless otherwise noted), and ideally the actual d-spacing used to turn angle into Energy are "highly recommended" values for XAS data.

Including polarization is a good idea. That would need to be in some reference frame, right?

For NXamplifier, I would recommend considering an "Intensity Counter" which might include ion chambers and PIN diodes, and other integrating detectors. This would include not only the settings of a current-to-voltage amplifier but also the voltage-to-counter chain, and the physical properties (gas mixture or semiconductor material, thickness) of the detector itself. Together, those could then be used to turn the measured signals back into actual X-ray fluence.

[phyy-nx]

Hi @newville, turns out your question on how the NIAC moves on definitions was also asked in the Telco. Quote from the minutes, which I've embellished some here:

General requirements for changing definitions:

• Modification in Base Classes or Application Definitions which affect reading and writing files: a NIAC vote is needed, which typically happens during code camps (if we have sufficient attendees) or during the official NIAC meetings.
• Fixing documentation or bugs: PR and code review is needed. Anyone can issue a PR, but it is suggested to work in a small groups (with domain experts).

I can't review your XAS suggestions (not my field). @sanbrock do we have a date for the April Telco? @pshafer-als, @newville perhaps you would like to join for a larger discussion on XAS?

[newville]

@phyy-nx Thanks for the reply. As I said earlier, I would be willing to join a tele-meeting to discuss this. I may be willing to go into the technical details, but I think my main concerns are governance and review processes.

There appears to be an expectation from people who are using and/or promoting NeXus that the definitions it maintains and supports are actually useful and reflect current usage by the respective scientific communities. If the NIAC is assuming that role and claiming that the NeXus definitions are actually standards, it should explain its review process and show evidence of that process.

The current XAS definition suggests serious problems with whatever review processes are in place.

The 2014 paper on NeXus in J. Applied Crystallography says:

8. Governance
The development of NeXus is overseen by the NIAC (NIAC, 2014[NIAC (2014b). NeXus International Advisory Committee (NIAC), https://wiki.nexusformat.org/NIAC.]b). The NIAC seeks a balanced representation of the international community. Most major neutron, X-ray and muon facilities have appointed delegates. Other facilities are invited to join.

The NIAC reviews any proposed amendments to the NeXus base classes and application definitions, and holds online votes to ratify changes. A great number of candidate NeXus application definitions exist which were derived from our understanding of the technique described. For each of these, the NeXus team seeks community approval.

How was community approval sought for this definition?

[padraic-shafer]

@phyy-nx I won't be able to join the April 19 call, but hopefully there are other ways to get involved in the discussion?

[newville]

Sorry if this is wrong place to ask, but is this topic on the agenda for the April 19 teleconference? I'm sort of unclear on where and how to continue this discussion and how to think about XAS and related methods within NeXus.

[rayosborn]

"There appears to be an expectation from people who are using and/or promoting NeXus that the definitions it maintains and supports are actually useful and reflect current usage by the respective scientific communities. If the NIAC is assuming that role and claiming that the NeXus definitions are actually standards, it should explain its review process and show evidence of that process."

@newville, sorry not to comment when you first posted, but I thought a little background might be worth adding before tomorrow's telco. This represents my perspective, anyway. NIAC takes its duties of maintaining a coherent standard very seriously, and there are vigorous and generally well-informed debates about definitions in the core base classes. However, NIAC membership is still a volunteer activity, and with only one member per facility it is impossible to have sufficient expertise to cover all neutron and x-ray instrumentation equally well. So the status of application definitions, like NXxas, is much more patchy. Some definitions are the result of communities getting together outside NIAC to generate detailed application definitions that represent a broad consensus (e.g., NXmx and NXcanSAS), whereas others may have been thrown together nearly twenty years ago when application definitions first started getting discussed, by one or two keen individuals who wanted to get the ball rolling. There is, therefore, the possibility of historical cruft, and it will probably not be trivial to decide which application definitions are really reliable and which are not.

I don't know the history of the NXxas definition, but a look at Git Blame shows that NXxas was added in 2010 along with several others in a group commit, with only minor revisions since. If you think it doesn't meet the needs of your community, then I would encourage you to team together with other XAS scientists to propose revisions, which I am sure NIAC will welcome. Ultimately, NeXus requires crowd-sourcing to get it right, at least until the facilities invest more resources in maintaining NeXus. Your own interest is an important first step in making NXxas better.

Your concerns do however suggest that we need to find a way to identify which category best describes a particular application definition, perhaps by providing some provenance information or defining a maintainer. I can only attend the first half hour tomorrow, but I believe this is something well worth discussing.

[phyy-nx]

Hi @newville, I'm putting together the agenda for the Apr 19 telco now. I can add this to the agenda if you'd like to join us, and we can pass on any comments to @pshafer-als using this thread.

Regarding your comment on governance, allow me to offer up an example, NXmx, as it's the one I'm most familiar with. I recommend this paper: https://journals.iucr.org/m/issues/2020/05/00/ti5018/index.html

NXmx is a great example of how standards are updated and improved through the NIAC. A small working group was formed to define exactly what we needed for MX, and we put together a series of proposals. We created github issues to define specific problems, and then created pull requests that fixed those problems. The PRs were iterated on, voted on, and then merged.

I similar process is starting here. I like where @pshafer-als is going, creating a new base class (NXamplifier) and a new contributed definition (NXmultixas) which uses it, but I doubt it resolves all the problems pointed out in original post, which is complex. I recommend the following:

• Create a PR for NXamplifier only
• Create a PR for NXmultixas that is based on the NXamplifier branch
• Note in these PRs how much of this issue (#1011) is resolved by them
• Break up any problems not resolved by these PRs into separate issues. Then consider either closing this one, or rescoping it.

When this is done, we can discuss it on Telcos and/or the upcoming code camp in June. June is a great target for pushing this work through as we can get quorum for votes. Following this approach will be similar to how work of this scale has been done in the past.

[phyy-nx]

Thanks @rayosborn, I see our comments landed at the same time :)

[newville]

@phyy-nx @rayosborn Thanks - I am willing to join the Apr 19 discussion, though I do admit some skepticism. I do not have much familiarity with NeXus definitions but see important discrepancies between the definitions here and with actual usage and terms used in the XAS community.

[benajamin]

I'll just add a bookmark comment for now and edit in something intelligent later. @woutdenolf wants to be included in discussions.

[woutdenolf]

I'll try to bring in other spectroscopists from the ESRF to advice as well.

@maurov: FYI the Nexus application definition for XAS (NXxas) will go through a major revision.

[phyy-nx]

Discussion from Code Camp Summer 2023: waiting for a specific proposal from the XAS community. We recommend this take the form of a pull request to modify the application definition however you like, and once we have that as a proposal, we can provide any feedback and move it through to adoption. We'll leave this issue open.

FYI, here is schedule for the code camp if anyone from XAS want to join to ask questions or discuss this more: https://www.nexusformat.org/CodeCampJune2023.html We have a few more session today, and two more days on Tuesday and Wednesday, June 20-21 Thanks!

[newville]

@phyy-nx @woutdenolf @benajamin Sorry for not being more on top of this, and thanks for the reminder. I have been looking at this, and I think there are still aspects that I don't fully understand, but I have some ideas and/or questions. I could join the code-a-thon on June 20 in the B, C, or D time slots to discuss details. I'm less available on Wednesday. Sorry for the short notice.

[phyy-nx]

Hi, no worries. I'd say Session B will have the highest attendance. Stop on by!

[phyy-nx]

From Code Camp, questions from @newville:

• What NXclass should the element and energy edge being probed be recorded? Suggestion from committee: first look at existing possibilities (NXreflection, NXscan, NXmonitor). Then, feel free to propose a new base class with this information
• Need to store a data table with channel information (spec scan), columns and rows. This is typically done with fields with column names and 1D arrays. But here the column names can vary. So consider NXdata as a 'spreadsheet' instead, then perhaps document the most common names in the application definition.