Clarify that neither "axis" nor "axes" attributes are required in NXdata

[prjemian]

The developers of the EPICS area detector plugin have received conflicting advice about NeXus. They are revising the software that writes area detector images into HDF5 files and wish to support the proper protocols. The HDF5 files are produced according to a template file (XML). The template describes where to place data within the HDF5 files. There are templates for NeXus files as well templates for other (non-NeXus) standards.

We must clarify that neither "axis" nor "axes" attributes are required in NXdata or state specific rules and reasons why this must be requirement.

Here is the communication thread:

On 10/8/2014 3:28 PM, Mark Rivers wrote:

Hi Pete,

Thanks, I looked at the document.

[NIAC member] stated that the files we are producing are non-compliant because they don't contain the axes definitions. Can you point me to the place in the manual where this requirement is stated?

Does this mean that if I am doing radiography with a 2K x 2K detector I must create 2 1-D datasets, axis1 and axis2 each containing [0,1,2,3...2047] or else my NeXus file is non-compliant? This seems silly. Sometimes pictures really don't need axes to be meaningful.

Mark

see my response on GitHub: https://github.com/areaDetector/ADCore/issues/31#issuecomment-58884713

[mkoennecke]

Dear Pete, and Mark,

Am 13.10.2014 um 14:40 schrieb Pete R Jemian notifications@github.com<mailto:notifications@github.com>:

The developers of the EPICS area detector plugin have received conflicting advice about NeXus. They are revising the software that writes area detector images into HDF5 files and wish to support the proper protocols. The HDF5 files are produced according to a template file (XML). The template describes where to place data within the HDF5 files. There are templates for NeXus files as well templates for other (non-NeXus) standards.

We must clarify that neither "axis" nor "axes" attributes are required in NXdata or state specific rules and reasons why this must be requirement.

Here is the communication thread:

On 10/8/2014 3:28 PM, Mark Rivers wrote:

Hi Pete,

Thanks, I looked at the document.

[NIAC member] stated that the files we are producing are non-compliant because they don't contain the axes definitions. Can you point me to the place in the manual where this requirement is stated?

Does this mean that if I am doing radiography with a 2K x 2K detector I must create 2 1-D datasets, axis1 and axis2 each containing [0,1,2,3...2047] or else my NeXus file is non-compliant? This seems silly. Sometimes pictures really don't need axes to be meaningful.

The question seems irrelevant to me. Why? With the area detector module I expect only the detector data to be written much like the DECTRIS writer. So its should put the detector data in a reasonable place. But I expect anything else, axes assignments, additional metatdata and such to be added by the DAQ software before or after data collection. The area detector module simply does not have all the data to write a complete NeXus file.

Or is the additional data in the XML file? In which case the description of the module would be area detector mixed with a generalized NeXus writer.

Regards,

  Mark

Mark

see my response on GitHub: areaDetector/ADCore#31 (comment)https://github.com/areaDetector/ADCore/issues/31#issuecomment-58884713

— Reply to this email directly or view it on GitHubhttps://github.com/nexusformat/definitions/issues/335.

[prjemian]

The question is most relevant. The data file will be written from the data acquisition and not added to later by DAQ software.

Neither the XML Schema nor the NXDL require either of these attributes. Thus, they are not required. On Oct 13, 2014 7:48 AM, "mkoennecke" notifications@github.com wrote:

Dear Pete, and Mark,

Am 13.10.2014 um 14:40 schrieb Pete R Jemian <notifications@github.com mailto:notifications@github.com>:

The developers of the EPICS area detector plugin have received conflicting advice about NeXus. They are revising the software that writes area detector images into HDF5 files and wish to support the proper protocols. The HDF5 files are produced according to a template file (XML). The template describes where to place data within the HDF5 files. There are templates for NeXus files as well templates for other (non-NeXus) standards.

We must clarify that neither "axis" nor "axes" attributes are required in NXdata or state specific rules and reasons why this must be requirement.

Here is the communication thread:

On 10/8/2014 3:28 PM, Mark Rivers wrote:

Hi Pete,

Thanks, I looked at the document.

[NIAC member] stated that the files we are producing are non-compliant because they don't contain the axes definitions. Can you point me to the place in the manual where this requirement is stated?

Does this mean that if I am doing radiography with a 2K x 2K detector I must create 2 1-D datasets, axis1 and axis2 each containing [0,1,2,3...2047] or else my NeXus file is non-compliant? This seems silly. Sometimes pictures really don't need axes to be meaningful.

The question seems irrelevant to me. Why? With the area detector module I expect only the detector data to be written much like the DECTRIS writer. So its should put the detector data in a reasonable place. But I expect anything else, axes assignments, additional metatdata and such to be added by the DAQ software before or after data collection. The area detector module simply does not have all the data to write a complete NeXus file.

Or is the additional data in the XML file? In which case the description of the module would be area detector mixed with a generalized NeXus writer.

Regards,

Mark

Mark

see my response on GitHub: areaDetector/ADCore#31 (comment)< https://github.com/areaDetector/ADCore/issues/31#issuecomment-58884713>

— Reply to this email directly or view it on GitHub< https://github.com/nexusformat/definitions/issues/335>.

— Reply to this email directly or view it on GitHub https://github.com/nexusformat/definitions/issues/335#issuecomment-58887031 .

[zjttoefs]

As is clear from the referenced thread in areaDetector, I like axes but I do not believe they are a strict requirement. Legacy detectors write TIFF without and people are able to do science. If we make axes a formal requirement we end up with [0,1,2,3,4,..] as default. I rather have a situation where I know an axis dataset when provided is actually useful.

[yayahjb]

It all depends on the experiment, but in most cases, working with a 2-D detector without some information somewhere on the actual positions of the pixels in space leads to confusion and mistakes -- most commonly swapped and flipped axes. While the information does not have to be in the data file, it has to be somewhere, and putting it in the data file can be very useful. I am not sure where the [0,1,2,3, ...] is coming from. We're putting actual pixel offsets in mm. -- Herbert

On Mon, Oct 13, 2014 at 8:40 AM, Pete R Jemian notifications@github.com wrote:

The developers of the EPICS area detector plugin have received conflicting advice about NeXus. They are revising the software that writes area detector images into HDF5 files and wish to support the proper protocols. The HDF5 files are produced according to a template file (XML). The template describes where to place data within the HDF5 files. There are templates for NeXus files as well templates for other (non-NeXus) standards.

We must clarify that neither "axis" nor "axes" attributes are required in NXdata or state specific rules and reasons why this must be requirement.

Here is the communication thread:

On 10/8/2014 3:28 PM, Mark Rivers wrote:

Hi Pete,

Thanks, I looked at the document.

[NIAC member] stated that the files we are producing are non-compliant because they don't contain the axes definitions. Can you point me to the place in the manual where this requirement is stated?

Does this mean that if I am doing radiography with a 2K x 2K detector I must create 2 1-D datasets, axis1 and axis2 each containing [0,1,2,3...2047] or else my NeXus file is non-compliant? This seems silly. Sometimes pictures really don't need axes to be meaningful.

Mark

see my response on GitHub: areaDetector/ADCore#31 (comment) https://github.com/areaDetector/ADCore/issues/31#issuecomment-58884713

— Reply to this email directly or view it on GitHub https://github.com/nexusformat/definitions/issues/335.

[prjemian]

The dataset [0,1,2,3, ...] is the most general instance of a "definition scale" in the trivial case of row or column indices. Without engineering units supplied (which has been a very good suggestion but does not address the question of required or not), this is the common information known.

Why can't this trivial case ([ [0,1,2,3, ...]) be the default condition when neither "axis" nor "axes" are not provided?

[zjttoefs]

In the absence of an axis (however that may be defined) plotting software can pick something and that makes [0,1,2,3,...] a reasonable choice, in the absence of other clues. But the software may infer better values from other information in the file (like the pixel sizes in NXdetector).

Excplicitly having to provide an axis like for formal reasons is what I want to avoid. There is no point is storing meaningless defaults.

[rayosborn]

I think I was the guilty party in telling Mark that axes were required, which is a bit ironic, because I have faced this issue with our own x-ray scattering workflow and share his view that adding extra arrays containing array indices is a bit pointless. The convention has been that we should add axes, but whether that was formalized in a NIAC motion will probably require some archival digging. In the NeXpy GUI (but not the command-line API), if there are no axes, the array indices are offered as alternatives, although if you try to set the signal, it complains if there are no axes specified. I could easily change that, and probably will.

[prjemian]

To summarize, there is no documented rule in any of the NXDL or the XML Schema requiring dimension scales to be defined. Thus, it is not required to define the "axes" attribute or the "axis" attribute.