Fairmat 2023

[sanbrock]

Base Classes and Application Definitions to support:

• MPES(ARPES,XPS,PEEM)
• OptSpec(Ellipsometry,Luminescence,Raman)
• EM(EBSD,EDX,EELS)
• APM

[sanbrock]

NXaperture:

• NXpositioner
• shape
• size

NXbeam:

• incident_energy: modified doc
• incident_energy_spread
• incident_energy_weights
• incident_polarization: modified doc
• @ units
• final_polarization: modified doc
• pulse_energy
• average_power
• fluence
• @ units
• pulse_duration
• frog_trace
• frog_delays
• frog_frequencies
• chirp_type
• chirp_GDD

NXinstrument:

• energy_resolution
• momentum_resolution
• angular_resolution
• spatial_resolution
• temporal_resolution
• NXenvironment

NXprocess:

• NXregistration
• NXdistortion
• NXcalibration

NXsample:

• sample_id
• sample_history
• state
• purity
• orientation
• layer
• chemical_name
• chem_id_cas
• gas
• gas_pressure
• surface_dopant
• surface_dopant_coverage
• bias
• growth_method
• vendor
• substrate_material
• substrate_state
• drain_current
• bias_voltage
• notes

NXsensor:

• measurement: added enum item current
• type: added enum item Pt1000

[mkuehbach]

@sanbrock: from all these changes in the above-mentioned base classes I recall that I introduced exactly one: NXsample/sample_history and that still backup to enable the possibility for linking in a parent or connection to a workflow management (system) (work with Area A!)

Let me comment on some of the other cases, personal suggestions: NXsample: (i.e. the concept of a sample in its thermodynamic environment)

• "layer" is too unspecific
• "surfacedopant*" and "growth method" should not be there because they are defined by previous processing steps if they are there they should only be used in a specific appdefs until one has a better way to describe this
• "gas" (make an instance of a chemical i.e. NXion or NXsomething similar) similar thoughts to chem_id_gas rather as id under gas(NXion or NXsomething) -"purity" is vague (NXchemical_composition) instead, see NXapm/entry/sample

NXprocess:

• all base classes inferred in here should either in the future be by default be in there or better (use the sub-classes shown here in the respective appdefs)

NXinstrument:

• I see the value of _resolution but how obtained? why not encapsulate in NXprocess/_resolution again in the appdef although I see value of naming *_resolution the same way across base classes or appdefs but then where to draw the line so that a base class does not become a whole dictionary of a specific human language? You measure the resolution or you assume it because of previous knowledge, i.e. previous measurements which are then assumed to hold valid also for an instance of the next measurements eventually

NXbeam:

• these additions make sense talking about a beam (wave front inspected in a confined spatiotemporal ROI, or group of particles) but pulse_* mmh I think this is related to beam but maybe better modelled as a property of the pulsing unit which emits the beam
• a freely dangling \@units under NXbeam what is this for?

NXaperture:

• I agree but be careful with the term size if the docstring is not more specific and one cannot safely assume a specific shape is expected (circular, ellipsoidal) in which case size may be too vague, in NXapm we had the same for grain_size and we finally decided to go for grain_diameter with an explicit docstring

NXsensor:

• type: Pt1000, I second this but keep in mind there are different types of Pt1000 elements which might be of interest (physical response) a free text field could allow for this, thinking further the idea of enums but extending it to have external references glossaries could be more useful than a plain string list, i.e. the concept type remains conceptually a description of what the enum offers but the resolving can happen using an external resource

[phyy-nx]

Feedback from Code Camp:

• Lots of new application definitions and base classes. The committee recommends restructuring and organizing by topic, potentially using subfolders
• The larger conversation about adding workflows to NeXus was argued following these lines: -- NeXus has always been about raw or at the most lightly calibrated data formats, not processed data. The rationale is because processing software changes constantly, but the means to record data at a facility is slower to change. Further, the decision trees in a workflow are subject to change, to cycles, and the full decision tree is difficult to capture. -- Contrariwise, the provenance of data processing is required in order to satisfy FAIR principles, particularly reproducibility, for example how a figure in a paper was generated from the original data, including all the intervening processing steps. Hence the NXam* files included in this PR
• Comment from @yayahjb: the PR provides many application definitions that are part of a workflow, but is the workflow itself defined? @yayahjb argues that NeXus doesn't provide the tools to connect parts of a workflow together, and the only way to represent workflows fully is add to NeXus the UML-style tools that allow diagrams of steps linked together. This may or not may not be out of scope for NeXus.
• From @sanbrock, consider these new application definitions as extensions of NXprocess. This has implications on how we should organize this work

[phyy-nx]

More feedback from Code Camp discussion:

• Generally the idea of better formalizing workflow provenance is supported by the code camp. We want to think about where we want to go up front and try to make changes that get us there as we go
• To this end, we suggest a multi stage approach. For this PR, change the list of contributed definitions to a list of subcategories with sub pages that have related definitions, for easier navigation of the contribute definitions
• In a future PR, the actual .nxdl.xml files could also be organized info subfolders.
• And finally, a wild comment for the larger community: what about "super application definitions"? In other words, the workflow steps defined as new application definitions here could be related to each other using NXsubentries, and that larger organization could be formalized into a specification itself. In other words, a larger grouping of expected application definitions in a fuil workflow. This would allow capturing full provenance of a data processing project.

[sanbrock]

first step is implemented

[woutdenolf]

The rendered version can be found here: https://hdf5.gitlab-pages.esrf.fr/nexus/fairmat_2023/

[mkuehbach]

The restructuring looks good. However, some subsets of proposed classes in contributed do currently not show up as entries in the HTML summary list of the above-mentioned rendered version: https://hdf5.gitlab-pages.esrf.fr/nexus/fairmat_2023 (was this version generated from commit 8f0b40b using which make which-command?). Unlisted contributed classes include a substantial number of NXcg* base classes and NXms . Is there a reason that these have no corresponding subset page? Some of these classes are searchable though via the quicklink right-hand side navigation pane. This is the respective page of the nexus-fairmat-proposal which includes an equivalent rst cgms-structure.rst file https://github.com/FAIRmat-NFDI/nexus_definitions/blob/fairmat/manual/source/cgms-structure.rst for these NXcg_ and NXms_* contributed classes.

[woutdenolf]

Here is the CI that builds the fairmat_2023 branch:

https://gitlab.esrf.fr/hdf5/nexus/fairmat_2023/-/blob/main/.gitlab-ci.yml

I used make local

git clone https://github.com/FAIRmat-NFDI/nexus_definitions.git definitions
cd definitions
git checkout -t origin/fairmat_2023
make install
make local

Note that I need to run this manually. It doesn't get triggered when you push changes to that branch.

Check when the last build was: https://gitlab.esrf.fr/hdf5/nexus/fairmat_2023/-/pipelines

There is an issue about rendering html for merge requests: https://github.com/nexusformat/definitions/issues/1280

[phyy-nx]

@sanbrock I have a suggestion you could incorporate as part of this PR around the proposed NXcoordinate_system_set about McStas. Here is the current language:

Many application definitions in NeXus refer to this McStas coordinate system.

In reality, all application definitions use McStas (at least that's my understanding), as stated in the design doc. So perhaps it could be simply:

NeXus uses the McStas coordinate system.

And elsewhere you could add a note like:

One use of NXcoordinate_system_set is to transform from McStas to the coordinate system of an application definition.

Reasonable?

[prjemian]

NeXus uses the McStas coordinate system.

Makes it sound like NeXus depends on McStas. That's not the case. The NeXus coordinate system is the same as defined by McStas and this information might be useful for those at neutron instruments. Other standards must also use that coordinate system.

The wording in the design section is succinct.

The NeXus coordinate system is shown below. Note that it is the same as that used by McStas (http://mcstas.org/).

Lots of discussion preceded this wording. Let's stick with that.

[phyy-nx]

👍

[phyy-nx]

Hi @mkuehbach @sanbrock @PeterC-DLS, if you could re-check the unresolved conversations prior to our Telco on Wednesday, that would be great! Thanks!

[PeterC-DLS]

Hi @mkuehbach @sanbrock @PeterC-DLS, if you could re-check the unresolved conversations prior to our Telco on Wednesday, that would be great! Thanks!

I did so yesterday. If you go to the files tab and click on the conversations drop-down, you see there are 38 unresolved and 32 resolved conversations.

[phyy-nx]

Ok the conversations drop-down didn't work for me. Instead I used the "hidden conversations" link on this page to find the unresolved conversations.

[sanbrock]

Please check the modifications and resolve the conversations if the answers are OK.