Include additional information in reduced NeXus format

[rbauststfc]

Requested by Max at ISIS.

At the moment not all of the metadata included in the original raw/NeXus file is added to the processed/post-reduction NeXus file. Max is keen for us to investigate how to change the reduced NeXus file to include this additional information. This is expected to help with retrieving data automatically for writing .ort files (see #31073).

It's been noticed that all you have to do to lose data is load and then save out a file. This could have overlap with (or could even be the same behaviour as) #35507.

[rbauststfc]

These are the things I found when looking into this:

• Not all of the metadata from the raw NeXus file is loaded into a Mantid workspace by the load algorithms, and this is acknowledged in the documentation for many of the load algorithms. Having asked around, the convention seems to be that we only load information that we have been told is needed. We don't load everything by default to avoid any performance hits from loading information that isn't required. Additionally each entry has to be added explicitly in the code, so it helps with maintainability to only load the necessary information.

• There are a number of different load algorithms and each one loads slightly different things. For the Reflectometry reduction, we use the LoadNexus algorithm when the reduction doesn't perform any time slicing. If we are time slicing then we use the LoadEventNexus algorithm. The majority of the entries that are pulled into the Sample Logs from both are the same. However, depending on the dataset, there are sometimes slight differences in the period entries (i.e. whether we have current_period, period1 and periods log entries). There are also the following differences that appear consistent across datasets:

  • Logs we only get if we use LoadNexus: freq, goodfrm, nchannels, nspectra, rawfrm, rb_proposal, run_end, tot_prtn_chrg
  • Logs we only get if we use LoadEventNexus: Filename, duration (I think this would be pulled in from LoadNexus too as a log called "dur" if the NeXus file didn't have an isis_vms_compat group), experiment_identifier (this is the same value as the "rb_proposal" log that we get from LoadNexus but is a string data type instead of a number)

• Sometimes logs are changed depending on the operations run on the workspace. A Reflectometry reduction on an INTER dataset, with or without time slicing, will add a log called deltaE-mode to the lvsQ_binned workspace, which I believe is added as part of the convert units step. When we are not time slicing we do not lose any log information. When we are time slicing we lose the running log entry from the lvsQ_binned workspace, which I think is happening because of the behaviour described in #35387 where the monitor logs are the ones retained after slicing.

• If I load a workspace, save it as a NeXus file and then re-load it, I don't lose any log information from the save and re-load steps.

Some technical notes:

• The code seems complex in this area with loading algorithms going through a number of different pathways. When making changes we will need to test the full, load, reduction, save and re-load cycle to ensure that the logs remain sensible throughout.

• If we want to load additional information then we need to ensure compatibility with files that may not contain the entries we want to load. This could be done simply by adding try/catch clauses or using other relevant checks.

• Most system tests shouldn't compare logs by default, but any that do will fail if new logs are added and would require updating as part of any changes.

• I think that loading a small number of extra entries into the logs that are common to all ISIS NeXus files should not be a significant change. I had a very quick go at adding the code below into LoadISISNexusHelper::loadRunDetails to load the user_1/name information when loading via algorithm LoadISISNexus. This loaded the information into the sample log and the new log seemed to be retained after reducing, saving and re-loading. We would need to test carefully to confirm that there were no unexpected knock on impacts, however it seems reasonable to expect that small additions to the logs should be fairly straight-forward. If, however, we want to load a lot of additional entries then I think this becomes a larger piece of work that would need more consideration. I think it's very unlikely we would be looking to make this change within the existing loading algorithms. One option could be adding a new algorithm to load the required missing logs that we could call as a final step in the reduction, but it needs careful thought and possibly some prototyping to test performance etc.

  NXClass user = entry.openNXGroup("user_1");
  NXChar user_name = user.openNXChar("name");
  user_name.load();
  std::string user_name_str = std::string(user_name(), user_name.size());
  run.addProperty("user_name", user_name_str);

[rbauststfc]

Closing as the investigation is now complete and issue #36399 has been opened for the follow-up to this.