In Scipp, we are considering the addition of a DataGroup container. This would be similar to Dataset, but without coords and without restricting the dims or shapes of the items. This is thus quite similar to a Nexus "group". We would therefore like to support loading groups in ScippNexus, returning a DataGroup. There are a number of things to consider:
57 changed __getitem__ to return Python scalars instead of scipp.Variable if not shape are unit is given. This was for more convenient storage in a Python dict. For DataGroup, the we are currently leaning towards requiring items to have dims and shape. Should we thus undo this change in ScippNexus? Or should DataGroup be more flexible?
How to handle errors while loading? In practice these are unfortunately quite frequent, so failing the entire load may not be so useful. One promising approach would be to fall back to a "plain" load as (nested) DataGroup, since most errors are from "higher level" logic, such as trying to interpret fields for an NXevent_data or NXdetector group. There are a number of subtleties here, especially implementation wise, as the current design puts some hurdles here.
One subproblem is the handling of coords that fail to insert due to a DimensionError. Currently these are skipped with a warning. We could instead return the entire NXdata as DataGroup, but this would likely not be useful in many cases. But not doing that would be inconsistent.
Handling of links and references. For example, NXevent_data (or its fields) has two places that reference it in SNS files. I think any attempt to do something "smart" like returning a tree with sharing setup will just cause headaches. I feel we should probably make a clear statement to not support this (i.e., data will simply be loaded twice).
In Scipp, we are considering the addition of a
DataGroupcontainer. This would be similar toDataset, but without coords and without restricting the dims or shapes of the items. This is thus quite similar to a Nexus "group". We would therefore like to support loading groups in ScippNexus, returning aDataGroup. There are a number of things to consider:57 changed
__getitem__to return Python scalars instead ofscipp.Variableif not shape are unit is given. This was for more convenient storage in a Pythondict. ForDataGroup, the we are currently leaning towards requiring items to havedimsandshape. Should we thus undo this change in ScippNexus? Or shouldDataGroupbe more flexible?DataGroup, since most errors are from "higher level" logic, such as trying to interpret fields for an NXevent_data or NXdetector group. There are a number of subtleties here, especially implementation wise, as the current design puts some hurdles here.DimensionError. Currently these are skipped with a warning. We could instead return the entireNXdataasDataGroup, but this would likely not be useful in many cases. But not doing that would be inconsistent.