search

NeurodataWithoutBorders / lindi

Linked Data Interface (LINDI) - cloud-friendly access to NWB data
BSD 3-Clause "New" or "Revised" License
4 stars 1 forks source link

LINDI does not differentiate between datasets and links to datasets #70

Closed rly closed 6 months ago

rly commented 6 months ago

In Dandiset 000054, the NWB files have datasets at the paths:

processing/ophys/Fluorescence/Deconvolved/rois
processing/ophys/Fluorescence/Neuropil/rois
processing/ophys/Fluorescence/RoiResponseSeries/rois

The first two are links to processing/ophys/Fluorescence/RoiResponseSeries/rois. In the LINDI JSON, these are represented as:

    "processing/ophys/Fluorescence/Deconvolved/rois/.zattrs": {
      "description": "region for Imaging plane0",
      "namespace": "hdmf-common",
      "neurodata_type": "DynamicTableRegion",
      "object_id": "f5bbf768-f39f-4139-b4dc-08f71abb157d",
      "table": {
        "_REFERENCE": {
          "object_id": "71e75a45-6ea9-42e3-9ec7-e5a45c45c6fd",
          "path": "/processing/ophys/ImageSegmentation/PlaneSegmentation",
          "source": ".",
          "source_object_id": "ef7da125-0bb8-4138-ac16-bce3a35b1a8f"
        }
      }
    },
    "processing/ophys/Fluorescence/Neuropil/rois/.zattrs": {
      "description": "region for Imaging plane0",
      "namespace": "hdmf-common",
      "neurodata_type": "DynamicTableRegion",
      "object_id": "f5bbf768-f39f-4139-b4dc-08f71abb157d",
      "table": {
        "_REFERENCE": {
          "object_id": "71e75a45-6ea9-42e3-9ec7-e5a45c45c6fd",
          "path": "/processing/ophys/ImageSegmentation/PlaneSegmentation",
          "source": ".",
          "source_object_id": "ef7da125-0bb8-4138-ac16-bce3a35b1a8f"
        }
      }
    },
    "processing/ophys/Fluorescence/RoiResponseSeries/rois/.zattrs": {
      "description": "region for Imaging plane0",
      "namespace": "hdmf-common",
      "neurodata_type": "DynamicTableRegion",
      "object_id": "f5bbf768-f39f-4139-b4dc-08f71abb157d",
      "table": {
        "_REFERENCE": {
          "object_id": "71e75a45-6ea9-42e3-9ec7-e5a45c45c6fd",
          "path": "/processing/ophys/ImageSegmentation/PlaneSegmentation",
          "source": ".",
          "source_object_id": "ef7da125-0bb8-4138-ac16-bce3a35b1a8f"
        }
      }
    },

There does not appear to be a differentiation between a linked dataset and a dataset. This causes an issue when calling nwbfile.objects because the object IDs are expected to be unique. But more importantly, this issue means the JSON does not accurately reflect the hdf5 structure.

I think processing/ophys/Fluorescence/Deconvolved/rois/ and processing/ophys/Fluorescence/Neuropil/rois should be soft links.

I'll take a look at this later this weekend.