Virtual Views/Grouping

[hoodriverheather]

Two views/groupings to start:

Group by Organ, AssayType, Site MCL PCA Pilot [Collection]

• Breast [Dataset]
  • DNA Sequencing [Dataset]
    • DNA Files
  • RNA Sequencing [Dataset]
    • RNA Files
  • H&E Images [Dataset]
    • H&E Files
  • …AssayType [Dataset]
• Pancreas [Dataset]
  • DNA Sequencing [Dataset]
    • DNA Files
  • RNA Sequencing [Dataset]
    • RNA Files
  • H&E Images [Dataset]
    • H&E Files
  • …AssayType [Dataset]

Group by Site, then ParticipantID MCL IPMN[Collection]

• MD Anderson [Dataset]
  • Case 001 [Dataset]
    • H&E [Dataset]
      • Svs Files
    • mIF Panel 1 [Dataset]
      • Multiplex Files
    • mIF Panel 2 [Dataset]
      • Multiplex Files
  • Case 002 [Dataset]
    • H&E Files
    • …
• MD Anderson Old [Dataset]
  • Case 001 [Dataset]
    • H&E [Dataset]
      • Svs Files
    • mIF Panel 1 [Dataset]
      • Multiplex Files
    • mIF Panel 2 [Dataset]
      • Multiplex Files
  • Case 002 [Dataset]
    • H&E Files
    • …

Draft Metadata: https://docs.google.com/spreadsheets/d/1ar2SSk4vMK-I7bVMQZK7hff83m36LtV0Eb6COz0ZSac/edit#gid=523331231

Below scenarios (thoughts dump!) were added by Asitang:

Regular File System Search: original organization: -> datasets by sites -> datasets by organs -> files (has attribute type)

• no downward flow:
  • if we want to get all files for a particular organ:
    • solr query:
      • 1. look for all files and datasets that have that organ as metadata
      • 2. find all files and datasets that have the above datasets as their predecessors (for this we will need to attach a list of all predecessors to each file and dataset)
• no upward flow:
  • if we want to get all collections and datasets that have a particular type of file:
    • solr query:
      • 1. find all dataset and files with this attribute
      • 2. aggregate their predecessor list to get all the unique datasets and collections

Virtual FS Search: original organization: -> datasets by sites -> datasets by organs -> files (has attribute type)

want to browse by the following hierarchy: -> datasets by type -> datasets by sites -> datasets by organs -> files

• no upward/downward flow: the files don't have any metadata about which organ or site they belong to! (assuming that we know the hierarchy on disk already! which is not going to generalize well!)
  • 1. get all the unique types available in general (dataset, files)
  • 2. for say a particular type, we need to get all the unique sites for it:
       • find all files with that type
       • find all its predecessors
       • find all the unique sites in them (this step is going to be very costly!)
  • 3. Now, for a particular site, we have to find all the organs that they have, while also making sure that those organs still maintain type as what was chosen! (quite involved)
• for a general solution for this, we treat each selected metadata field as a filter.
  • for the first selected metadata, we get a list of all datasets and files that directly have that metadata or have a predecessor with that metadata.
  • we also get all the datasets that have successors with that metadata!
  • we will call this subset 1 (which is essentially a tree)
  • then we get all the unique metadata values of second selected metadata field
  • the selection of a single value in above will prune the tree further:
    • either we search first all the predecessors and successors of all the files and datasets which has that particular value for field 2, then taken an intersection with the list we have already
    • or, we search in that list itself that we have already!

Solutions:

• have a heirarchial metadata enrichment step where metadata files are created where h-metadata is written on disk (maybe a mirroring dir structure like thumbnails dirs), which will be then used by the publishing step
  • this way more control can be exerted on which metadata is allowed to flow up and/or down

[hoodriverheather]

@yuliujpl One thing I just noticed when testing. If I select a "custom" hierarchy for a collection, that "custom" hierarchy doesn't change when I move to another collection. If that new collection doesn't have that metadata, no data shows up. Is there a way to have the hierarchy "stay" with the user and the collection?

[yuliujpl]

Yes! Let me implement that

[yuliujpl]

Done in dev