RFC: Supporting Large Datasets, Multiple Backends and Improving Cloud Usability

[seveibar]

Acknowledgements

• Huge thanks to @Ownmarc and @CedricJean for their incredible work adding S3 support to UDT and getting this process started

Motivation

• We want to add authentication mechanisms and make various cloud providers extremely usable with UDT
• We want to support large datasets with and without backend services
  • A Large Dataset is a dataset greater than 100,000 samples
  • We want to support Large Datasets in the Desktop Application, Backend Services and on the Web
• To do this, we want to avoid loading entire datasets into memory
• We want to allow datasets to be browsed easily, even if Large
• We want to enable Discovery of datasets you’ve created or shared datasets on cloud platforms

Overview

• [ ] P1: Support URL Protocols requiring authentication
• [ ] P2: Support Custom Loading Mechanism with new UDT Reference File Style
• [ ] P3: Support Asynchronous Retrieval of UDT Data

P1: Support URL Protocols requiring authentication

Support datasets with S3 URIs and attempt to authenticate and get a usable https/http URL.

{
  "imageUrl": "s3://mybucket/path/to/file.png"
}

P2: Support Custom Loading Mechanism with new UDT Reference File Style

• This enables intuitive cloud-native structures for storing data
• Allows adaptation to Postgres, SQLite, Azure, GCP or whatever the user is using

{
  "interface": { /* ... */ },
  "sampleLoader": {
    "type": "s3",
    "s3Path": "s3://mybucket/subdir",
    // implies existence of...
    // s3://mybucket/subdir/data
    // s3://mybucket/subdir/annotations
  }
}

P3: Support Asynchronous Retrieval of UDT Data

• Allows us to avoid loading large samples into memory
• Allows better separation of logic for supporting local storage, filesystem, collaborative session, s3 and others

import {
  useInterface,     // [loading, interface, changeInterface]
  useSample,        // [loading, sample, changeSample]
  useMetaData,      // {loading, numberOfSamples, numberOfSamplesCompleted, sampleIds}
  useImportSamples, // async (samples) => null
  useDeleteSamples  // async (sampleIdsOrIndices) => null
} from "../hooks/use-udt-reference"

const AsyncUniversalDataViewer = ({ sampleIndexOrId, udtRef }) => {
    const [loadingInterface, interface] = useInterface(udtRef)
    const [loadingSample, sample] = useSample(udtRef, sampleIndexOrId)

    if (loadingInterface||loadingSample) return "loading..."

    return (
      <UniversalDataViewer
        sample={sample}
        interface={interface}
      />
    )
}

[seveibar]

P3 Modifications

1. No Array Returns

Let's just not use array returns:

// BAD
const [loadingInterface, interface] = useInterface(udtRef)

// GOOD
const {loadingInterface, interface} = useInterface(udtRef)

2. useSampleSummary

I think useSampleSummary should be another method that returns a clone of samples with less information, such that grid views can be populated, e.g.

const {loadingSampleSummary, sampleSummary} = useSampleSummary()
// returns: [
//   { color: "#ff0000", hasAnnotation: false },
//   ...
// ]

3. No udtRef

Using udtRef everywhere (which btw, since the refactoring should be datasetRef) is unnecessary since there is only ever one dataset loaded. So just nix it and have it provided by context or whatever.

4. useFullDataset

const { loadingDataset, dataset, datasetLoadingProgress } = useFullDataset()
// loads ALL samples into dataset, this would be used before downloading