Streamline datasets for documenation

[LucaMarconato]

We should make the usage of datasets more heterogenous across the notebooks in the docs.

Practically:

• [ ] select 1, max 3, small datasets (<1 GB each, ideally ~100 MB), use these datasets in all the notebooks across the repos:
  • [ ] spatialdata
  • [ ] spatialdata-plot
  • [ ] napari-spatialdata
• [ ] in particular, remove the non-bio datasets from the docs (e.g. remove the raccoon dataset from the transformation notebook, and the blobs dataset from the aggregation and rasterize notebooks)
• [ ] implement a dataset class, like in squidpy, to automatically download the datasets

CC @timtreis @melonora

[LucaMarconato]

We will use the following datasets:

• [ ] Visium: P2 CRC from here https://www.10xgenomics.com/products/visium-hd-spatial-gene-expression/dataset-human-crc (subsampled)
• [ ] Xenium: XOA 3.0 5K https://www.10xgenomics.com/datasets/xenium-prime-fresh-frozen-mouse-brain (subsampled)
• [ ] Visium HD: P2 CRC from here https://www.10xgenomics.com/products/visium-hd-spatial-gene-expression/dataset-human-crc (subsampled)
• [ ] CosMx: NSCLC dataset (we need to subsample)
• [x] MERFISH prototype pipeline
• [x] MIBITOF (the one coming from squidpy, converted in spatialdata-sandbox)

For citations (used in the readme of spatialdata-notebooks/datasets and when we use the data in the tutorial notebooks), here are the referneces:

• VIsium and Visium HD: https://pages.10xgenomics.com/WEB-2024-05-TECH-LIT-VIS-HD-PREPRINT-PROMO_LP.html
  • [ ] we should parse the SpaceRanger clusters into the data (they are not parsed by spatialdata-io)
• Xenium: not sure there is a paper, but we link the data page above
• CosMx, MERFISH and MIBITOF: the paper is already linked in the readme in spatialdata-notebooks/datasets

spatialdata notebooks

• [x] I. Use SpatialData with your data: the SpatialData object. No dataset used -> the SpatialData object. No dataset used
• [x] II. Use SpatialData with your data: SpatialElements and tables: custom dataset -> no change
• [ ] (needed for the workshops) Transformations and coordinate systems: raccoon -> Xenium
• [ ] Spatial query: visium mouse brain -> Visium
• [ ] Annotating regions of interest with napari: visium breast cancer -> Visium
• [ ] Use landmark annotations to align multiple -omics layers: visium + xenium breast cancer -> Visium + Visium HD
• [ ] Working with annotations in SpatialData: blobs -> Xenium
• [ ] Integrate/aggregate signals across spatial layers: blobs and custom dataset -> Xenium
• [ ] Interchangeability between raster and vector representations: blobs + custom binned dataset -> Xenium + Visium HD (for the bins)
• [ ] Squidpy integration: xenium breast cancer -> Xenium
• [ ] (needed for the workshops) Deep learning example on image tiles: xenium breast cancer (annotated with the xenium_visium_00 paper notebook) + visium breast cancer
  • [ ] move the notebook into the paper reproducibillity notebook
  • [ ] link the old notebook in the one
  • [ ] the new notebook is going to be very lightweight and not training a model (just dataloader): using Visium

spatialdata-plot notebooks

• [ ] Technology notebooks Visium: visium mouse brain -> Visium
• [x] Technology notebooks MIBI-TOF: no change
• [ ] Technology notebooks MERFISH: no change, but:
  • [ ] rename the title to "MERFISH prototype pipeline"
  • [ ] explain in the notebook that this is not MERFISH from Vizgen, but indeed the prototype pipeline
• [x] Technology notebooks CosMx: no change
• [ ] Technology notebooks Visium HD: visium hd mouse intenstine -> Visium HD
• [ ] Technology notebooks Xenium: XOA 2.0.x -> Xenium
• [ ] Implicit performance improvements when plotting raster data: visium breast cancer -> Visium

napari-spatialdata notebooks

• [x] Analyse MibiTOF in Napari-SpatialData: no change
• [ ] Analyse Nanostring data in Napari-SpatialData: nanostring cosmx -> CosMx (it's the same dataset, but here I mean to use the subsampled dataset and the download API)
• [ ] Using Napari-SpatialData: nanostring cosmx -> CosMx
• [ ] Use the Scatterwidget with AnnData from Notebook scatterwidget.ipynb: visium_hne_adata AnnData format -> choose a bigger object and SpatialData object: use Visium
• [ ] Use the Scatterwidget with AnnData from Notebook scatterwidget_annotation.ipynb: same as above
• [ ] annotation widget notebook: Visium

tasks

• [ ] cosmx, mibitof and merfish are already available in spatialdata-sandbox, add the 3 missing datasets.
• [ ] we add the missing datasets to the Readme in spatialdata-notebooks/datatasets.
  • [ ] add a disclaimer that the data is a subet
• [ ] and for the elected 6 datasets above, we say that we use them in the docs
• [ ] add a job in the data CI
  • [ ] that converts the raw data to SpatialData
  • [ ] subsets the data
  • [ ] write them to Zarr
  • [ ] upload them to S3.
• [ ] write a small frontend (like the squidpy one), so that we download the data via code, better for the user
• [ ] add a disclaimer that the data is a subset also in each notebook