If you haven't already, please read through our contributing guidelines before opening your PR
What is the purpose of this PR?
Main Features
Adds a notebook for converting the cell table to per-FOV AnnData objects. By default, they get saved to data/<my-cohort>/anndata in the notebook. In addition, it briefly covers viewing components of the AnnData object, and creating data pipelines with a small application for constructing spatial neighbors with squidpy.
Added the conversion class which creates a AnnData object with:
Dropped settings.CELL_SIZE in favor of the generic "area" as a column name in obs, as we can have tables for many kinds of segmentations / observations (cell, fiber, ez_seg, etc...)
Added a function to load all AnnData Zarr stores in a directory to an AnnCollection.
Added a custom datapipe to iterate over the AnnCollection FOV by FOV.
Documentation
Added a brief AnnData section for data_types.md with figures.
Added a AnnData section for development.md.
Testing
Adjusted test_utils.make_cell_table to have the expected ordering for Cell Tables. Added n_cells and n_markers to create different sized cell tables.
Added a utility function to generate AnnData tables with various values of n_obs, and floating point and categorical column types. (test_utils.generate_anndata_table and test_utils.generate_anncollection)
How did you implement your changes
Used Dask to work with converting large cell tables.
Used TorchData to create the AnnDataIterDataPipe. This allows us to implement more intricate data pipelines with shuffling, mapping, and filtering functionality, and construct data loaders with DataLoader2
Adjusted tests which utilized test_utils.make_cell_table.
Remaining Issues
Need a plan for structuring how we will write modules and functions for interacting with AnnData objects. We can try some stuff, and see what works best.
If you haven't already, please read through our contributing guidelines before opening your PR
What is the purpose of this PR?
Main Features
Adds a notebook for converting the cell table to per-FOV
AnnData
objects. By default, they get saved todata/<my-cohort>/anndata
in the notebook. In addition, it briefly covers viewing components of theAnnData
object, and creating data pipelines with a small application for constructing spatial neighbors withsquidpy
.AnnData
object with:centroid-0
$\to$centroid_y
,centroid-1
$\to$centroid_x
), stored inobsm["spatial"]
.var_names
/X
.obs
.settings.CELL_SIZE
in favor of the generic"area"
as a column name inobs
, as we can have tables for many kinds of segmentations / observations (cell, fiber, ez_seg, etc...)AnnCollection
.datapipe
to iterate over theAnnCollection
FOV by FOV.Documentation
AnnData
section fordata_types.md
with figures.AnnData
section fordevelopment.md
.Testing
test_utils.make_cell_table
to have the expected ordering for Cell Tables. Addedn_cells
andn_markers
to create different sized cell tables.AnnData
tables with various values ofn_obs
, and floating point and categorical column types. (test_utils.generate_anndata_table
andtest_utils.generate_anncollection
)How did you implement your changes
Dask
to work with converting large cell tables.TorchData
to create theAnnData
IterDataPipe
. This allows us to implement more intricate data pipelines with shuffling, mapping, and filtering functionality, and construct data loaders withDataLoader2
test_utils.make_cell_table
.Remaining Issues
AnnData
objects. We can try some stuff, and see what works best.