This package is a framework for automated tissue classification and segmentation on medical hyperspectral imaging (HSI) data. It contains:
This framework is designed to work on HSI data from the Tivita cameras but you can adapt it to different HSI datasets as well. Potential applications include:
htc
framework to work on the data (cf. our tutorials).If you use the htc
framework, please consider citing the corresponding papers. You can also cite this repository directly via:
This package can be installed via pip:
pip install imsy-htc
This installs all the required dependencies defined in requirements.txt
. The requirements include PyTorch, so you may want to install it manually before installing the package in case you have specific needs (e.g. CUDA version).
⚠️ This framework was developed and tested using the Ubuntu 20.04+ Linux distribution. Despite we do provide wheels for Windows and macOS as well, they are not tested.
⚠️ Network training and inference was conducted using an RTX 3090 GPU with 24 GiB of memory. It should also work with GPUs which have less memory but you may have to adjust some settings (e.g. the batch size).
imsy-htc[extra]
)This framework can be configured via environment variables. Most importantly, we need to know where your data is located (e.g. PATH_Tivita_HeiPorSPECTRAL
) and where results should be stored (e.g. PATH_HTC_RESULTS
). For a full list of possible environment variables, please have a look at the documentation of the Settings
class.
💡 If you set an environment variable for a dataset path, it is important that the variable name matches the folder name (e.g. the variable name
PATH_Tivita_HeiPorSPECTRAL
matches the dataset pathmy/path/HeiPorSPECTRAL
with its folder nameHeiPorSPECTRAL
, whereas the variable namePATH_Tivita_some_other_name
does not match). Furthermore, the dataset path needs to point to a directory which contains adata
and anintermediates
subfolder.
There are several options to set the environment variables. For example:
~/.bashrc
or before running each command:
PATH_HTC_RESULTS="~/htc/results" htc training --model image --config "models/image/configs/default"
However, this might get cumbersome or might not give you the flexibility you need.
.env
file in the repository root and add your variables, for example:
export PATH_Tivita_HeiPorSPECTRAL=/mnt/nvme_4tb/HeiPorSPECTRAL
export PATH_HTC_RESULTS=~/htc/results
~/.config/htc/variables.env
. Please run htc info
upon package installation to retrieve the exact location on your system. The content of the file is of the same format as of the .env
above.After setting your environment variables, it is recommended to run htc info
to check that your variables are correctly registered in the framework.
A series of tutorials can help you get started on the htc
framework by guiding you through different usage scenarios.
💡 The tutorials make use of our public HSI dataset HeiPorSPECTRAL. If you want to directly run them, please download the dataset first and make it accessible via the environment variable
PATH_Tivita_HeiPorSPECTRAL
as described above.
htc
framework. Namely, it demonstrates the usage of the DataPath
class which is the entry point to load and process HSI data. For example, you will learn how to read HSI cubes, segmentation masks and meta data. Among others, you can use this information to calculate the median spectrum of an organ.DataPath
class so that you can load and process your images and annotations. We collected some tips on how this can be achieved.We do not have a separate documentation website for our framework yet. However, most of the functions and classes are documented so feel free to explore the source code or use your favorite IDE to display the documentation. If something does not become clear from the documentation, feel free to open an issue!
This framework gives you access to a variety of pretrained segmentation and classification models. The models will be automatically downloaded, provided you specify the model type (e.g. image
) and the run folder (e.g. 2022-02-03_22-58-44_generated_default_model_comparison
). It can then be used for example to create predictions on some data or as a baseline for your own training (see example below).
💡 The modality
param
refers to stacked tissue parameter images (named TPI in our paper “Robust deep learning-based semantic organ segmentation in hyperspectral images”). For the model typepatch
, pretrained models are available for the patch sizes 64 x 64 and 32 x 32 pixels. The modality and patch size is not specified when loading a model as it is already characterized by specifying a certain run folder.
After successful installation of the htc
package, you can use any of the pretrained models listed in the table. There are several ways to use them but the general principle is that models are always specified via their model
and run_folder
.
Every model class listed in the table has a static method pretrained_model()
which you can use to create a model instance and initialize it with the pretrained weights. The model object will be an instance of torch.nn.Module
. The function has examples for all the different model types but as a teaser consider the following example which loads the pretrained image HSI network:
import torch
from htc import ModelImage, Normalization
run_folder = "2022-02-03_22-58-44_generated_default_model_comparison" # HSI model
model = ModelImage.pretrained_model(model="image", run_folder=run_folder, n_channels=100, n_classes=19)
input_data = torch.randn(1, 100, 480, 640) # NCHW
input_data = Normalization(channel_dim=1)(input_data) # Model expects L1 normalized input
model(input_data).shape
# torch.Size([1, 19, 480, 640])
💡 Please note that when initializing the weights as in this example, the segmentation head is initialized randomly. Meaningful predictions on your own data can thus not be expected out of the box, but you will have to train the model on your data first.
The models can be used to predict segmentation masks for your data. The segmentation models automatically sample from your input image according to the selected model spatial granularity (e.g. by creating patches) and the output is always a segmentation mask for an entire image. The set of output classes is determined by the training configuration, e.g. 18 organ classes + background for our semantic models. The CreatingPredictions
notebook shows how to create predictions and how to map the network output to meaningful label names.
htc
packageIf you are using the htc
framework to train your networks, you only need to define the model in your configuration:
{
"model": {
"pretrained_model": {
"model": "image",
"run_folder": "2022-02-03_22-58-44_generated_default_model_comparison"
}
}
}
This is very similar to option 1 but may be more convenient if you train with the htc
framework.
💡 We have a JSON Schema file which describes the structure of our config files including descriptions of the attributes.
There is a common command line interface for many scripts in this repository. More precisely, every script which is prefixed with run_NAME.py
can also be run via htc NAME
from any directory. For more details, just type htc
.
This repository contains code to reproduce our publications listed below:
In this paper, we tackled fully automatic organ segmentation and compared deep learning models on different spatial granularities (e.g. patch vs. image) and modalities (e.g. HSI vs. RGB). Furthermore, we studied the required amount of training data and the generalization capabilities of our models across subjects. The pretrained networks are related to this paper. You can find the notebooks to generate the paper figures in paper/MIA2022 (the folder also includes a reproducibility document) and the models in htc/models. For each model, there are three configuration files, namely default
, default_rgb
and default_parameters
, which correspond to the HSI, RGB and TPI modality, respectively. You can also download the NSD thresholds which we used for the NSD metric (cf. Fig. 12).
📂 The dataset for this paper is not publicly available.
This MICCAI2023 paper is the direct successor of our MIA2022 paper. We analyzed how well our networks perform under geometrical domain shifts which commonly occur in real-world open surgeries (e.g. situs occlusions). The effect is drastic (drop of Dice similarity coefficient by 45 %) but the good news is that performance on par with in-distribution data can be achieved with our simple, model-independent solution (augmentation method). You can find all the code in htc/context and paper figures as well as reproducibility instructions in paper/MICCAI2023. Pretrained models are available for our organ transplantation networks with HSI and RGB modalities.
💡 If you are only interested in our data augmentation method, you can also head over to Kornia where this augmentation is implemented for generic use cases (including 2D and 3D data). You will find it under the name
RandomTransplantation
.📂 The dataset for this paper is not publicly available.
The poster was presented at the PyTorch Conference 2023 and presents several solutions to improve data loading for faster network training. This originated from our MICCAI2023 paper, where we load huge amount of data while using a relatively small network resulting in GPU idle times when the GPU has to wait for the CPU to deliver new data. This requested the need for fast data loading strategies so that the CPU delivers data in-time for the GPU. The solutions include (1) efficient data storage via Blosc compression, (2) appropriate precision settings, (3) GPU instead of CPU augmentations using the Kornia library and (4) a fixed shared pinned memory buffer for efficient data transfer to the GPU. For the last part, you will find the relevant code to create the buffer in this repository as part of the SharedMemoryDatasetMixin class (_add_tensor_shared()
method).
You can find the code to generate the results figures of the poster in paper/PyTorchConf2023 including reproducibility instructions.
📂 The dataset for this poster is not publicly available.
In this paper, we trained a classification model based on median spectra from HSI data. You can find the model code in htc/tissue_atlas and the confusion matrix figure of the paper in paper/NatureReports2022 (including a reproducibility document).
📂 The dataset for this paper is not fully publicly available, but a subset of the data is available through the public HeiPorSPECTRAL dataset.
This paper introduces the HeiPorSPECTRAL dataset containing 5756 hyperspectral images from 11 subjects. We are using these images in our tutorials. You can find the visualization notebook for the paper figures in paper/NatureData2023 (the folder also includes a reproducibility document) and the remaining code in htc/tissue_atlas_open.
If you want to learn more about the HeiPorSPECTRAL dataset (e.g. the underlying data structure) or you stumbled upon a file and want to know how to read it, you might find this notebook with low-level details helpful.
📂 The dataset for this paper is publicly available.
This paper presents several applications of intraoperative HSI, including our organ segmentation and classification work. You can find the code generating our figure for this paper at paper/Chirurg2022.
📂 The sample image used here is contained in the dataset from our paper “Robust deep learning-based semantic organ segmentation in hyperspectral images” and hence not publicly available.
This project has received funding from the European Research Council (ERC) under the European Unions Horizon 2020 research and innovation programme (NEURAL SPICING, grant agreement No. 101002198) and was supported by the German Cancer Research Center (DKFZ) and the Helmholtz Association under the joint research school HIDSS4Health (Helmholtz Information and Data Science School for Health). It further received funding from the Surgical Oncology Program of the National Center for Tumor Diseases (NCT) Heidelberg.