_Oncology FM Evaluation Framework by kaiko.ai_
[![PyPI](https://img.shields.io/pypi/v/kaiko-eva.svg?logo=python)](https://pypi.python.org/pypi/kaiko-eva)
[![docs](https://img.shields.io/badge/š_docs-latest-green)](https://kaiko-ai.github.io/eva/latest)
[![license](https://img.shields.io/badge/āļø_License-Apache%202.0-blue.svg?labelColor=gray)](https://github.com/kaiko-ai/eva#license)
[![paper](http://img.shields.io/badge/OpenReview-MIDL_2024-B31B1B.svg)](https://openreview.net/forum?id=FNBQOPj18N¬eId=FNBQOPj18N)
Installation ā¢
How To Use ā¢
Documentation ā¢
Datasets ā¢
Benchmarks
Contribute ā¢
Acknowledgements
eva
is an evaluation framework for oncology foundation models (FMs) by kaiko.ai.
Check out the documentation for more information.
Highlights:
- Easy and reliable benchmark of Oncology FMs
- Automatic embedding inference and evaluation of a downstream task
- Native support of popular medical datasets and models
- Produce statistics over multiple evaluation fits and multiple metrics
Installation
Simple installation from PyPI:
# to install the core version only
pip install kaiko-eva
# to install the expanded `vision` version
pip install 'kaiko-eva[vision]'
# to install everything
pip install 'kaiko-eva[all]'
To install the latest version of the main
branch:
pip install "kaiko-eva[all] @ git+https://github.com/kaiko-ai/eva.git"
You can verify that the installation was successful by executing:
eva --version
How To Use
eva
can be used directly from the terminal as a CLI tool as follows:
eva {fit,predict,predict_fit} --config url/or/path/to/the/config.yaml
When used as a CLI tool, eva
supports configuration files (.yaml
) as an argument to define its functionality.
Native supported configs can be found at the configs directory
of the repo. Apart from cloning the repo, you can download the latest config folder as .zip
from your browser from
here. Alternatively,
from a specific release the configs can be downloaded from the terminal as follows:
curl -LO https://github.com/kaiko-ai/eva/releases/download/0.0.1/configs.zip | unzip configs
For example, to perform a downstream evaluation of DINO ViT-S/16 on the BACH dataset with
linear probing by first inferring the embeddings and performing 5 sequential fits, execute:
# from a locally stored config file
eva predict_fit --config ./configs/vision/dino_vit/offline/bach.yaml
# from a remote stored config file
eva predict_fit --config https://raw.githubusercontent.com/kaiko-ai/eva/main/configs/vision/dino_vit/offline/bach.yaml
[!NOTE]
All the datasets that support automatic download in the repo have by default the option to automatically download set to false.
For automatic download you have to manually set download=true.
To view all the possibles, execute:
eva --help
For more information, please refer to the documentation
and tutorials.
Benchmarks
In this section you will find model benchmarks which were generated with eva
.
Table I: WSI classification tasks
| Model | BACH | CRC | MHIST | PCam | Camelyon16 | PANDA |
|---------|-------|-------|-------|--------|------------|-------|
| ViT-S/16 _(random)_ [1] | 0.411|0.613|0.5|0.752|0.551|0.347|
| ViT-S/16 _(ImageNet)_ [1] | 0.675|0.936|0.827|0.861|0.751|0.676|
| DINO(p=16) [2] | 0.77|0.936|0.751|0.905|0.869|0.737|
| Phikon [3] | 0.715|0.942|0.766|0.925|0.879|0.784|
| UNI [4] | 0.797|0.95|0.835|0.939|0.933|0.774|
| ViT-S/16 _(kaiko.ai)_ [5] | 0.8|0.949|0.831|0.902|0.897|0.77|
| ViT-S/8 _(kaiko.ai)_ [5] | 0.825|0.948|0.826|0.887|0.879|0.741|
| ViT-B/16 _(kaiko.ai)_ [5] | 0.846|0.959|0.839|0.906|0.891|0.753|
| ViT-B/8 _(kaiko.ai)_ [5] | 0.867|0.952|0.814|0.921|0.939|0.761|
| ViT-L/14 _(kaiko.ai)_ [5] | 0.862|0.935|0.822|0.907|0.941|0.769|
_Table I: Linear probing evaluation of FMs on patch-level downstream datasets.
We report averaged balanced accuracy
over 5 runs. Results are reported on the "test" split if available and otherwise on the "validation" split.
References:
- "Emerging properties in self-supervised vision transformersā, arXiv
- "Benchmarking self-supervised learning on diverse pathology datasetsā, arXiv
- "Scaling self-supervised learning for histopathology with masked image modelingā, medRxiv
- "A General-Purpose Self-Supervised Model for Computational Pathologyā, arXiv
- "Towards Training Large-Scale Pathology Foundation Models: from TCGA to Hospital Scaleā, arXiv
Contributing
eva
is an open source project and welcomes contributions of all kinds. Please checkout the developer
and contributing guide for help on how to do so.
All contributors must follow the code of conduct.
Acknowledgements
Our codebase is built using multiple opensource contributions
[![python](https://img.shields.io/badge/-Python-blue?logo=python&logoColor=white)](https://github.com/pre-commit/pre-commit)
[![pytorch](https://img.shields.io/badge/PyTorch-ee4c2c?logo=pytorch&logoColor=white)](https://pytorch.org/get-started/locally/)
[![lightning](https://img.shields.io/badge/-ā”ļø_Lightning-792ee5?logo=pytorchlightning&logoColor=white)](https://pytorchlightning.ai/)
[![black](https://img.shields.io/badge/Code%20Style-Black-black.svg?labelColor=gray)](https://black.readthedocs.io/en/stable/)
[![isort](https://img.shields.io/badge/%20imports-isort-%231674b1?style=flat&labelColor=ef8336)](https://pycqa.github.io/isort/)
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
[![Checked with pyright](https://microsoft.github.io/pyright/img/pyright_badge.svg)](https://microsoft.github.io/pyright/)
[![pdm-managed](https://img.shields.io/badge/pdm-managed-blueviolet)](https://pdm-project.org)
[![Nox](https://img.shields.io/badge/%F0%9F%A6%8A-Nox-D85E00.svg)](https://github.com/wntrblm/nox)
[![Built with Material for MkDocs](https://img.shields.io/badge/Material_for_MkDocs-526CFE?logo=MaterialForMkDocs&logoColor=white)](https://squidfunk.github.io/mkdocs-material/)
Citation
If you find this repository useful, please consider giving a star ā and adding the following citation:
@inproceedings{kaiko.ai2024eva,
title={eva: Evaluation framework for pathology foundation models},
author={kaiko.ai and Ioannis Gatopoulos and Nicolas K{\"a}nzig and Roman Moser and Sebastian Ot{\'a}lora},
booktitle={Medical Imaging with Deep Learning},
year={2024},
url={https://openreview.net/forum?id=FNBQOPj18N}
}