CuNeRF

The source code for our paper "CuNeRF: Cube-Based Neural Radiance Field for Zero-Shot Medical Image Arbitrary-Scale Super Resolution", Zixuan Chen, Lingxiao Yang, Jian-Huang Lai, Xiaohua Xie, IEEE/CVF International Conference on Computer Vision (ICCV), 2023.

Project Page | Paper

Abstract

Medical image arbitrary-scale super-resolution (MIASSR) has recently gained widespread attention, aiming to supersample medical volumes at arbitrary scales via a single model. However, existing MIASSR methods face two major limitations: (i) reliance on high-resolution (HR) volumes and (ii) limited generalization ability, which restricts their applications in various scenarios. To overcome these limitations, we propose Cube-based Neural Radiance Field (CuNeRF), a zero-shot MIASSR framework that is able to yield medical images at arbitrary scales and free viewpoints in a continuous domain. Unlike existing MISR methods that only fit the mapping between low-resolution (LR) and HR volumes, CuNeRF focuses on building a continuous volumetric representation from each LR volume without the knowledge from the corresponding HR one. This is achieved by the proposed differentiable modules: cube-based sampling, isotropic volume rendering, and cube-based hierarchical rendering. Through extensive experiments on magnetic resource imaging (MRI) and computed tomography (CT) modalities, we demonstrate that CuNeRF can synthesize high-quality SR medical images, which outperforms state-of-the-art MISR methods, achieving better visual verisimilitude and fewer objectionable artifacts. Compared to existing MISR methods, our CuNeRF is more applicable in practice.

1) Get start

Python 3.9.x
CUDA 11.1 or higher
NVIDIA RTX 3090
Torch 1.8.0 or higher

Create a python env using conda

conda create -n cunerf python=3.9 -y

Install the required libraries

bash setup.sh

[option] Install FFmpeg

apt install ffmpeg -y

2) Training CuNeRF for medical volumes

python run.py <expname> --cfg <config file> --scale <SR scale> --mode train --file <filepath>

See example_train.sh for details, we also provide an example config file in the configs dir.

3) Arbitrary rendering for medical slices

Render slices at arbitrary positions (zpos: $-0.1$ ~ $0.1$), scales ($1$.x ~ $2$.x) and viewpoints (angles: $0$ ~ $360$ degrees) with an rotation axis $[1,1,0]$:

python run.py <expname> --cfg <config file> --mode test --file <filepath> --scales 1 2 --zpos -0.1 0.1 --angles 0 360 --axis 1 1 0 --asteps 45

See example_test.sh for details.

Citation

@InProceedings{Chen_2023_ICCV,
author    = {Chen, Zixuan and Yang, Lingxiao and Lai, Jian-Huang and Xie, Xiaohua},
title     = {CuNeRF: Cube-Based Neural Radiance Field for Zero-Shot Medical Image Arbitrary-Scale Super Resolution},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
month     = {October},
year      = {2023},
pages     = {21185-21195}
}

Acknowledgement

We build our project based on NeRF-Pytorch. We thank them for their wonderful work and code release.

NarcissusEx / CuNeRF

readme