search

caiyuanhao1998 / SAX-NeRF

"Structure-Aware Sparse-View X-ray 3D Reconstruction" (CVPR 2024)
https://arxiv.org/abs/2311.10959
MIT License
465 stars 19 forks source link

About your strategies #3

Closed Zhentao-Liu closed 6 months ago

Zhentao-Liu commented 6 months ago

A great work on CT reconstruction. I wanna ask a question. The strategies you developed in your paper: lineformer and MLG, are they suitable for natural scenes, i.e. RGB NeRF? It seems that they are not specific for X-rays.

caiyuanhao1998 commented 6 months ago

Hi, thanks for your interest.

The lineformer can replace the MLP in NeRF but I do not know its performance. The MLG sampling uses a binary mask first. This step I guess can not be used in RGB imaging. But the subsequent steps of MLG can be applied.

We will release our code before 2024.06.16. I will let you know and you are welcome to have a try.

Zhentao-Liu commented 6 months ago

I am still confusing. Should we cast some rays at the blank area (where mask=0)? Otherwise, how did the model get to know whether its empty? Wouldnt be blurry?

caiyuanhao1998 commented 6 months ago

For CT, no. It's very sparse. According to the experiments, the rendered projections are not blurry

Zhentao-Liu commented 6 months ago

How do you achieve non-overlap patch sampling in the mask image? Could you simply describe it?

Zhentao-Liu commented 6 months ago

How about apply mamba to replace your transformer block?

caiyuanhao1998 commented 5 months ago

it is interesting. I just released code. Feel free to have a try