search

arcadelab / deepdrr

Code for "DeepDRR: A Catalyst for Machine Learning in Fluoroscopy-guided Procedures". https://arxiv.org/abs/1803.08606
GNU General Public License v3.0
209 stars 60 forks source link

How to provide a torch tensor volume as input to DeepDRR without detaching from the computational graph. #63

Closed rassaire closed 2 years ago

rassaire commented 2 years ago

Hello, I am using DeepDRR and I need help. I really like the DRR output, it's really looks like real X-rays.

I am using a DeepDRR in a pytorch. So my input to the DeepDRR and the output DRR must be torch tensors. This is because I want to keep these tensors attached to the computational graphs. Converting them to numpy or whatever would detach them from the computational graphs, which would be a problem because the model weights won't be updated.

In the code execution guide, you explain how to use it in a pytorch code. But it was not clear if the input volume can be a torch tensor as well as the output.

Your help will be appreciated.

mathiasunberath commented 2 years ago

At this moment, DeepDRR is not designed to be used in end-to-end training setups like the one you are describing. We do not use gradient checkpointing etc. to enable backpropagation through the projections. We have other modules and solutions to do that effectively, see here: https://github.com/gaocong13/Projective-Spatial-Transformers

I agree that having this capability would be appealing, but we are currently prioritizing other features. If you are interested in contributing the features you talk about, we'd of course welcome them.

rassaire commented 2 years ago

Thanks Mathias, I will it out.