Closed alvaro-budria closed 1 year ago
I found the problem and a solution for it.
Turns out that at some point I was calling the following snippet in my geometry network:
def gradient(self, x,):
with torch.set_grad_enabled(True):
x.requires_grad_(True)
y = self.sdf(x)
gradients = torch.autograd.grad(
outputs=y,
inputs=x,
grad_outputs=torch.ones_like(y),
create_graph=True,
retain_graph=True,
only_inputs=True,
)[0]
return gradients.reshape(-1, 3)
The line
x.requires_grad_(True)
was causing the graph on x
to be empty, and thus an illegal memory access happened, or at least that's the explanation I could come up with.
At any rate, changing the line above for
x = x + 0 * self.__hidden__(x)
solved the problem! I also needed to add this
self.__hidden__ = torch.nn.Linear(3, 1, bias=False)
to the __init__
of my class.
Thanks for you sharing! Can I ask that how to use "with torch.no_grad():" in "packed_info, t_starts, t_ends = ray_marching( rays_o, rays_d, scene_aabb=self.scene_aabb, grid=None, # self.occupancy_grid if self.grid_prune else None, alpha_fn=alpha_fn, near_plane=near.squeeze(), far_plane=far.squeeze(), render_step_size=self.render_step_size, stratified=self.randomized, cone_angle=0.0, alpha_thre=0.0, )" To be specific, the gradient calculation in alpha_fn has to use grad_fn of pytorch. It's that possible to use "with torch.no_grad():" here? Thanks again!
Yes, you can wrap the ray_marching
in a with torch.no_grad()
, however inside the forward
of the geometry module, you need to add a line enabling grad for the forward pass there: with torch.set_grad_enabled(self.training or (with_grad and self.grad_type == 'analytic')):
.
You can check the repo I based my code on, to see the details: https://github.com/bennyguo/instant-nsr-pl/tree/main/models
Thanks for the good work. I was trying out the CUDA accelerated rendering provided in this repo. I am trying to use the alpha-based rendering.
So far it seems that training proceeds for a few epochs (~1-300, usually not more), and then an error pops up.
I tried executing the script both with
CUDA_LAUNCH_BLOCKING=1
and without it. When this flag is set, the error iswithout any further clarifications, and it always happens when running the following line:
Without this flag, the error message is
and as you can see from the error trace, it is the same line that leads to the crash
Reducing the number of samples per ray and the batch size to extremely small values does drive away the error. However, I have strong doubts that this is an OOM problem because:
nerfacc
's CUDA rendering, I was using my own pure PyTorch pipeline with a large batch size and a large number of samples per ray.watch -d -n 0.01 nvidia-smi
. At no point does the memory shoot up to the 23028MiB limit of the server'sNVIDIA A10
.The rendering code I have is
Any ideas?