How to create an object in another location

[kei971031]

Hello, I'm very confused about how to render using nerf_volume_renderer and have been struggling for days...

For example, I want to create the object centered around [-0.5,0.5,0] rather than [0,0,0] point. RandomCameraIterableDataset defined in data/uncond.py creates random cameras surrounding the [0,0,0] point. From what I understand, _nerf_volumerenderer shoots rays from rays_o (camera position) in the direction of rays_d and accumulates them.

Therefore, I thought that in order to create an object in [-0.5,0.5,0] rather than [0,0,0], I would just use rays_d without changing it and add [-0.5,0.5,0] to rays_o. However, this only works properly when the camera is in the front view and does not work with random cameras. Only when rendering with front view, the object is rendered in the center of the image, which is what I want.

Below is an example of nerf optimization using DreamCraft3D. In the picture above, an object was created at the [0,0,0] position without changing ray_o, and the object is in the center of the image even when rendered in random view. In the picture below, to create the object at the [-0.5, 0.5, 0] location, [-0.5, 0.5, 0] was added to the components of rays_o and an object was created. In the front view, the object is rendered in the center of the image without a problem, but in the random view, the object is not rendered in the center of the image.

I suspected there was a problem with the direction of the ray, so I tested the code below, but the center point was [-0.5, 0.5, 0], so there seemed to be no problem.....

# move ray origin [-0.5, 0.5, 0]
rays_o = batch["rays_o"].clone() 
rays_o[:,:,:,0] += dist_x # -0.5
rays_o[:,:,:,1] += dist_y # 0.5
rays_o[:,:,:,2] += dist_z # 0
batch["rays_o"] = rays_o

# view center point of world space
rays_d = batch["rays_d"].clone()
b, w, h, c = rays_d.shape #shape 1, 128, 128, 3
ww, hh = w//2, h//2
print(ww, hh, rays_d.shape) # center of image 64, 64
direction2origin = rays_d[:,ww, hh, :] # ray direction of center point
print("object_origin", rays_o[:, ww, hh, :] + direction2origin * batch["camera_distances"].item()) # center point of world space [-0.5, 0.5, 0]

My question is, do I need to modify something other than rays_o? I tried modifying things like c2w, camera_position, and mvp_matrix, but it didn't seem to affect rendering result. nerf_volume_rendering appears to only use rays_o and rays_d. Are there other factors to consider?

Also, since rays_d is understood as the direction of rays defined in world space, I thought it could be used as is when moving the position of an object. Isn't that correct?

[kei971031]

The issue has been partially resolved. The camera parameter (rays_o, rays_d) was not problem.

The core problem stemmed from blob_magic3d, which initializes density around the coordinates [0,0,0]. When using SDS Loss with Stable Diffusion and similar methods, a suitable gradient is generated even if the object is not centered within the camera's view. This means the process can function to an extent, despite the initial density not being positioned at the desired location. Ultimately, it was necessary to generate the initial density at the specific location I required, such as [-0.5,0.5,0].

To address this, I modified the get_activated_density function in _implicitvolume.py as follows.

#density_bias = (self.cfg.density_blob_scale  * (1 - torch.sqrt(( (points)**2).sum(dim=-1)) / self.cfg.density_blob_std)[..., None]
density_bias = (self.cfg.density_blob_scale  * (1 - torch.sqrt(( (points - center_point)**2).sum(dim=-1)) / self.cfg.density_blob_std)[..., None]