georghess / neurad-studio

[CVPR2024] NeuRAD: Neural Rendering for Autonomous Driving
https://research.zenseact.com/publications/neurad/
Apache License 2.0
356 stars 25 forks source link

bug: asseme_ego_compensated not work #25

Open blackmrb opened 6 months ago

blackmrb commented 6 months ago

The default value of asseme_ego_compensated is True, but my laser point cloud does not have motion compensation, so I set asseme_ego_compensated to false.
However, asseme_ego_compensated in _generaterarys'from_points() is still true.

@dataclass
class CustomData(ADDataParser):
    .....
    .....
    def _get_lidars(self) -> Tuple[Lidars, List[Path]]:
        """Returns lidar info and loaded point clouds."""

        lidars = Lidars(
            lidar_to_worlds=poses[:, :3, :4],
            lidar_type=LidarType.VELODYNE128,
            times=times,
            assume_ego_compensated=False, # change here
            metadata={"sensor_idxs": idxs},
            horizontal_beam_divergence=HORIZONTAL_BEAM_DIVERGENCE,
            vertical_beam_divergence=VERTICAL_BEAM_DIVERGENCE,
        )

        return lidars, lidar_filenames
    def _generate_rays_from_points(
        self,
        lidar_indices: Int[Tensor, "*num_rays num_lidars_batch_dims"],
        points: Float[Tensor, "*num_points point_dim"],
        lidar_opt_to_lidar: Optional[Float[Tensor, "*num_rays 3 4"]] = None,
    ) -> RayBundle:
       .......
        if points.shape[-1] >= 5 and self.metadata and "velocities" in self.metadata:
            # Offset the point origins according to timediff and velocity
            origins = origins + points[..., 4:5] * self.metadata["velocities"][lidar_indices.squeeze(-1)]
            if not self.assume_ego_compensated: # here
                # offset the world points according to velocity too
                points_world = points_world + points[..., 4:5] * self.metadata["velocities"][lidar_indices.squeeze(-1)]

image

the callback stack is below: image

I think it's the class Lidar base class TensorDataclass problem, but I don't konw how to fix it. I temporarily set the default value of asseme_ego_compensated to false.

georghess commented 6 months ago

Good catch! It seems that when slicing a TensorDataclass it will init a new TensorDataclass with the default init params. For your application, the easiest thing will likely be to change the default. But I'll look at a more general solution