Support for multiple intrinsics and different image sizes

[pooncs]

Hi I am coming from Instant-ngp, and lately, they added the capability to load individual intrinsics (by moving the global intrinsics in the json to each image). This also allows for loading different size images (or rotated images). I was wondering if there is any plans to add this capability? Thanks.

[orperel]

Hi @pooncs, Thanks for your interest! Since we're currently preoccupied with other tasks, I could guide you how to do it if you wish to do so.

Variable-sized images and custom intrinsics can be supported by adding a new type of pytorch dataset, similar to how the MultiviewDataset was created (see: https://github.com/NVIDIAGameWorks/kaolin-wisp/blob/8df9d034224da45053bf135a2e9a640edf23d084/wisp/datasets/multiview_dataset.py#L17):

1. Create a MultiviewIntrinsicsDataset class. You can use MultiviewDataset as a reference.
2. Have a look at nerf_standard.py - you want to update this logic (https://github.com/NVIDIAGameWorks/kaolin-wisp/blob/8df9d034224da45053bf135a2e9a640edf23d084/wisp/datasets/formats/nerf_standard.py#L74).
   • [ ] Avoid batching the images together (see: https://github.com/NVIDIAGameWorks/kaolin-wisp/blob/8df9d034224da45053bf135a2e9a640edf23d084/wisp/datasets/formats/nerf_standard.py#L166)
   • [ ] Make sure to move the intrinsics calculation in these lines within the for-loop that goes over the camera.
3. In the new MultiviewIntrinsicsDataset class, invoke your updated logic and again take care not to break due to stacking images together (the assumption in MultiviewDataset no longer holds).

If you post a MR, I can help quickly review and integrate it (we could also replace the current logic if the performance penalty is negligible).