NeRF?

[antithing]

Hi! Random question, could this code be used to render NeRFs in UE?

[ShiinaMitsuki]

Hi! Random question, could this code be used to render NeRFs in UE?

Yes，you can extract rgb density slices, and use the nerf rendering formula by modify little code.

[kenichdietrich]

Hi, how can we combine RGB and density? I was able to import the density in MHD format, but I don't know how to combine it with RGB arrays. Maybe importing each channel in separated MHD files?

[ShiinaMitsuki]

Hi, how can we combine RGB and density? I was able to import the density in MHD format, but I don't know how to combine it with RGB arrays. Maybe importing each channel in separated MHD files?

Use 4 channel (RGBA) floating point array, alpha channel for density, remember to rescale the density range to [0, 1]. You can save in "exr" format using opencv imwrite function, and then import to UE.

[kenichdietrich]

Thank you for your response, I am getting started with this plugin and I am a bit confused on how to use it. I discretized a NeRF to a volume of voxels/pixels, with the RGB and density channels, so I have an array of dimensions (L, L, L, 4). I have tried to generate the EXR file you pointed out but I found out that it is not possible for a 3D image. Does it mean I have to generate each layer of the volume as a 2D EXR image? Can I import a folder of images into UE and use it with the plugin?

[ShiinaMitsuki]

Thank you for your response, I am getting started with this plugin and I am a bit confused on how to use it. I discretized a NeRF to a volume of voxels/pixels, with the RGB and density channels, so I have an array of dimensions (L, L, L, 4). I have tried to generate the EXR file you pointed out but I found out that it is not possible for a 3D image. Does it mean I have to generate each layer of the volume as a 2D EXR image? Can I import a folder of images into UE and use it with the plugin?

Lets say that you have a DxHxWx4 (RGB and density) 3D data, D for axis z, H for axis y, W for axis x in the nerf coordinate for example. So you got a number of D 2D image slices along z axis.

1st, save the image slices as a tiled image, for example a (D/32xW)x(32xH)x4 image in "exr" format.

2nd, import the tiled image to UE, and create a volume texture from it, adjust the "Tile Size X" , "Tile Size Y" to H and W, the "Format" to floating point format(RGBA32f/RGBA16f).

3rd, use this volume texture to sample some value when ray casting in UVW space(this plugin), and composite the final color and alpha along the ray.

[antithing]

@kenichdietrich if you successfully use this to render a nerf, would you be able to share your work flow here? Thanks!

[marospekarik]

I would be interested to see that implementation as well @kenichdietrich <3

[wannaseoji]

Hi. I want to ask a question about this plugin.

1. Could I render a .den file format?
2. Is there any way to add a physics engine(c++ codes) to this plugin? thank you