Render with blender

[silence401]

thanks for your wonderful work! I try to render the output with blender, but i can't get the result the mess_pass output.

[jmunkberg]

Thanks for your kind words!

Unfortunately, we cannot actively support the Blender workflow, as Blender keeps changing etc. I've attached a version of a script we use to load our exported scenes in Blender, which may serve as a starting point. Not entirely sure if roughness should be squared etc.

Note also that the NerFactor versions of the NerF scenes are easier (and what we used for the relighting examples in the paper).

Here is an example on the nerf_hotdog training

import os
import bpy 
import numpy as np

# path to your mesh
MESH_PATH = ".../nerf_hotdog/mesh"

bpy.ops.file.pack_all()
scene = bpy.context.scene
scene.world.use_nodes = True
scene.render.engine = 'CYCLES'

################### Import obj mesh ###################

imported_object = bpy.ops.import_scene.obj(filepath=os.path.join(MESH_PATH, "mesh.obj"))
obj_object = bpy.context.selected_objects[0]
obj_object.rotation_euler = [0, 0, 0]

################### Fix material graph ###################

# Get material node tree, find BSDF and specular texture
material = obj_object.active_material
bsdf = material.node_tree.nodes["Principled BSDF"]
image_node_ks = bsdf.inputs["Specular"].links[0].from_node

# Split the specular texture into metalness and roughness
separate_node = material.node_tree.nodes.new(type="ShaderNodeSeparateRGB")
separate_node.name="SeparateKs"
material.node_tree.links.new(image_node_ks.outputs[0], separate_node.inputs[0])
material.node_tree.links.new(separate_node.outputs[2], bsdf.inputs["Metallic"])
material.node_tree.links.new(separate_node.outputs[1], bsdf.inputs["Roughness"])

material.node_tree.links.remove(bsdf.inputs["Specular"].links[0])

# Set default values
bsdf.inputs["Specular"].default_value = 0.5
bsdf.inputs["Specular Tint"].default_value = 0.0
bsdf.inputs["Sheen Tint"].default_value = 0.0
bsdf.inputs["Clearcoat Roughness"].default_value = 0.0

################### Load HDR probe ###################

envmap = scene.world.node_tree.nodes.new(type="ShaderNodeTexEnvironment")
envmap.image = bpy.data.images.load(os.path.join(MESH_PATH, "probe.hdr"))
scene.world.node_tree.links.new(envmap.outputs[0], scene.world.node_tree.nodes['Background'].inputs['Color'])

[jmunkberg]

Here is a visual example of the shading network as well

[silence401]

Thanks for your kind words!

Unfortunately, we cannot actively support the Blender workflow, as Blender keeps changing etc. I've attached a version of a script we use to load our exported scenes in Blender, which may serve as a starting point. Not entirely sure if roughness should be squared etc.

Note also that the NerFactor versions of the NerF scenes are easier (and what we used for the relighting examples in the paper).

Here is an example on the nerf_hotdog training

import os
import bpy 
import numpy as np

# path to your mesh
MESH_PATH = ".../nerf_hotdog/mesh"

bpy.ops.file.pack_all()
scene = bpy.context.scene
scene.world.use_nodes = True
scene.render.engine = 'CYCLES'

################### Import obj mesh ###################

imported_object = bpy.ops.import_scene.obj(filepath=os.path.join(MESH_PATH, "mesh.obj"))
obj_object = bpy.context.selected_objects[0]
obj_object.rotation_euler = [0, 0, 0]

################### Fix material graph ###################

# Get material node tree, find BSDF and specular texture
material = obj_object.active_material
bsdf = material.node_tree.nodes["Principled BSDF"]
image_node_ks = bsdf.inputs["Specular"].links[0].from_node

# Split the specular texture into metalness and roughness
separate_node = material.node_tree.nodes.new(type="ShaderNodeSeparateRGB")
separate_node.name="SeparateKs"
material.node_tree.links.new(image_node_ks.outputs[0], separate_node.inputs[0])
material.node_tree.links.new(separate_node.outputs[2], bsdf.inputs["Metallic"])
material.node_tree.links.new(separate_node.outputs[1], bsdf.inputs["Roughness"])

material.node_tree.links.remove(bsdf.inputs["Specular"].links[0])

# Set default values
bsdf.inputs["Specular"].default_value = 0.5
bsdf.inputs["Specular Tint"].default_value = 0.0
bsdf.inputs["Sheen Tint"].default_value = 0.0
bsdf.inputs["Clearcoat Roughness"].default_value = 0.0

################### Load HDR probe ###################

envmap = scene.world.node_tree.nodes.new(type="ShaderNodeTexEnvironment")
envmap.image = bpy.data.images.load(os.path.join(MESH_PATH, "probe.hdr"))
scene.world.node_tree.links.new(envmap.outputs[0], scene.world.node_tree.nodes['Background'].inputs['Color'])

thank you for your qucik reply, I think the script is very useful! Is there any render engine can repeat the results OR is it possible to use nvdiffrast to get the results?

[jmunkberg]

nvdiffrec is rendering the scenes in the validation loop already, so that is the "reference" engine (rasterizer with direct illumination without shadows), and the renderer we use in the optimization process. In Blender, we render with path tracing, so that is not a perfect match, but works reasonably well.

[silence401]

Got it, thanks for your kindly reply.

[andrewyguo]

@jmunkberg, I am having trouble running the script you provided above.

When I run it, I am getting the error:

Info: No new files have been packed
Info: No new files have been packed
(  0.0000 sec |   0.0000 sec) Importing OBJ '/home/andrewg/Documents/nvdiffrec/out/nerf_hotdog/hotdog_mesh_nvdiffrec/mesh.obj'...
  (  0.0001 sec |   0.0000 sec) Parsing OBJ file...
    (  0.5009 sec |   0.5008 sec) Done, loading materials and images...
WARNING: '/home/andrewg/Documents/nvdiffrec/out/nerf_hotdog/hotdog_mesh_nvdiffrec/mesh.obj':b'bsdf   pbr' (ignored)
    (  0.5024 sec |   0.5023 sec) Done, building geometries (verts:27788 faces:55632 materials: 1 smoothgroups:1) ...
    (  0.9443 sec |   0.9442 sec) Done.
  (  0.9444 sec |   0.9444 sec) Finished importing: '/home/andrewg/Documents/nvdiffrec/out/nerf_hotdog/hotdog_mesh_nvdiffrec/mesh.obj'
Progress: 100.00%

1
Exception ignored in: <function CyclesRender.__del__ at 0x7f7cd9679a70>
Traceback (most recent call last):
  File "/home/andrewg/miniconda3/envs/bpy/lib/python3.7/site-packages/2.91/scripts/addons/cycles/__init__.py", line 67, in __del__
TypeError: 'NoneType' object is not callable
Segmentation fault (core dumped)

I am new to using Blender and bpy. What is the expected output of this script?

Update

I figured that I needed to run this script inside the Blender script editor. However, after I run the script, the loaded mesh does not have any colour.

[jmunkberg]

Hello,

Please use this script: https://github.com/NVlabs/nvdiffrecmc/blob/main/blender/blender.py I just tested it in Blender, and it generates reasonable results. You need to look at the shading tab or render the mesh in Cycles to see the shading.

One example on the Bob model below (tested in Blender 3.3.1)