BachiLi / redner

Differentiable rendering without approximation.
https://people.csail.mit.edu/tzumao/diffrt/
MIT License
1.39k stars 139 forks source link

CUDA Runtime Error ... at /app/buffer.h:86 #58

Closed lonelywm closed 5 years ago

lonelywm commented 5 years ago

Such errors occur randomly and irregularly: CUDA Runtime Error: an illegal memory access was encountered at /app/buffer.h:86

My Configuration: Ubuntu 16.4 CUDA 10.0 Test both on OptiX 6.0.0 and OptiX5.1.1 Pytorch 1.1.0 g++ 7 NVIDIA 1080ti

And I just set mesh(shape) and it's translation & rotation to be parameters

BachiLi commented 5 years ago

Does this happen for the scripts in tests directory? If no can you share your test script with me?

lonelywm commented 5 years ago
# input an image and output a vertor of (199 + 29 + 3 + 3) , 199 + 29 if the shape and expression of 3dmm face model, 3 + 3 is for the translation and rotation.
class Encoder(nn.Module):
    def __init__(self, size):
        super().__init__()
        self.seq_1 = nn.Sequential(
            # 128->64
            nn.Conv2d(3, 32, 3, padding = 1),
            nn.LeakyReLU(),
            nn.BatchNorm2d(32),
            nn.MaxPool2d(2, 2),
            # 64->32
            nn.Conv2d(32, 64, 3, padding = 1),
            nn.LeakyReLU(),
            nn.BatchNorm2d(64),
            nn.MaxPool2d(2, 2),
            # 32->16
            nn.Conv2d(64, 128, 3, padding = 1),
            nn.LeakyReLU(),
            nn.BatchNorm2d(128),
            nn.MaxPool2d(2, 2),
            # 16->8
            nn.Conv2d(128, 256, 3, padding = 1),
            nn.LeakyReLU(),
            nn.BatchNorm2d(256),
            nn.MaxPool2d(2, 2),
            # 8->4
            nn.Conv2d(256, 512, 3, padding = 1),
            nn.LeakyReLU(),
            nn.MaxPool2d(2, 2),
        )
        self.seq_2 = nn.Sequential(
            # 128->64
            nn.Linear(512 * 4 * 4, 1024),
            nn.LeakyReLU(),
            nn.Linear(1024, size),
            nn.Tanh()
        )

    def forward(self, image):
        output = self.seq_1(image)
        output = self.seq_2(output.view(-1, 512 * 4 * 4))
        return output.view(-1) * 0.5

# generate mesh vector based on the shape and expression 3dmm face 
class MModel(nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, shape_para, exp_para, mm):
        vertices = mm.model['shapeMU'] + mm.model['shapePC'].mm(shape_para) + mm.model['expPC'].mm(exp_para)
        vertices = vertices.contiguous().view(-1, 3)
        vertices = vertices.permute(1, 0)
        return vertices.t()

def train():
    target = pyredner.imread('results/test_single_triangle/target.png')
    input = target.view(1, 3, 128, 128).cuda()
    if pyredner.get_use_gpu():
        target = target.cuda(device = pyredner.get_device())

    cam = pyredner.Camera(position = torch.tensor([0.0, 0, 4.0], requires_grad=False),
                      look_at = torch.tensor([0.0, 0.0, 0.0]),
                      up = torch.tensor([0.0, 1.0, 0.0]),
                      fov = torch.tensor([45.0]), # in degree
                      clip_near = 1e-2, # needs to > 0
                      resolution = (128, 128),
                      fisheye = False)
    mat_grey = pyredner.Material(diffuse_reflectance = torch.tensor([1., 1., 1.], device = pyredner.get_device()))
    materials = [mat_grey]

    model = MModel()
    encoder = Encoder(199 + 29 + 3 + 3).cuda() # ep sp tran rot
    mm = MorphabelModel()
    output = encoder(input)
    sp = mm.get_shape_para('random')
    ep = mm.get_exp_para('random')

    sp, ep, translation_params, euler_angles = torch.split(output, [199, 29, 3, 3], 0)
    sp = sp.view([-1, 1])
    ep = ep.view([-1, 1])
    vert = model(sp, ep, mm)
    mm.full_triangles = mm.full_triangles.cuda()
    vert = vert.cuda()

    mesh = pyredner.Shape(\
        vertices = vert,
        indices = mm.full_triangles,
        uvs = None,
        normals = None,
        material_id = 0)
    mesh.normals = pyredner.compute_vertex_normal(mesh.vertices, mesh.indices)

    shape_light = pyredner.Shape(\
        vertices = torch.tensor([[-1.0, -1.0, 7.0],
                                [ 1.0, -1.0, 7.0],
                                [-1.0,  1.0, 7.0],
                                [ 1.0,  1.0, 7.0]], device = pyredner.get_device()),
        indices = torch.tensor([[0, 2, 1],[1, 2, 3]], dtype = torch.int32, device = pyredner.get_device()),
        uvs = None,
        normals = None,
        material_id = 0)

    shapes = [mesh, shape_light]
    light = pyredner.AreaLight(shape_id = 1, intensity = torch.tensor([15.0,15.0,15.0]))
    area_lights = [light]
    scene = pyredner.Scene(cam, shapes, materials, area_lights)
    scene_args = pyredner.RenderFunction.serialize_scene(\
        scene = scene,
        num_samples = 128,
        max_bounces = 1)

    render = pyredner.RenderFunction.apply
    img = render(0, *scene_args)

    pyredner.imwrite(img.cpu(), 'results/test_single_triangle/init.png')
    diff = torch.abs(target - img)
    pyredner.imwrite(diff.cpu(), 'results/test_single_triangle/init_diff.png')
    optimizer = torch.optim.Adam(encoder.parameters(), lr=1e-5)

    for t in range(3000):
        print('iteration:', t)
        optimizer.zero_grad()
        # Forward pass: render the image.
        output = encoder(input)
        sp, ep, translation_params, euler_angles = torch.split(output, [199, 29, 3, 3], 0)
        sp = sp.view([-1, 1])
        ep = ep.view([-1, 1])
        mesh.vertices = model(sp, ep, mm).cuda()

        rotation_matrix = pyredner.gen_rotate_matrix(euler_angles)
        rotation_matrix = rotation_matrix.cuda()
        translation = translation_params

        center = torch.mean(mesh.vertices.clone(), 0)
        mesh.vertices =  (mesh.vertices - center) @ torch.t(rotation_matrix) + center + translation
        mesh.normals = pyredner.compute_vertex_normal(mesh.vertices, mesh.indices)

        scene_args = pyredner.RenderFunction.serialize_scene(\
            scene = scene,
            num_samples = 16, # We use less samples in the Adam loop.
            max_bounces = 1)
        # Important to use a different seed every iteration, otherwise the result
        img = render(t+1, *scene_args)
        pyredner.imwrite(img.cpu(), 'results/test_single_triangle/iter_{}.png'.format(t))
        loss = (img - target).pow(2).sum()
        print('loss:', loss.item())
        loss.backward()
        torch.nn.utils.clip_grad_norm_(encoder.parameters(), 100)
        optimizer.step()

    scene_args = pyredner.RenderFunction.serialize_scene(\
        scene = scene,
        num_samples = 16,
        max_bounces = 1)
    img = render(302, *scene_args)
    # Save the images and differences.
    pyredner.imwrite(img.cpu(), 'results/test_single_triangle/final.exr')
    pyredner.imwrite(img.cpu(), 'results/test_single_triangle/final.png')
    pyredner.imwrite(torch.abs(target - img).cpu(), 'results/test_single_triangle/final_diff.png')

    # Convert the intermediate renderings to a video.
    from subprocess import call
    call(["ffmpeg", "-framerate", "24", "-i",
        "results/test_single_triangle/iter_%d.png", "-vb", "20M",
        "results/test_single_triangle/out.mp4"])

if __name__ == "__main__":
    train()
lonelywm commented 5 years ago

Not in test dir. Thank you , my code is a bit long...

BachiLi commented 5 years ago

I am missing the MorphableModel module. Where do I get it?

lonelywm commented 5 years ago

I have uploaded all my code and data to github: https://github.com/lonelywm/face3d-model

lonelywm commented 5 years ago

I think I have got the reason, which is on the vertex normal, after I del all the "mesh.normals = pyredner.compute_vertex_normal(mesh.vertices, mesh.indices)", the err disappeared.

BachiLi commented 5 years ago

Still looking into this but quick comment:

    target = pyredner.imread('results/test_single_triangle/target.png')
    input = target.view(1, 3, 128, 128).cuda()

This is incorrect -- pyredner.imread returns an array with shape (height, width, channel). You need to use torch.permute to get the NCHW format used by pytorch.

Also I'm getting

ValueError: Unknown mat file type, version 50, 55

when reading from BFM.mat

BachiLi commented 5 years ago

Ok it seems that I'll need to run the generation script at https://github.com/YadiraF/face3d/tree/master/examples/Data/BFM Let me download matlab first...

BachiLi commented 5 years ago

I have to clear up space in harddisk for matlab installation. This will take long.

BachiLi commented 5 years ago

I think the problem is that the optimization generates invalid vertices/normals after some iterations. In particular if you have degenerate triangles in the mesh compute_vertex_normal would generate 0/0. I'll fix compute_vertex_normal. In general we need better error messages.

BachiLi commented 5 years ago

I think I fix this in the latest commit. Let me know if this fixes your issue.

lonelywm commented 5 years ago

I'm sorry to take up so much of your time. My problem has indeed been solved. The system now works very well, and it can fit the simple 3dmm.

BachiLi commented 5 years ago

No problem, I was just ranting. Thanks for reporting and feel free to reopen the issue if the problem occurs again.