CUDA Runtime Error ... at /app/buffer.h:86

[lonelywm]

Such errors occur randomly and irregularly： CUDA Runtime Error: an illegal memory access was encountered at /app/buffer.h:86

My Configuration: Ubuntu 16.4 CUDA 10.0 Test both on OptiX 6.0.0 and OptiX5.1.1 Pytorch 1.1.0 g++ 7 NVIDIA 1080ti

And I just set mesh(shape) and it's translation & rotation to be parameters

[BachiLi]

Does this happen for the scripts in tests directory? If no can you share your test script with me?

[lonelywm]

# input an image and output a vertor of (199 + 29 + 3 + 3) , 199 + 29 if the shape and expression of 3dmm face model, 3 + 3 is for the translation and rotation.
class Encoder(nn.Module):
    def __init__(self, size):
        super().__init__()
        self.seq_1 = nn.Sequential(
            # 128->64
            nn.Conv2d(3, 32, 3, padding = 1),
            nn.LeakyReLU(),
            nn.BatchNorm2d(32),
            nn.MaxPool2d(2, 2),
            # 64->32
            nn.Conv2d(32, 64, 3, padding = 1),
            nn.LeakyReLU(),
            nn.BatchNorm2d(64),
            nn.MaxPool2d(2, 2),
            # 32->16
            nn.Conv2d(64, 128, 3, padding = 1),
            nn.LeakyReLU(),
            nn.BatchNorm2d(128),
            nn.MaxPool2d(2, 2),
            # 16->8
            nn.Conv2d(128, 256, 3, padding = 1),
            nn.LeakyReLU(),
            nn.BatchNorm2d(256),
            nn.MaxPool2d(2, 2),
            # 8->4
            nn.Conv2d(256, 512, 3, padding = 1),
            nn.LeakyReLU(),
            nn.MaxPool2d(2, 2),
        )
        self.seq_2 = nn.Sequential(
            # 128->64
            nn.Linear(512 * 4 * 4, 1024),
            nn.LeakyReLU(),
            nn.Linear(1024, size),
            nn.Tanh()
        )

    def forward(self, image):
        output = self.seq_1(image)
        output = self.seq_2(output.view(-1, 512 * 4 * 4))
        return output.view(-1) * 0.5

# generate mesh vector based on the shape and expression 3dmm face 
class MModel(nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, shape_para, exp_para, mm):
        vertices = mm.model['shapeMU'] + mm.model['shapePC'].mm(shape_para) + mm.model['expPC'].mm(exp_para)
        vertices = vertices.contiguous().view(-1, 3)
        vertices = vertices.permute(1, 0)
        return vertices.t()

def train():
    target = pyredner.imread('results/test_single_triangle/target.png')
    input = target.view(1, 3, 128, 128).cuda()
    if pyredner.get_use_gpu():
        target = target.cuda(device = pyredner.get_device())

    cam = pyredner.Camera(position = torch.tensor([0.0, 0, 4.0], requires_grad=False),
                      look_at = torch.tensor([0.0, 0.0, 0.0]),
                      up = torch.tensor([0.0, 1.0, 0.0]),
                      fov = torch.tensor([45.0]), # in degree
                      clip_near = 1e-2, # needs to > 0
                      resolution = (128, 128),
                      fisheye = False)
    mat_grey = pyredner.Material(diffuse_reflectance = torch.tensor([1., 1., 1.], device = pyredner.get_device()))
    materials = [mat_grey]

    model = MModel()
    encoder = Encoder(199 + 29 + 3 + 3).cuda() # ep sp tran rot
    mm = MorphabelModel()
    output = encoder(input)
    sp = mm.get_shape_para('random')
    ep = mm.get_exp_para('random')

    sp, ep, translation_params, euler_angles = torch.split(output, [199, 29, 3, 3], 0)
    sp = sp.view([-1, 1])
    ep = ep.view([-1, 1])
    vert = model(sp, ep, mm)
    mm.full_triangles = mm.full_triangles.cuda()
    vert = vert.cuda()

    mesh = pyredner.Shape(\
        vertices = vert,
        indices = mm.full_triangles,
        uvs = None,
        normals = None,
        material_id = 0)
    mesh.normals = pyredner.compute_vertex_normal(mesh.vertices, mesh.indices)

    shape_light = pyredner.Shape(\
        vertices = torch.tensor([[-1.0, -1.0, 7.0],
                                [ 1.0, -1.0, 7.0],
                                [-1.0,  1.0, 7.0],
                                [ 1.0,  1.0, 7.0]], device = pyredner.get_device()),
        indices = torch.tensor([[0, 2, 1],[1, 2, 3]], dtype = torch.int32, device = pyredner.get_device()),
        uvs = None,
        normals = None,
        material_id = 0)

    shapes = [mesh, shape_light]
    light = pyredner.AreaLight(shape_id = 1, intensity = torch.tensor([15.0,15.0,15.0]))
    area_lights = [light]
    scene = pyredner.Scene(cam, shapes, materials, area_lights)
    scene_args = pyredner.RenderFunction.serialize_scene(\
        scene = scene,
        num_samples = 128,
        max_bounces = 1)

    render = pyredner.RenderFunction.apply
    img = render(0, *scene_args)

    pyredner.imwrite(img.cpu(), 'results/test_single_triangle/init.png')
    diff = torch.abs(target - img)
    pyredner.imwrite(diff.cpu(), 'results/test_single_triangle/init_diff.png')
    optimizer = torch.optim.Adam(encoder.parameters(), lr=1e-5)

    for t in range(3000):
        print('iteration:', t)
        optimizer.zero_grad()
        # Forward pass: render the image.
        output = encoder(input)
        sp, ep, translation_params, euler_angles = torch.split(output, [199, 29, 3, 3], 0)
        sp = sp.view([-1, 1])
        ep = ep.view([-1, 1])
        mesh.vertices = model(sp, ep, mm).cuda()

        rotation_matrix = pyredner.gen_rotate_matrix(euler_angles)
        rotation_matrix = rotation_matrix.cuda()
        translation = translation_params

        center = torch.mean(mesh.vertices.clone(), 0)
        mesh.vertices =  (mesh.vertices - center) @ torch.t(rotation_matrix) + center + translation
        mesh.normals = pyredner.compute_vertex_normal(mesh.vertices, mesh.indices)

        scene_args = pyredner.RenderFunction.serialize_scene(\
            scene = scene,
            num_samples = 16, # We use less samples in the Adam loop.
            max_bounces = 1)
        # Important to use a different seed every iteration, otherwise the result
        img = render(t+1, *scene_args)
        pyredner.imwrite(img.cpu(), 'results/test_single_triangle/iter_{}.png'.format(t))
        loss = (img - target).pow(2).sum()
        print('loss:', loss.item())
        loss.backward()
        torch.nn.utils.clip_grad_norm_(encoder.parameters(), 100)
        optimizer.step()

    scene_args = pyredner.RenderFunction.serialize_scene(\
        scene = scene,
        num_samples = 16,
        max_bounces = 1)
    img = render(302, *scene_args)
    # Save the images and differences.
    pyredner.imwrite(img.cpu(), 'results/test_single_triangle/final.exr')
    pyredner.imwrite(img.cpu(), 'results/test_single_triangle/final.png')
    pyredner.imwrite(torch.abs(target - img).cpu(), 'results/test_single_triangle/final_diff.png')

    # Convert the intermediate renderings to a video.
    from subprocess import call
    call(["ffmpeg", "-framerate", "24", "-i",
        "results/test_single_triangle/iter_%d.png", "-vb", "20M",
        "results/test_single_triangle/out.mp4"])

if __name__ == "__main__":
    train()

[lonelywm]

Not in test dir. Thank you , my code is a bit long...

[BachiLi]

I am missing the MorphableModel module. Where do I get it?

[lonelywm]

I have uploaded all my code and data to github: https://github.com/lonelywm/face3d-model

[lonelywm]

I think I have got the reason, which is on the vertex normal, after I del all the "mesh.normals = pyredner.compute_vertex_normal(mesh.vertices, mesh.indices)", the err disappeared.

[BachiLi]

Still looking into this but quick comment:

    target = pyredner.imread('results/test_single_triangle/target.png')
    input = target.view(1, 3, 128, 128).cuda()

This is incorrect -- pyredner.imread returns an array with shape (height, width, channel). You need to use torch.permute to get the NCHW format used by pytorch.

Also I'm getting

ValueError: Unknown mat file type, version 50, 55

when reading from BFM.mat

[BachiLi]

Ok it seems that I'll need to run the generation script at https://github.com/YadiraF/face3d/tree/master/examples/Data/BFM Let me download matlab first...

[BachiLi]

I have to clear up space in harddisk for matlab installation. This will take long.

[BachiLi]

I think the problem is that the optimization generates invalid vertices/normals after some iterations. In particular if you have degenerate triangles in the mesh compute_vertex_normal would generate 0/0. I'll fix compute_vertex_normal. In general we need better error messages.

[BachiLi]

I think I fix this in the latest commit. Let me know if this fixes your issue.

[lonelywm]

I'm sorry to take up so much of your time. My problem has indeed been solved. The system now works very well, and it can fit the simple 3dmm.

[BachiLi]

No problem, I was just ranting. Thanks for reporting and feel free to reopen the issue if the problem occurs again.