In training, loss starts from a very small number.

[LinyeLi60]

lly@lly-XPS-13-9360:~/Downloads/Deep_Object_Pose/src/training$ python train.py --data ~/fat/ --object mug --outf mug start: 23:05:16.797258 load data training data: 61500 batches load models Train Epoch: 1 [0/246000 (0%)] Loss: 0.035773992538452 Train Epoch: 1 [400/246000 (0%)] Loss: 0.000075019132055 Train Epoch: 1 [800/246000 (0%)] Loss: 0.002809691475704 Train Epoch: 1 [1200/246000 (0%)] Loss: 0.000005863605111 Train Epoch: 1 [1600/246000 (1%)] Loss: 0.000004666615496 Train Epoch: 1 [2000/246000 (1%)] Loss: 0.000004310214536 Train Epoch: 1 [2400/246000 (1%)] Loss: 0.002799089299515

This is how the loss change when I train an object from scratch with a batchsize of 4 on my gtx1070, learing rate is 0.0001.I wonder if there is some problem.

[TontonTremblay]

These are very low loss values, ~0.0028 is more common to me for a fully trained model. Can you investigate how the belief maps look like, there might be some black images in there.

Can you try to overlay the belief maps onto the image:

def OverlayBeliefOnImage(img, beliefs, name, path="", factor=0.7, grid=3, 
        norm_belief = True):
    """ python
    take as input 
    img: a tensor image in pytorch normalized at 0.5
            3xwxh
    belief: tensor of the same size as the image to overlay over img 
            nb_beliefxwxh
    name: str to name the image, e.g., output.png
    path: where to save, e.g., /where/to/save/
    factor: float [0,1] how much to keep the original, 1 = fully, 0 black
    grid: how big the grid, e.g., 3 wide. 
    norm_belief: bool to normalize the values [0,1]
    """

    tensor = beliefs
    belief_imgs = []
    in_img = img
    in_img *= factor           
    norm_belief = True
    for j in range(tensor.size()[0]):
        belief = tensor[j].clone()
        if norm_belief:
            belief -= float(torch.min(belief).data.cpu().numpy())
            belief /= float(torch.max(belief).data.cpu().numpy())
        belief = torch.clamp(belief,0,1).cpu()
        belief = torch.cat([
                    belief.unsqueeze(0) + in_img[0,:,:],
                    belief.unsqueeze(0) + in_img[1,:,:],
                    belief.unsqueeze(0) + in_img[2,:,:]
                    ]).unsqueeze(0)
        belief = torch.clamp(belief,0,1) 

        belief_imgs.append(belief.data.squeeze().numpy())

    # Create the image grid
    belief_imgs = torch.tensor(np.array(belief_imgs))

    save_image(belief_imgs, "{}{}".format(path, name), 
        mean=0, std=1, nrow=grid)

Also if you are training on a textureless mug I am not sure it is going to work well given the symmetries of the object.