Understand results

[legoc1986]

Hi @xiangruhuang , I have tried to reproduce results using your codes. I generated point clouds and mesh_Correspondences. Here are the input shown in image.

Then I input the point cloud into your proposed method, Here are input point clouds(in green) and template points (green dark)

The correspondences between them obtained are very few, like this

The question is why obtained correspondences are that few number?

here are my codes.

import os, os.path as osp
import sys
project_name1 = os.path.abspath(os.path.join(os.path.dirname(__file__)))
sys.path.insert(0, project_name1)
sys.path.append(os.path.join(project_name1,'src'))
sys.path.append(os.path.join(project_name1,'src','human_corres'))
sys.path.append(os.path.join(project_name1,'src','training'))

import re
import numpy as np
import glob
import progressbar
import torch
from human_corres.utils import helper
from training.model import HybridModel
from training.utils import parse_args
from training.multi_gpu_handler import MultiGPUHandler
from human_corres.datasets.amass import amassTestFEPts
from torch_geometric.data import Data, DataLoader
from matplotlib import pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from mpl_toolkits.mplot3d.art3d import Poly3DCollection
from smpl.smpl_np import SMPLModel
from human_corres.utils.visualization import *
if __name__ == '__main__':
  args = parse_args()
  print(args)
  torch.cuda.manual_seed_all(816)
  # Load Checkpoint to Model
  model = HybridModel(args)
  device = torch.device('cuda:0')

  ckpt = '/home/wang/disk/Schedar/HumanCorresViaLearn2Sync/data/ckpt/HumanMultiGPU/phase2/13.pt' 
  checkpoint = torch.load(ckpt, map_location=device)
  model_dict = model.state_dict()
  pretrained_dict = checkpoint['model_state_dict']
  pretrained_dict = {k: v for k, v in pretrained_dict.items() if k in model_dict}
  model_dict.update(pretrained_dict)
  model.load_state_dict(model_dict)

  test_dataset= amassTestFEPts(
      descriptor_dim=args.embed_dim,
      split=args.test_split,
      cls=False,
      desc=args.desc,
    )

  # # Create loader
  test_loaders =  hc.data.DataLoader(test_dataset, batch_size=1, shuffle=False)
  handler = MultiGPUHandler(test_loaders, args, training=False,reg=False)

  # Look into Data and process data with Model
  for batch in test_loaders:

    out_dict = model(batch)

    loss, corres_list = handler.parse(out_dict, [batch], require_corres=True)
    correspondence= corres_list['corres'][0]
    print(np.unique(correspondence))
    # plt.hist(correspondence,bins=6890)
    # plt.show()
    print('correspondence shape',correspondence.shape)
    points3d_i=corres_list['ori_pos'][0]
    pred_p=corres_list['pred_before_reg'][0]
    template_points=corres_list['template_points'][0]
    targets =template_points[correspondence, :]

[xiangruhuang]

Let me see if I understand your question correctly.

For each point in the point cloud, we output an integer between [0, 6890). For example, if the output for the i-th point is 6300, it means this point corresponds to the 6300-th point (starting from 0) in the SMPL template.

Does this answer your question?

[legoc1986]

Hi @xiangruhuang yes, but I obtained correspondences, with very few numbers, most of input points share the same vertex in the SMPL template, which is not correct. Best

[xiangruhuang]

Can you explain what you mean by "with very few numbers" and "most of input points share the same vertex in the SMPL template" ?

Can you report the geodesic or euclidean error of the correspondences?

[xiangruhuang]

You can also upload a the correspondence output and the input point cloud to let me take a look.

[legoc1986]

Can you explain what you mean by "with very few numbers" and "most of input points share the same vertex in the SMPL template" ?

Can you report the geodesic or euclidean error of the correspondences?

This means, for example a input points cloud has [5135,3], the output correspondence gives [5135], where each value is between 0 and 6890, where if I use np.unique(correspondence), gives only [27] different values:

 [ 130  347  570  702 1547 1550 1551 1556 1557 1578 1579 1692 1693 1946
 1947 1948 1971 2375 2382 2383 2390 2395 3534 3535 3739 3853 6822].

targets =template_points[correspondence, :] show targets

[xiangruhuang]

Hi,

This is not the expected performance of the network. But it's hard to guess what went wrong. Can you send me the input point cloud and the ground truth correspondence? I can run a quick test to make sure.

Best, Xiangru

[legoc1986]

@xiangruhuang , I run another experiments on surreal test set of 1K test scans download https://github.com/xiangruhuang/HumanCorresViaLearn2Sync/blob/main/docs/human/surreal.md. Here are correspondences read from .mat, which is good while input to model pretrained here: HumanCorresViaLearn2Sync/data/ckpt/HumanMultiGPU/phase2/13.pt. I get correspondences like this:

[xiangruhuang]

Hi,

Sorry for the late reply. Finally found time to work on this after spring festival and a power outage in Texas.

Did you check if the checkpoint is correctly loaded? What I have found is that recent version (1.6.x) of pytorch-geometric seems to have different attribute names for the class PointConv? In my checkpoint, I have variable names like sa1_module.mlp.seq.x where the new version of pytorch-geometric produces names like sa1_module.conv.local_nn.x. Therefore I worry that the variables are not correctly loaded.

Can you dump the keys of pretrained_dict and model_dict right after this line to confirm if you're in the same situation? https://github.com/xiangruhuang/HumanCorresViaLearn2Sync/blob/25615f50c47f4cebe2aaccb7826aef94cffdc72f/src/training/multi_train.py#L133

I'm working on translating the variables to newer versions. Will get back to you soon. (This times much faster!)

Best, Xiangru

[xiangruhuang]

If this is your case, I can upload a new checkpoint that matches the key names.

[xiangruhuang]

Hi @legoc1986,

Here is an updated version of checkpoint that works for recent versions of PyTorch and torch-geometric.

If you are still interested in reproducing the results, please take a look. https://www.dropbox.com/s/ofot6r4k19so6zv/human_ckpt_torch17.zip

Best, Xiangru

[legoc1986]

Thank you @xiangruhuang I will make a try and get back to you

[xiangruhuang]

Thanks a lot!

I've found that the new versions of PyTorch/Torch-Geometric seems to have trouble reproducing the results by simply reading the old checkpoint (which may or may not be your case). So I've made a few new checkpoints re-trained on the new versions.

These checkpoints are trained from PyTorch 1.7.1+cu101 and Torch-Geometric 1.6.3. https://www.dropbox.com/s/sve168gstekfbym/Human_TG_1_7.zip You should be able to evaluate the results by

python -m training.multi_train --checkpoints Human_TG_1_7 --embed_dim 100 --desc Laplacian_n --warmstart latest --testing --batch_size 3 --transf_reg

Let me know if any of those helps.

Best, Xiangru

[legoc1986]

@xiangruhuang I have successfully installed PyTorch 1.7.1+cu101 and Torch-Geometric 1.6.3. and make run the network. Here are some run history: I obtained results like this:

I have single GPU