Preprocessing for Motion Denoising

[vadeli]

Thanks for the great work! I tried using the motion denoising model on my data, but the results do not make sense. Is there any preprocessing needed for the data to run the optimization smoothly? I would really appreciate your help so that I can run the denoising on my data. Here is my sample motion: sample_motion I used this command to run the code: python -m run.motion_denoising --config configs/subvp/amass_scorefc_continuous.py --file-path ./converted2amass.npy --noise-std 0.0

and changed the denoise function to only read my skeleton motion and run the optimization:

def denoise(config, args, model, data_file, out_path, std=0.04, verbose=False):
    device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')
    sample_motion = np.load(data_file)
    joints3d = torch.from_numpy(sample_motion).to(device)

   # THE REST OF THE CODE HASN'T CHANGED
    batch_size = len(joints3d)
    #  load body model
    body_model = BodyModel(bm_path=args.bodymodel_path, model_type='smplx',
                           batch_size=batch_size, num_betas=10).to(args.device)

    if args.time_strategy in ['1'] or args.dataset == 'HPS':
        sds_weight = 0.5
    else:
        sds_weight = 1.0  # If you try to enlarge 't_max' or 't_fixed', reduce weight for converge.
    # create Motion denoiser layer
    smpl_mean_params = np.load(constants.SMPL_MEAN_PATH)
    rot6d_poses = torch.tensor(smpl_mean_params['pose'], dtype=torch.float32)
    axis_poses = rot6d_to_axis_angle(rot6d_poses.reshape(-1, 6)).reshape(-1)[None, 3:66].to(args.device) \
                 + torch.randn((batch_size, 63), device=args.device) * 0.01
    motion_denoiser = MotionDenoise(config, args, model, body_model=body_model,
                                    sds_weight=sds_weight,  # For axis setting 1e-1,
                                    batch_size=batch_size, out_path=out_path,
                                    pose_init=axis_poses)
    post_smooth = args.post_smooth
    if std <= 0.02:
        kwargs = {'iterations': 3, 'steps_per_iter': 40, 't_max': 0.2, 't_min': 0.05, 't_fixed': 0.1}
        smooth_args = {'post_smooth': post_smooth, 'window_size': 3, 'sigma': 2}
    elif std == 0.04:
        kwargs = {'iterations': 3, 'steps_per_iter': 60, 't_max': 0.25, 't_min': 0.05, 't_fixed': 0.1}
        smooth_args = {'post_smooth': post_smooth, 'window_size': 3, 'sigma': 2}
    elif std == 0.1:
        kwargs = {'iterations': 3, 'steps_per_iter': 80, 't_max': 0.15, 't_min': 0.05, 't_fixed': 0.1}
        smooth_args = {'post_smooth': post_smooth, 'window_size': 3, 'sigma': 2}
    else:
        raise NotImplementedError()
    kwargs.update(smooth_args)

    if args.file_path is not None:  # visualization for toy data
        verbose = True
        kwargs['vis'] = True

    motion_denoiser.optimize(joints3d, args.time_strategy, verbose=verbose, **kwargs)

This is one frame SMPL output that I am getting after the optimization process:

[moonbow721]

Hi there,

Thank you for reaching out and sharing your sample_motion data.

Upon reviewing your data, I noticed that it includes both translation and global orientation from the SMPL model. However, the code we provide only considers local poses using joints3d, which is why it fails to fit the data you supplied.

To improve this, you can modify the optimize method in the MotionDenoise class to include translation and global orientation parameters for the optimizer. You can reference the following line in the code: MotionDenoise.optimize method.

Additionally, when obtaining the final result, make sure to pass these two parameters for proper visualization: Final result visualization.

I hope this helps! Please let me know if you have any further questions or need additional assistance.